April 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Social Impact, Inc. It was authored by Saaka Adams, Charles Nornoo, S. Dunham Rowley, and Brenda Sinclair. FINAL PERFORMANCE EVALUATION OF USAID/GHANA’S PARTNERSHIP FOR ACCOUNTABLE GOVERNANCE IN EDUCATION (PAGE) PROJECT Final Report FINAL PERFORMANCE EVALUATION OF USAID/GHANA’S PARTNERSHIP FOR ACCOUNTABLE GOVERNANCE IN EDUCATION (PAGE) PROJECT Final Report April 4, 2014 Contract Number: AID-OAA-I-10-00013 Order Number: AID-641-TO-14-00001 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Final Evaluation of the USAID/Ghana PAGE Project Final Report i TABLE OF CONTENTS Table of Contents ........................................................................................................................................... i Acronyms ..................................................................................................................................................... iii Executive Summary ...................................................................................................................................... v Project Background and Purpose .................................................................................................................. v Evaluation Questions, Design, and Methods ................................................................................................ v Findings, Conclusions, and Recommendations ............................................................................................ v Introduction ................................................................................................................................................... 1 Project Background ....................................................................................................................................... 1 Evaluation Purpose and Questions ................................................................................................................ 1 Evaluation Purpose ....................................................................................................................................... 1 Evaluation Questions .................................................................................................................................... 2 Evaluation Methods and Limitations ............................................................................................................ 2 Methodology ................................................................................................................................................. 2 Findings, Conclusions, and Recommendations ............................................................................................ 5 Evaluation Question 1: Effect on Student Achievement ............................................................................... 5 Evaluation Question 2: Improving Governance and Supervision in Schools ...............................................9 Evaluation Question 3: Lessons Learned and Best Practices ......................................................................14 Evaluation Question 4: Ownership, Engagement, and Sustainability ......................................................... 20 Evaluation Question 5: Performance Monitoring Processes, Systems, and Tools ...................................... 23 General Conclusion for PAGE Performance Evaluation ............................................................................ 27 ANNEX I: WORKPLAN ........................................................................................................................... 28 ANNEX II: PAGE PROJECT BACKGROUND AND PARTNER ROLES ............................................. 75 ANNEX III: CORE INTERVIEW QUESTIONNAIRE............................................................................. 78 ANNEX IV: PUPIL QUESTIONNAIRE ................................................................................................... 81 ANNEX V: GENDER ANALYSIS ............................................................................................................ 84 ANNEX VI: DESCRIPTION OF FINDINGS BY IR AND EVALUATION QUESTION ....................... 99 ANNEX VII: PAGE M&E SYSTEM – PROCESSES AND TOOLS ..................................................... 108 ANNEX VIII: REFERENCES ................................................................................................................. 109 ANNEX IX: EVIDENCE MATRIX......................................................................................................... 110 ANNEX X: USAID EVALUATION REPORT REVIEW PROCESS .................................................... 116 Tables Table 1: Sampled Districts and Schools ....................................................................................................... 3 Table 2: Respondents by Gender .................................................................................................................. 4 Table 3: PAGE Student Achievement Targets and Results per PAGE Indicators ........................................ 6 Final Evaluation of the USAID/Ghana PAGE Project Final Report ii Table 4: Performance Evaluation Reading Results ....................................................................................... 7 Table 5: Improvements in School Performance through Results-Oriented Planning ................................. 21 Final Evaluation of the USAID/Ghana PAGE Project Final Report iii ACRONYMS ADEOP Annual District Education Operational Plan AOTR Authorized Officer for Technical Review BECE Basic Education Certification Exam CARE CARE International COP Chief of Party CPC Community Participation Coordinator CRDD Curriculum, Research and Development Department, GES CS Circuit Supervisor, DEO CSO Civil Society Organization DA District Assembly DCOP Deputy Chief of Party DDE District Director of Education DEO District Education Office DEOC District Education Oversight Committee DGEO District Girls’ Education Officers DFID Department for International Development (UK) ECE Early Childhood Education EGRA Early Grade Reading Assessment EMIS Education Management Information System EQUALL Education Quality for All ESP Education Sector Plan GAIT II Government Accountability Improves Trust II GEO Girl Education Officer GES Ghana Education Service GoG Government of Ghana GPEG Ghana Primary Education Grant (World Bank) GRAIL Grants and Reporting Accounts to Improve Literacy HRMD Human Resource Management Department, GES HT Head Teacher IEC Information, Education, Communication INSET In-Service Education and Training IR Intermediate Result JHS Junior High School JPPM Joint Partnership and Planning Meeting KG Kindergarten M&E Monitoring and Evaluation MDE Municipal Director of Education MOE Ministry of Education NALAP National Language Acceleration Program NEA National Education Assessment NSC National Steering Committee Final Evaluation of the USAID/Ghana PAGE Project Final Report iv PAGE Partnership for Accountable Governance in Education PMP Performance Monitoring Plan QUIPS Quality Improvement in Primary Schools SBA School-Based Assessment SI Social Impact, Inc. S&L Supply and Logistics, GES SKII Semi-Structured Key Informant Interview SMC/PTA School Management Committee/Parent Teacher Association SPAM School Performance Appraisal Meeting SPIP School Performance Improvement Plan SRC School Report Card THM Town Hall Meetings TLM Teaching and Learning Materials UNICEF United Nations Children’s Fund USAID United States Agency for International Development Final Evaluation of the USAID/Ghana PAGE Project Final Report v EXECUTIVE SUMMARY Project Background and Purpose This report documents the final performance evaluation conducted by Social Impact, Inc. (SI) of the $8,250,363 USAID/Ghana Partnership for Accountable Governance in Education (PAGE) project. The evaluation took place between December 12, 2013 and March 31, 2014. The PAGE project is implemented by CARE through a collaborative partnership with the Ghana Education Service (GES) and USAID/Ghana. The goal of the PAGE project is to improve student achievement in basic schools through strengthened educational governance and supervision. The three-year project received a one-year no-cost extension resulting in a revised project period of performance of June 2010 to June 2014. The purpose of the evaluation is to assess the extent to which PAGE interventions have contributed towards achieving USAID and GES development objectives related to strengthened governance, improved quality of basic education, and student achievement. The evaluation will provide USAID, its implementing partners, and education stakeholders with data on outcomes, best practices, and lessons learned to inform future education projects. Evaluation Questions, Design, and Methods The evaluation responds to five critical questions posed by USAID. 1. Have governance and supervision interventions at the district level resulted in improved student achievement? If so, to what extent and why? If not, why not? 2. What intended and unintended contributions, results, and/or impact have the PAGE approach and activities achieved relative to improving governance and supervision in schools? How? 3. What factors affected the achievements and results of the project? What can be identified as lessons learned and best practices and from which stakeholders or beneficiaries? 4. What are the best ways to ensure that progress and results are captured and/or continued that also promote ownership, engagement, and sustainability of interventions after it has ended? 5. What performance monitoring processes, systems, and tools were used to ensure accurate, timely, reliable, and valid performance and indicator tracking, reporting, and feedback at all levels? How did this contribute to logical implementation of activities, data validation, documentation, and review of work plans, indicators, and/or activities? Data Collection Methods The team used a mixed methods approach to complete a thorough and effective performance evaluation within the timeline and budget parameters set in the contract. Through document review, in-depth semi￾structured key informant interviews, focus groups, and pupil reading assessments, the evaluation methods yielded both quantitative and qualitative data about the performance of the PAGE project. The team applied robust data analysis techniques including triangulation, contribution analysis, and a gender matrix to draw conclusions and recommendations linked to key findings. The team compared data collected in the field with results published by the most recent Annual Review of PAGE program results and the updated targets for Year 4. Using a stratified sampling design with support from USAID, 11 districts were selected from high-performing, average-performing, and low-performing project sites. Findings, Conclusions, and Recommendations The following summarized findings, conclusions, and recommendations are discussed in the main report. Evaluation Question 1: Effect on Student Achievement Question 1 Findings Stakeholders interviewed in nine of the 11 districts reported that student achievement has improved in primary school. The majority relied on perceptions but could not quantify acknowledging there is no Final Evaluation of the USAID/Ghana PAGE Project Final Report vi standard school-based assessment system in place to track reading progress and report actual figures. The perceived increase was based on proxy measures such as on-the-spot reading assessments, improved rankings in reading competitions, and Basic Education Certification Exam (BECE) results, with no specific reference to early grade reading. This was not a surprise since PAGE did not have any direct classroom interventions and the target set from baseline to endline was a percentage increase of two points, moving from 46% to 48% for the Ghanaian language and from 51% to 53% for English. Based on an informal reading assessment conducted in seven regions by the evaluation team, the results showed 50% of those tested in the Ghanaian language and 51% tested in English met the PAGE performance indicators. Of those who demonstrated progress toward achieving literacy, 24% read fluently in both languages and 26.5% read with difficulty. Nearly 50% of those tested in the Ghanaian language and 32% tested in English could not read at all. Evaluators observed that pupils who read fluently were also those who had homework assistance from parents or siblings. According to teachers interviewed, the greatest obstacles affecting performance were overcrowded classrooms, inadequate supply of stationery, and a lack of supplementary teaching and learning materials (TLMs). Question 1 Conclusions PAGE has exceeded its target by two percentage points for improving student achievement in the Ghanaian language but has shown zero improvement in English compared to the 2010 baseline assessment results. Achievement is constrained by challenges related to overcrowded classrooms, teaching quality and shortages, and insufficient materials. There is an absence of a formal early grade reading assessment to monitor reading progress at the school level against baseline and target indicators. The BECE has been used as a proxy measure of achievement in primary schools, and some evidence suggests PAGE influence on improved results, but a more accurate measure is necessary to monitor pupil achievement against predefined benchmarks. Question 1 Recommendations 1. USAID/GES should include direct teaching and learning interventions in current or upcoming projects to increase the capacity of teachers to effectively deliver language lessons in the early grades. 2. USAID and District Education Offices (DEO) should provide resources and strategies to support effective classroom management 3. Recognizing that USAID has recently implemented a large-scale EGRA with a nationally representative sample of 8,000 students and has established a baseline, USAID should now consider dovetailing this initiative with school-level and district-level EGRAs. Evaluation Question 2: Improving Governance And Supervision In Schools Question 2 Findings As a result of PAGE activities, educational stakeholders have increased awareness of their roles and responsibilities, have a shared vision for improved school performance, and are collaborating in the monitoring and supervision of schools. The District Education Office Committee (DEOC) is now functional and visiting schools more regularly, the Circuit Supervisors (CS) have been resourced to do their work effectively, and School Management Committees/Parent Teacher Associations (SMC/PTA) are closely monitoring schools and assisting schools to solve challenges. Teacher attendance has improved as a result of increased monitoring, supervision, and accountability. The team found exceptional examples of community contributions to schools and improved governance; yet, none of the findings showed unintended contributions, results or impact beyond the scope of the PAGE project. Question 2 Conclusions The PAGE project contributed to the empowerment of the SMC/PTA to work collaboratively with the DEO in school governance and supervision. DEOC/DEO and CS monitoring has been highly effective in improving teacher attendance and increasing accountability, but effective monitoring strategies, such as use of work plans, checklists, and performance indicators, are not being systematically implemented Final Evaluation of the USAID/Ghana PAGE Project Final Report vii across all districts. CSs and Head Teachers (HT) have not yet fully exercised their pedagogical and managerial roles to improve teacher performance and student achievement. Question 2 Recommendations 1. USAID and GES should continue supporting SMC/PTAs to effectively carry out their roles. 2. The GES should standardize reporting structure for monitoring visits. 3. USAID and GES should increase the capacities of CSs and HTs to exercise their pedagogical and managerial roles. Evaluation Question 3: Lessons Learned and Best Practices Question 3 Findings Training and capacity building of education stakeholders on management and monitoring roles and responsibilities, particularly the SMC/PTA, contributed greatly to achievement of outcomes; however, there were numerous challenges at the national, district, and school levels. At the national level, the delay in setting up the funding mechanism, late release of funds, and inaccurate record keeping by the DEO, which caused some districts to repeat activities, contributed to a slow start-up and lag time in meeting targets. At the district level, the most significant challenge was the poor quality of motorbikes supplied by the PAGE project, resulting in frequent breakdowns, high fuel consumption, and costly repairs that exceeded the maintenance allowance. The project also struggled with obtaining commitments from District Assemblies (DA) to achieve indicator targets intended to sustain monitoring visits. Although PAGE did not work directly at the school level, challenges that hampered student achievement include teacher absenteeism, insufficient TLMs, overcrowded classrooms, lack of school-based assessment tools to track student performance, and insufficient parental supervision and support for children’s education. HTs have limited capacity to effectively use the School Performance Improvement Plan (SPIP) and capitation grant to address resource constraints. Best practices at the national level include rewarding good district performance with an extra motorbike, pooling resources so that all CSs benefit, and training multiple GES staff at all levels within the GES as a way to mitigate effects of high turnover. Question 3 Conclusions Persistent support and engagement of multiple GES staff is required for continuity and sustainability. Having appropriate financial mechanisms, documentation, and implementation guides in place are critical for adhering to project schedules. According to teachers’ interviewed, teacher absenteeism and poor performance is largely caused by inadequate government responsiveness to address issues. . HTs and teachers require greater competence and awareness in utilizing the capitation grant, SPIP and other channels (e.g., process for lobbying the DA, private sector) for accessing government and donor support. Provision of durable motorbikes and sufficient fuel allowance is critical for the effectiveness of the CSs to carry out their monitoring and supervision roles. Rewarding good district performance, training multiple GES staff, and providing continuous capacity building, and equally distributing resources have improved the efficiency and internal capacity of the system within the GES and benefited other donors. Best practices for improving student performance, particularly in reading, include use of TLMs, reading competitions, use of reading progress charts in classrooms, intensified training and support provided to low-performing schools, and encouraging the inclusion of reading materials and other supportive activities in the SPIP. Incentives, stringent promotion practices, sanctions, and re-assigning teachers based on capabilities have been effective practices for improving teacher performance. To increase the quality and outcomes of CS monitoring visits, best practices include CS monitoring work plans, CS monitoring checklists, CS review meetings, and provision of fuel allowances contingent on submission of CS monitoring reports. Regular unannounced visits by the DEOC and DEO increase accountability and effectiveness of HTs and CS. Use of radio messaging and community drama to increase awareness of the parents’ role in education and training illiterate parents on how to interpret marks were in engaging parents to monitor pupil performance. Final Evaluation of the USAID/Ghana PAGE Project Final Report viii Question 3 Recommendations 1. The GES and USAID should incorporate best practices for improving student performance, teacher performance, CS monitoring, and parental participation from the PAGE project in future programming. 2. The GES and USAID should provide HTs with training on how to maximize the use of the capitation grants, SPIP, and other sources of funding to address school resource constraints, such as increasing the availability of TLMs. Evaluation Question 4: Ownership, Engagement, and Sustainability Question 4 Findings Even though there is increased interest and use of performance data for decision making at the school, community and DEO levels, the enthusiasm might dissipate if resources are not made available for joint partnership planning meetings, monitoring visits, the organization of School Performance Appraisal Meetings (SPAM) (at the community level), town hall meetings (at the DEO level), and other activities where these results are utilized. During interviews, District Chief Executives in many districts expressed a commitment to sustaining the PAGE project. However, given the over-dependence of the districts on the DA and funding challenges at the central level of government, this could serve as a limiting factor in ensuring sustainability of PAGE activities. Question 4 Conclusion The best way to ensure that progress and results are captured and/or continued in a manner that also promotes ownership, engagement, and sustainability of interventions is to develop an exit strategy with the GES. Question 4 Recommendation 1. USAID should work with PAGE before it closes to develop an exit strategy that identifies specific activities to prepare the DEO/DEOC for raising funds from public, private or international sources. Evaluation Question 5: Performance Monitoring Processes, Systems, and Tools Question 5 Findings There is evidence that PAGE provided training and technical support to GES finance, M&E, and frontline officers on work plan development, accounting, data collection, and reporting and grants management. PAGE also provided guidelines, report templates, motorbikes, and community volunteers to assist with data collection. As a result of PAGE assistance, the quality of activity and financial reports has improved, benefiting all donors. Transportation funds and capacity building activities from the PAGE project also mobilized GES staff to visit schools and collect data. PAGE also contributed to the logical implementation of activities through data validation, review of work plans/indicators with key DEO stakeholders through joint planning meetings and the development of plans at the district level, but the dissemination and feedback loops at the school level are still in their nascent stages. SMC/PTA level of awareness of the School Report Card (SRC) and involvement in developing the SPIP is limited. Question 5 Conclusions Systems and tools have been instituted in the districts the evaluation team visited, but evidence shows that dissemination and feedback processes at the school level are slowly being initiated. SMC/PTAs are not fully engaged in tracking school and pupil performance. The SRC is still relatively new and not yet fully institutionalized at all levels of the system. Question 5 Recommendations 1. The GES and USAID should continue building capacity with specific processes, systems and tools that enhance the function of the M&E system within the DEO. USAID should offer training on the interpretation and utilization of the SRC so that it is fully integrated at all levels of decision-making. Final Evaluation of the USAID/Ghana PAGE Project Final Report 1 INTRODUCTION Project Background Social Impact, Inc. (SI) was contracted to undertake a final project performance evaluation of the $8,250,363 USAID/Ghana Partnership for Accountable Governance in Education (PAGE) project (an original Scope of Work can be found in Annex I, Part A). PAGE was initiated in 2010 through a collaborative partnership between the Ghana Education Service (GES), USAID/Ghana, and CARE; the goal of the project is to improve student achievement in basic schools through strengthened educational governance and supervision. The development hypothesis is that strengthened systems of governance and accountability will lead to better teaching practices, better learning outcomes, and higher achievement at the primary level, especially in acquisition of literacy skills in the early grades. Achieving the project goal demands a highly collaborative approach to governance and invites the participation of multiple levels of school and district management, with specific roles and responsibilities (see Annex II). The three-year project received a one-year no-cost extension resulting in a revised project period of June 2010 to June 2014. EVALUATION PURPOSE AND QUESTIONS Evaluation Purpose The purpose of the evaluation was to assess the extent to which PAGE interventions have contributed to achieving USAID result indicators related to improved educational governance, quality of education, and student achievement, which contribute to USAID/Ghana’s Assistance Objective 8 and Ghana’s Education Strategic Plan (ESP). The evaluation also provides USAID, its implementing partners, and education stakeholders with best practices and lessons learned generated over the life of the PAGE project to inform future education projects. The final project evaluation sought to respond to five USAID evaluation questions intended to measure the contributions of PAGE interventions towards strengthened governance, accountability, and student achievement. The evaluation data collection took place in seven regions of Ghana from January 20 through February 12, 2014.1 PAGE Project Components PAGE activities support two of the four ESP objectives as laid out in the Medium-Term Development Plan Objectives for Education, namely, improved quality of teaching and learning and improved management of education service delivery. PAGE also supports USAID/Ghana’s Assistance Objective 8 (Improved Quality of, and Access to, Basic Education) through two Intermediate Results (IR) indicators (IR2—increased quality of education and IR3—improved educational governance). PAGE works through four components, which contribute toward achieving improved governance and accountability in education and improved student performance in English and Ghanaian languages. These are: Component One, Stakeholder outreach; Component Two, DEO-DEOC collaboration and development of district-specific frameworks for education governance and supervision; Component 1As per the Evaluation Conflict of Interest Form, all members of the SI evaluation team met the conditions of compliance. The forms indicated no conflict of interest with businesses or relationships with USAID technical staff that would influence the oversight of project implementation or the process of procurement of goods or services for USAID. Final Evaluation of the USAID/Ghana PAGE Project Final Report 2 Three: Improve school supervision through training Circuit Supervisors and Head Teachers; and Component Four: Capacity building for SMCs and PTAs. Evaluation Questions The final project evaluation responds to five evaluation questions posed by USAID: 1. Have governance and supervision interventions at the district level resulted in improved student achievement? If so, to what extent and why? If not, why not? 2. What intended and unintended contributions, results, and/or impact have the PAGE approach and activities achieved relative to improving governance and supervision in schools? How? 3. What factors affected the achievements and results of the project? What can be identified as lessons learned and best practices and from which stakeholders or beneficiaries? 4. What are the best ways to ensure that progress and results are captured and/or continued that also promote ownership, engagement, and sustainability of interventions after it has ended? 5. What performance monitoring processes, systems, and tools were used to ensure accurate, timely, reliable, and valid performance and indicator tracking, reporting, and feedback at all levels? How did this contribute to logical implementation of activities, data validation, documentation, and review of work plans, indicators, and /or activities? EVALUATION METHODS AND LIMITATIONS Methodology The team used a variety of mixed methods to complete a thorough and effective performance evaluation within the timeline and budget parameters as set out in the contract. Through document review, in-depth semi-structured key informant interviews (SKII), focus groups, and pupil reading assessments, the evaluation methods yielded both quantitative and qualitative data about the performance of the PAGE project. These data collection methods are described in greater detail below. The approved work plan (Annex I) presents the team’s initial methodology and approaches to data collection, analysis, and plan to mitigate anticipated challenges and limitations. Document Review Starting with a review of project documents, the team reviewed PAGE semi-annual reports, case studies, the Performance Monitoring Plan (PMP), past assessments, and other relevant documents to inform instrument design and data analysis. Instrument Development During the preparation phase, the team identified target groups for key informant interviews and focus group discussions and developed a semi-structured core questionnaire with key questions corresponding to each USAID evaluation question, which allowed for a level of standardization across interviews. Using the core questionnaire, the team adapted the instrument to suit each target group at the national, district, and school levels. The questionnaires were finalized after piloting the instrument with district and school representatives in Ga East district with the exception of the SMC/PTA because of unavailability. A pupil protocol guide was also developed for conducting interviews with pupils in P1, P2, and P3, and administering the reading assessment. A copy of the core SKII is located in Annex III and the pupil protocol guide is in Annex IV. Pupil Reading Assessment The evaluation team administered an informal reading assessment using the similar procedures and levels often used by the DEO, but with less rigorous protocols. In each of the sample schools, the team randomly selected one boy and one girl from P1, P2, and P3, and asked them to read a familiar passage from a corresponding grade-level textbook. To further assess whether pupils were simply reciting a memorized passage or decoding words, they were asked to read an unfamiliar text from the Final Evaluation of the USAID/Ghana PAGE Project Final Report 3 back of the book. The pupils’ scores were grouped into three levels: 1) Reading fluently—if they read without difficulty and at the pace of a child of his/her age; 2) Reading with difficulty—if the child reads with hesitation and some degree of difficulty; or 3) Cannot read at all. The reading assessment was piloted with one girl in P3 in Ga East. Sampling With inputs from USAID, the team applied a stratified sampling method to select a sample of seven regions across all 10 regions in which PAGE project operates. Eleven districts and schools were randomly selected from high-performing, average-performing, and low-performing strata. Table 1 lists each region, district, and school the evaluation team visited during data collection activities. Given the extensive geographic scope of PAGE, the team divided into two groups, each consisting of an expatriate and Ghanaian evaluator. One team traveled north and covered the Northern, Upper East, and Brong Ahafo regions, while the southern team covered the Central, Volta and Eastern regions. Table 1: Sampled Districts and Schools Region District Schools 1. Greater Accra 1. Ga East 1. Abokobi Presbyterian KG/Primary School 2. Volta 2. Akatsi 2. Avenorpeme D/A R/C Basic School? 3. Ketu North 3. Penyi Anglican Basic School 3. Central 4. Abura Asebu Kwamankese 4. Abura Dunkwa Methodist Basic School 5. Mfantseman 5. Saltpond Catholic Boys Primary 4. Eastern 6. Asuogyaman 6. Atmpoku RC Primary School 5. Upper East 7. Bawku West 7. Sakom Primary School 8. Bongo 8. Gowrie Tingre KG/Primary School 6. Northern 9. Tolon 9. Chirifoyili Primary School 10. Savelugu 10. Ansuari Suna Primary School 7. Brong Ahafo 11. Nkoranza 11. Beboano Basic School Source: Fieldwork Data Data Collection Data collection took place from January 20 through February 12, 2014. Immediately following arrival in country, the team held an in-briefing meeting with USAID to review the evaluation work plan and gather inputs for finalizing the evaluation design. During the first week in Accra, the team met with national stakeholders to gain an in-depth understanding of the project and gather additional insights that would inform logistical preparation for fieldwork. Meetings were held with PAGE key personnel (Chief of Party/Deputy Chief of Party) and GES accountants responsible for disbursing PAGE grants. At the end of the first week, the two teams conducted a joint site visit to Ga East district in the Greater Accra region, which served both as an orientation to the project at the district level and as a field test of protocols and methodology to assess the timing and ease of administering the tools. The instruments were finalized and printed. Each team departed for the regions on January 28. Fieldwork was expected to be completed by February 8; it was extended with USAID approval by several days to account for extensive travel between districts in the Northern region and Saturday meetings planned with the community that had to be postponed until the following week. In each district visited, the teams met with district- and school-level representatives. At the district level, the team held in-depth interviews with the District Director of Education (DDE) followed by Final Evaluation of the USAID/Ghana PAGE Project Final Report 4 focus groups with CS, DEOC, and DEO support staff (M&E, Finance, Girls’ Education, Community Participation). During school visits, each team held key informant interviews with the HT, assisted by the Assistant HT, and with community volunteers (CV). Focus groups took place with the SMC/PTA, teachers, and pupils. The teams also met with a few Regional Master Trainers and CARE Program Officers. A total of 115 meetings were conducted with 578 respondents. As shown in Table 2, which provides a list of each stakeholder group included in data collection activities, 44% of respondents were females and 56% were males. Table 2 should serve as a reference for sources of percentages presented throughout the report. Data Analysis Upon completion of fieldwork, the team reunited in Accra to perform data entry, cleaning, and consolidation. Once the data were tabulated by stakeholder, the team analyzed the findings to identify trends and themes in response to USAID questions. The team triangulated data across multiple stakeholders and districts and verified data in PAGE documents. Data were disaggregated by sex to pay particular attention to equality gaps between male and female project participants and beneficiaries. The team developed frequency tables to share key findings during USAID and CARE debriefing meetings. The debriefing presentations served as an opportunity to validate findings and seek clarification. Following the debriefing, further data analysis was conducted to respond to questions raised by USAID and the GES. Additionally, the team performed a gender analysis (see Annex V) using an evidence matrix and gender findings table that SI adapted from two USAID publications.2 Limitations The small sample size and non-experimental approach to the evaluation design limits the extent to which causal relationships and direct attributions can be drawn from the PAGE project. However, through use of mixed methods and triangulation, the team was able to identify contributions of the PAGE project to improved governance and supervision. The team compared evaluation results with monitoring data and targets to verify achievements to date. The qualitative data supported contribution analysis. It is much more difficult to analyze direct contributions to student achievement since PAGE did not have any direct classroom interventions and endline data were not available 2 The matrix and findings tables are derived from Deborah Caro (2009) A Manual for Integrating Gender into Reproductive Health and HIV Programs: From Commitment to Action. USAID/Interagency. USAID Interagency Gender Working Group, p. 22 and 27 Table 2: Respondents By Gender Source: Fieldwork Data Target Group Females Males No. % No. % Total CARE (PAGE) 5 45% 6 55% 11 National GES 4 27% 11 73% 15 Regional master trainers 3 100% 0 0% 3 DDE 6 55% 5 45% 11 DEOC 24 34% 47 66% 71 DEO support staff 27 24% 85 76% 112 Circuit Supervisors 12 16% 63 84% 75 Head Teachers 6 67% 3 33% 9 Teachers 38 60% 25 40% 63 Pupils 25 51% 26 49% 51 SMC/PTA 80 56% 63 44% 143 Community Volunteer 0 0% 4 100% 4 Grand Total 236 44% 342 56% 578 Final Evaluation of the USAID/Ghana PAGE Project Final Report 5 during the evaluation. As an indicative measure of PAGE contributions to reading achievement and as an approach to mitigating this limitation, which is not intended to replace the endline or other standardized formal assessments, the team collected primary data through a reading assessment in order to compare results against baseline and target indicators3 . Another limitation is the sample size of the reading assessment. A handful of students from 11 schools were tested, out of a much larger number of schools that received interventions through PAGE. To reduce selection bias, the districts, schools, and students were randomly selected by grade and gender. The team experienced several limitations during fieldwork that were mitigated early on after the first instance and did not jeopardize the validity, reliability or objectivity of the evaluation. First, there was not sufficient time to thoroughly revise the questionnaire prior to fieldwork. As a result, some of the questions were repetitive and the interview took longer than expected. In the end, the team found that asking the same question in multiple forms actually helped to obtain richer data that could be easily consolidated. Second, in the Northern and Upper East regions, the team encountered language barriers when interviewing the SMC/PTA in the first one or two schools and GES or school officials assisted with interpretation, which could have biased results, but the local Ghanaian evaluator was able to decipher the local dialect and intervene by re-phrasing the question and obtained an objective response. In subsequent schools, the team proactively hired professional interpreters to mitigate this risk. Regarding response bias, in two instances in the northern districts, male SMC/PTA chairpersons and DEOC members dominated the discussions. The team attempted to overcome this by calling on each person equally to answer the question. Another challenge faced by the team was recall bias. Some stakeholders interviewed could not recall the number of trainings or meetings they had attended. Through further probing, respondents were able to explain topics discussed in past trainings and how they have applied what they had learned to improve governance and supervision. Therefore, the anticipated challenge of recall bias did not pose a major problem that prevented the team from collecting relevant data. Additionally, engaging assistants of HTs and DDEs (which tend to have high turnover), the team was able to further mitigate this threat and gain additional insight into the history of the PAGE project. Selection bias is always a potential problem in performance evaluations that rely primarily on key informants. To address this, the team included multiple stakeholders and team members in each interview. This approach worked quite well in obtaining accurate, objective and thorough responses to the evaluation questions. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Evaluation Question 1: Effect on Student Achievement Question 1 Findings According to the PAGE project indicators, improved student achievement is measured by the percentage of pupils demonstrating progress toward achieving literacy in the Ghanaian language or English. Per levels defined in the Education Quality for All (EQUALL)/NALAP assessment, pupils 3 Similar to on-the-spot reading assessments conducted by the Ministry of Education (MOE), the evaluation test did not measure reading fluency per an established benchmark (e.g., words per minute) or comprehension. Similar to the National Language Acceleration Program (NALAP), it is also does not assess achievement of literacy but rather progress towards achieving literacy based on clearly defined levels of reading as observed by the evaluator and defined in the methodology section. Final Evaluation of the USAID/Ghana PAGE Project Final Report 6 are considered demonstrating progress toward achieving language literacy if they score at “transitioning”4 or “developing” levels. The EQUALL/NALAP assessment tool was administered during the baseline assessment introduced by DevTech in November 2010. Table 3 shows results indicating that 46% of pupils tested in the Ghanaian language and 51% of pupils tested in English met the performance indicator. Because PAGE did not have a direct teaching intervention, the project set a low target, moving from 46% to 48% for the Ghanaian language and from 51% to 53% for English. As the targeted percentage increase was only two percentage points between baseline and endline, student performance was not measured on an annual basis but planned for the endline assessment in April 2014. Table 3: PAGE Student Achievement Targets and Results per PAGE Indicators Indicator Baseline Value Target Reading Assessment Results Sample % Indicator 2.1.1: % of P1–3 pupils who demonstrate progress toward achieving Ghanaian language literacy 46% 48% 29 50% Indicator 2.1.2: % of P1–3 pupils who demonstrate progress toward achieving English language literacy 51% 53% 51 51% *PAGE Performance Monitoring Plan (Feb. 2011) For purposes of analyzing data from the performance evaluation against the baseline to provide an indicative measure of improved student achievement to date, the team categorized findings into three levels: reading fluently, reading with difficulty, or cannot read at all. The performance evaluation results showed 51% of 29 students who were tested in the Ghanaian language and 50% of 51 students who were tested in English achieved the indicator. When compared against baseline values, pupils gained four percentage points in the Ghanaian language but showed zero improvement in English. Table 4 shows the breakdown of the performance evaluation reading results administered by the team. Of the roughly 50% of pupils tested who met the indicator, 24% read fluently in both languages and 26.5% read with difficulty. Nearly 50% percent of pupils tested in the Ghanaian language and 32% tested in English could not read at all. A detailed analysis of the findings for each language is provided on the next page. 4 Gender Working Group, p. 22 and 27. Table 4: Performance Evaluation Reading Results Reading Assessment Ghanaian Results (n = 29) English Results (n = 51) No. % No. % Pupils reading fluently 7 24% 15 24% Pupils reading with difficulty 8 27% 16 26% Pupils who cannot read at all 14 48% 20 32% Performance Evaluation Reading Assessment (Feb. 2014) Final Evaluation of the USAID/Ghana PAGE Project Final Report 7 Ghanaian Language Reading Test Findings The local language reading assessment was administered in three languages: in Twi in Greater Accra and the Eastern region, in Ewe in the Volta region, and in Fante in the Central region. As shown in Table 4, of 29 pupils (14 girls and 15 boys) tested 15 students (eight boys and seven girls) demonstrated progress toward achieving Ghanaian language literacy. Seven pupils (three boys and four girls) read a Ghanaian language book with fluency and eight pupils (five boys and three girls) read with difficulty. Conversely, 48% of pupils tested (seven boys and seven girls) could not read at all. Boys and girls performed similarly across all categories. English Reading Test Findings The English language reading assessment was carried out in all regions. Similar to the local language test results, 50% of pupils tested by the team achieved the indicator for English. This is equal to the baseline value and two percentage points below the target, indicating that PAGE has not achieved results in English. Of 51 pupils tested, 31 (18 boys and 13 girls) showed progress toward achieving English language literacy. Fifteen pupils (six boys and nine girls) read an English language book with fluency and 16 pupils (seven boys and nine girls) read with difficulty. The remaining 20 pupils (13 boys and seven girls) could not read at all. Girls performed better than boys at all levels on the English Reading Test. Evaluator Observations The team recorded observations during the testing. They found that children’s performance on English reading exams might be reflective of the incorrect usage of English in the classroom and misguided teaching methods. A P3 girl in Ga East, for example, stated that her sister taught her to read English through “chorus reading and repeating words read aloud” (e.g. memorization), which she had likely learned from her teacher. As a result, the girl was able to read familiar words through memorization and sight recognition, but lacked decoding skills necessary for fluency and comprehension. This case was commonly found throughout the evaluation. Pupils in other schools visited could recite familiar text but froze when asked to read a new paragraph or a separate line. In one basic school visited, the HT proudly stated that reading had improved since teachers had been trained to teach key words prior to reading a passage aloud. This may explain why many pupils could read familiar words, but could not point to the words as they read indicating they were reciting rather than reading text. Apart from the reading assessment, the team gathered qualitative data on student achievement through SKIIs with pupils, teachers, HTs, SMC/PTA representatives, and DEOs to identify perceptions and factors contributing to or hindering student achievement. Interviews with 51 pupils revealed that students in eight schools had siblings who assisted them with homework/reading. Students reported parents assisted them with reading in only four schools. Two students, each in a separate school, were encouraged to speak English at home and in only one school was there a student whose parents purchased books. The evaluation team observed that pupils who read relatively fluently were also those who had siblings or parents who assisted them with homework. Over 80% of respondents (in nine of 11 schools visited) stated that student achievement has improved but not as much as they had expected. Based on focus groups discussions with teachers in 11 schools, the greatest obstacles to student achievement are overcrowded classrooms (seven schools), inadequate supply of stationery (five schools), and lack of supplementary readers and TLMs (four schools). The majority of HTs and SMC/PTA interviewed frequently stated that parents’ failure to supervise their children, encourage regular attendance, and monitor children’s performance, due to illiteracy and poverty factors, were major barriers. DEO staff interviewed believe challenges impeding reading achievement are chiefly related to the following: a high proportion of untrained teachers in primary and kindergarten (KG); high attrition of NALAP-trained teachers; lack of foundational skills due to not having attended KG or delayed enrollment; and inadequate use and availability of NALAP materials and insufficient quantities of English textbooks. Additionally, the implementation of NALAP has been challenged in Ga East, Bongo, Bawku West, and other districts in which the language most widely spoken differs from the approved NALAP language for the specified area/district. Final Evaluation of the USAID/Ghana PAGE Project Final Report 8 In the nine districts that reported improvements in student achievement, respondents offered several examples as their basis for measurement. The first was observations. When DDE and DEOC members conduct monitoring visits to assess literacy, they ask random students to read a passage from a grade￾level text. They stated they have noticed improvements but could not quantify these observation, acknowledging there is no baseline or targets to track individual achievement. One CV stated that he has observed reduced repetition in the early grades and improved reading skills in KG. Another CV claimed that students in P4, P5, and P6 are speaking English more fluently while the lower primary students have improved their oral comprehension skills. Secondly, improved rankings in reading competitions, mostly for upper primary grades, were mentioned in several districts. The M&E Officer in Nkoranza claimed a girl from a remote village (Dompose) located in the hinterland area won the reading competition. Filled with astonishment, he decided to visit the school and tested 15 out of 20 students in the class, asking them to read a passage out loud. According to his on-the-spot assessment, 15 were able to read well. Finally, another frequent response provided by DDEs, HTs, and DEO members interviewed was an increase in the number of pupils passing the BECE. In fact, PAGE/GES have been using the BECE as a proxy for achievement in primary school because they claim there is no school or district-level assessment system in place to measure early grade reading and literacy. This means that performance is being assessed and generalized for all students on the basis of junior high school (JHS) results. DEO staff and HTs, however, insist that PAGE has had a direct impact on BECE results at the JHS level. Some HTs trained by the PAGE project also serve as HTs for JHS. They have therefore implemented the same best practices as promoted by PAGE. Respondents asserted that the monitoring has put all teachers “on their toes” and SPAM meetings in which the BECE results are presented have engaged the community in improving education in basic schools. As a result of these efforts, some districts have seen a major improvement in BECE results. In one school visited in Bawku West, BECE results jumped from 21% in 2011 to 42% in 2012. Nkoranza district advanced from the lowest ranking school in the region in 2010 (53% passing BECE) to the highest in 2013 (with over 90 percent of JHS students passing the BECE). In Savelugu district, however, a DEO representative stated, “We can’t say PAGE has directly contributed to achievement. It would be untrue to say ‘yes’. If we look only at test scores, they have not improved. If we look at other performance indicators (e.g., attendance), it has improved.” The team observed factors that have contributed to the above results. Several districts that have shown high achievements have concentrated their monitoring visits and activities in the worst-performing schools. In Savelugu district, although they use the same approach, they have also increased transparency on the BECE exam. The DDE stated that it is a common practice for schools and parents to only register the best performing pupils for the BECE exam, or to allow older sibling or friends to sit for the exam to ensure successful pass rates. The DDE informed all schools that this is no longer allowed and is requiring all pupils to be registered for the exam. He did not state specific measures in place to avoid cheating the system. Further investigation is needed. However, this suggests that low-test scores in Savelugu district, which is a high-performing district, may be reflective of a larger student population included in the BECE sample and the real situation with regards to reading performance at the JHS level. Question 1 Conclusions 1. PAGE has exceeded its target by two percentage points for improving student achievement in the Ghanaian language but has shown zero improvement in English since the 2010 baseline assessment. 2. Teachers and schools face a number of challenges related to high enrollment, teacher quality and shortages, insufficient reading materials, and lack of parental support that hinder the achievement of results. 3. There is no formal early grade reading assessment to monitor achievement at the school level against baseline and target indicators. While the National Education Assessment (NEA) and the newly established USAID EGRA/EGMA measure reading outcomes on a representative sample of schools at the national level, stakeholders interviewed at the district and school levels indicated that there is not a school-based assessment system in place to support teachers and DEO administrators with monitoring pupil achievement against district and school benchmarks. As a Final Evaluation of the USAID/Ghana PAGE Project Final Report 9 result, they have been using the BECE as a proxy measure of achievement for primary and junior high school. 4. The BECE achievements in PAGE-supported schools are anecdotal. It is not possible to draw a causal link between the PAGE project and achievement on the BECE results due to extraneous factors affecting results. However, there is some evidence suggesting that intensified monitoring of schools may have positively affected JHS performance. Additionally, given the evidence that monitoring visits focused on a range of activities, it was not possible to draw a link between governance and student performance, other than increased enrolment and improved attendance. Question 1 Recommendations 1. In current or upcoming projects, USAID/GES should include direct teaching and learning interventions to increase capacity of teachers to effectively deliver language lessons, particularly in the early grades. Activities may include teacher training on phonics/reading methodology, distribution of TLMs and School-Based Assessment (SBA) booklets, EGRA or SRC training for continuous assessment, and building the capacity of HTs and supervisors to serve as pedagogical coaches. Building on the successful approaches and recommendations of some PAGE-supported schools, USAID should consider engaging teacher training colleges in early childhood education to provide training on phonics and promoting reading competitions and reading clubs. 2. Within the medium term, USAID should consider working with DEO to improve availability and sufficient use of resources in the classrooms so that teachers can teach effectively and engage parents in monitoring their children’s performance. For example, schools may establish libraries or resource centers for sharing supplementary readers and TLMs (which are currently stored in the HTs office and not easily accessible). Teachers may be trained in effective classroom management strategies and supported by volunteer teachers to overcome the challenges of overcrowded classrooms. Finally, launching a best practice from PAGE, teachers can train illiterate parents on how to interpret marks in exercise books, so that parents can check their child’s progress, inquire about lessons, and motivate good academic performance. 3. USAID has implemented a large-scale EGRA recently with a nationally representative sample of 8,000 students and has established a baseline, so it should now consider dovetailing this initiative with school-level and district-level EGRAs. For instance, teachers could be trained in conducting mini-EGRAs as a form of in-class reading assessment and administrators could be trained to conduct district-wide EGRAs with a representative sample of schools. Subsequently, the data could be shared and fed into the School Performance Improvement Plan, the School Report Card, School Performance Appraisal Meetings, and used for benchmark setting. The findings could also assist USAID with measuring student achievement on an annual basis in order to develop targeted interventions and collect evidence to assess project impact. Evaluation Question 2: Improving Governance and Supervision in Schools Question 2 Findings Stakeholder groups interviewed at the national level (GES Finance), district level (DEOC members, the DDE, GEO, CPC, M&E, and CS), school level (Head Teachers/Teachers), and community level (the SMC/PTA members and CVs) unanimously attest that PAGE has made a remarkable contribution to strengthening governance and supervision at the school level. One of the greatest contributions of the PAGE project is increased stakeholder awareness of their management role and responsibilities and mobilizing actors to conduct regular school monitoring visits. This is evidenced by the finding that PAGE has met or exceeded its target for increasing the percentage of stakeholders using effective management strategies. The project targets for stakeholders effectively supporting schools were 50% for DEOC, 55% for SMC/PTA, 85% for CS, and 50% for HT. Based on interviews with each stakeholder, the evaluation team found that 55% of DEOC members, 64% of SMC/PTAs, 90% of CSs, and 50% of HTs have been trained and are using effective management strategies as defined by the PAGE project (See Annex VI: Description of findings by IR and Evaluation Question). Final Evaluation of the USAID/Ghana PAGE Project Final Report 10 The team also obtained evidence of increased monitoring. CSs are conducting monitoring visits regularly in 10 of the 11 districts. DEOCs visited at least one school per term in eight districts and responded to issues in six districts. SMC/PTA visited the school at least once per term but on average nine times per term and in many cases once a week, and interacted with HTs. Teacher attendance has improved as a result of increased monitoring, supervision, and accountability. The team identified exceptional examples of community contributions to the school and exemplary districts demonstrating improved governance; however, none of the findings were outside the scope of the project or unintended. Additional examples of PAGE contributions to the strengthened role of DEOCs, CSs, and SMC/PTAs in school governance are discussed below. DEOC Prior to the PAGE training, many respondents stated that DEOC existed in name but was dormant. DEOC members visited schools once or twice a year, but were not functioning in their role to monitor schools and respond to demands raised by schools and communities. According to seven DEOC members interviewed in Bongo District, there existed some management strategies in the past; however, the PAGE project has reinforced implementation of the strategies by providing capacity building in terms of role clarity, funding, and logistics. As a result of PAGE contributions, over 50% are now using effective management strategies. The DEOC and DEO are collaborating to develop work plans and monitoring checklists and to conduct joint school monitoring visits. They are meeting regularly to discuss the outcomes of school visits and have responded to the basic needs of schools. With the last tranche of funding, DEOC members visited at least 13 schools within their circuit. Three districts used monitoring checklists, six produced reports, and three shared the reports with other stakeholders. Through increased monitoring visits, the DEOC has been able to assist with resolving land issues and infrastructural challenges (e.g., construction of classroom, furniture, sanitation facilities, and an access road). In Bongo district, the DA approved items (laptops, fridges, etc.) to reward teachers for good performance. Moreover, through unannounced monitoring visits, DEOCs and DDEs have observed performance issues such as teacher absenteeism, irregular monitoring of CS, and weak HTs. This has led districts to put measures in place to increase accountability through both punitive sanctions and incentives. For example, in Savelugu district, the DEO has placed stop payments on salaries of teachers who do not report to school; they have demoted non-performing HTs; and now require new HTs to pass a screening process and one-year probationary period prior to receiving tenure. Measures to increase CS accountability include requiring them to sign logbooks, conducting weekly CS review meetings, and allocating fuel allowance upon receipt of CS monitoring reports. Circuit Supervisors CSs have demonstrated the greatest improvement, moving from 4% making effective visits at baseline to an astonishing 90% achieving this indicator during the evaluation period. CS and DEO staff interviewed unanimously assert that PAGE provision of motorbikes, fuel and maintenance allowance has contributed to an increased number of monitoring visits by the CS. Performance evaluation results show that in 10 of the 11 districts sampled, CS visit each school in their circuit at least twice per term. Equally distributing the motorbikes, fuel allowance and schools within each circuit also expanded the reach of CS to visit schools more frequently, according to CS interviews. Some were able to visit every school within their circuit on a weekly basis. On average, each school was visited three times last term, as opposed to once per term in the past based on data analyzed from stakeholder interviews. In addition to increased frequency, the quality of monitoring visits has also improved in some districts. The CS now meets with the HT during each visit to discuss issues facing the school. Generally, issues discussed with HTs center on teacher and pupil attendance, effective teaching and learning, lesson notes preparation and delivery, challenges faced by teachers and solutions to address them, and challenges related to achieving SPIP targets. CS stated they normally do the following during school monitoring visits: vetting teachers’ lessons (eight districts), reviewing pupil exercises to assess the work output of teachers and pupils (six districts), delivering In-Service Education and Training (INSET) for teachers at least once per term in their circuit (four districts, one focused on reading); and interpreting education policy (five districts) for HTs and SMC/PTAs. Providing Final Evaluation of the USAID/Ghana PAGE Project Final Report 11 pedagogical support to assist teachers with lesson delivery was carried out in only a few districts. CSs mentioned visiting classes to observe teacher delivery of lessons, conducting demonstration lessons on challenging topics, and providing curricular advice to teachers and HTs in three districts; and guiding NALAP implementation in two districts. Unfortunately, CSs are carrying out the following activities critical for effective school management in only one district: monitoring capitation grant activities; crosschecking SPIP activities against budget allocations; enforcing sanctions of absentee teachers, drunken teachers, etc.; and arbitrating between the community, teacher, and HTs when problems arise. Teachers in nine out of 11 schools visited confirm that CSs promote effective teaching and learning by reviewing their lesson notes and pupils exercise books to assess work output and quality. When CS and DEOC began monitoring schools with PAGE support, they found that teachers were not preparing lessons in advance, so the focus is to ensure output, not necessarily the quality or the delivery of the lesson. HTs interviewed have confirmed that CSs are visiting the schools more frequently and recording findings in the logbook. In one school, the HT stated that other HTs have complained that the CSs cover so many pages of the logbook when they come to visit as opposed to before, indicating that they are carrying out the above roles and responsibilities. While monitoring visits and issues covered during monitoring visits have increased, there is still room for improvement. The PAGE performance evaluation results showed that few districts utilize tools and approaches to support monitoring visits. In Savelugu and Tolon districts, CS visits are guided by action plans and performance indicators; in Ketu and Savelugu, they utilize monitoring checklists; and in Abura, Akatsi, and Bawku West, they hold CS review meetings. It is possible that these activities are occurring in more districts, but when CS were asked which effective management strategies they have applied as a result of PAGE trainings to improve governance and supervision at the school level, only two to three districts stated the above examples. SMC/PTA The SMC has played a critical role in improving school management at the community level. In all districts visited, an overwhelming majority of DEO staff interviewed emphatically declared that SMC/PTA training has been the most effective strategy for improving school management. A common response echoed among DEOs, school representatives, and community members themselves was, “The community now owns the school.” As a result of PAGE training, SMC/PTA members now visit the school more frequently and are involved in SMC/PTA meetings, SPIP activities, conflict resolution between teachers and community members, and monitoring teacher and pupil attendance. The percentage of SMCs effectively supporting schools has increased from 13% at baseline to 64%, according to the team’s performance evaluation results. A. SMC/PTA monitoring visits: Nine of the nine SMC/PTAs interviewed stated they visit the school at least twice per term. The average number of visits per term was nine. During school visits, SMC/PTA members visit classrooms to ensure teacher and pupil punctuality and attendance. They also meet with the HT to plan or follow up on meetings. SMC/PTA visits are recorded in logbooks. One SMC Chairperson stated that not all visits are recorded. The reason is because some illiterate parents feel uncomfortable requesting teachers to sign the logbooks on their behalf. B. SMC/PTA meetings: According to HTs interviewed, the SMCs—not the HTs—now schedule meetings and propose the agenda. SMC/PTAs meet at least twice per term, at the beginning of the term to set objectives and at the end of the term to review student performance. Final Evaluation of the USAID/Ghana PAGE Project Final Report 12 C. SPIP activities: Nine districts reported that the SPIP was developed with participation from SMC/PTA members and the HT. Eight districts also included schoolteachers and one district involved community volunteers/members in the SPIP preparation. There is also evidence of increased transparency, as the SMC Chairperson is required to sign for the release of the capitation grant to the schools5 . According to nine of the SMC/PTAs interviewed in nine districts, 24 SPIP activities were implemented during the past year. Ketu North, Nkoranza, Tolon, and Akatsi are implementing three or more activities each. The types of activities are evenly split between teaching and learning and facilities/minor infrastructural improvements. SMC/PTA members have assisted teaching and learning through purchasing TLMs for the school (three districts); helping to establish a computer laboratory to aid learning of ICT (two districts); providing after-school tutoring (two districts); sponsoring cultural/public speaking activities (one district); and lobbying to the DEO and DA for provision of materials, classrooms, and teacher accommodations. Facilities and infrastructural contributions have included connecting the school to electricity (three districts); constructing a classroom block as impetus for government intervention (two schools); constructing teacher accommodations and sanitation facilities (one school each); installing fencing around the school campus; and planting trees to protect the school land and building. Data were not available for two districts, Ga East (since the SMC/PTA was unavailable) and Abura Asebu, as this was a newly formed SMC with a SPIP in place but not yet implemented. D. Monitoring teacher and HT performance: SMC/PTAs are also cooperating with the DEO to manage teacher performance. DDEs in Upper East and the Northern regions stated that SMC/PTAs have become so empowered and confident that they are not afraid to report financial malpractice, non-performing HTs, and absentee teachers directly to the DEO. The DDE in Ga East asserted that SMC/PTAs have helped to improve HTs’ performance. For example, HTs now attend meetings more regularly and report on school activities as compared to before PAGE, when they did not. She further added, “The capacity building for the SMCs was very helpful. It takes half of the load off our [DEO’s] shoulders.” Ga East attributes 85% of its improved governance in education to the PAGE project. Communities have also become heavily involved in monitoring pupil activities. Five districts have participated in at least one SPAM in which student performance was discussed. At Town Hall meetings and SPAMs (that predate the PAGE project), community, school, and government stakeholders share performance results, discuss problems impeding education, and develop joint solutions. According to CS, DEOC, CPC, and DDE staff interviewed, discussions have centered on enrollment, retention, effective teaching and learning, the role of parents, and the importance of investing in children’s (particularly girls’) education. With participation from a wide range of religious, government, and education stakeholders—e.g., Chief, Queen Mother, civil society organizations (CSOs), district assemblies, schools, and the DEO—communities have been able to take actions to address major problems that were impeding performance, such as banning all-night activities that led to teenage pregnancy and establishing a community bylaw that enforced a curfew to ensure pupils study and attend school regularly. 5 The SMC Chairperson in one community refused to sign expenditures for the capitation grant because, as reported by the DDE, he was not involved in preparation of the SPIP. The HT has since been transferred to another school and demoted. This is a sign of increased accountability and cooperation between the SMC and the DEO. Final Evaluation of the USAID/Ghana PAGE Project Final Report 13 E. Conflict resolution: GES, teacher, and HT respondents all indicated that the SMC/PTA training has helped to “bridge the gap between the school and the community.” SMC/PTAs are more competent in resolving conflicts between school and community members—between SMC and PTA or between the SMC/PTA and the HT. The Tolon SMC Chairman stated, “We were already visiting schools [prior to PAGE], but in the past, we would often come back home angry because we would have quarreled with the teachers.” (HTs also affirmed that some community members verbally and physically assaulted teachers.) “Today, we are friends and coworkers.” According to a Tolon Community Volunteer, the teachers and community members who were once foes now play regular football games, engage in tutoring students together, and live within the same community, since the community has built teacher accommodations to reduce commute times and teacher absenteeism. The overwhelming contributions of the SMC/PTA have resulted in improved teacher attendance, increased enrollment, and improved pupil attendance. Prior to the PAGE project, the SMC was inactive and weaker than the PTA. DDEs all affirm that the SMC level of involvement in the school has increased dramatically. According to one DDE interviewed, increased participation by the communities in education supervision and management at the school level was unexpected. Only the SMC/PTA executives were trained, but the effect trickled down to the entire community. Respondents stressed the importance of the SMC/PTA and their continued involvement in school management because unlike the CSs, they are able to visit the school on a daily basis and address problems at the household and community level. Their effect on pupil performance has not been as significant. SMC Chairpersons themselves or Community Volunteers attribute low pupil performance to parents’ illiteracy, which causes them to place a low value on education and limits their ability to interpret marks. SMC/PTA members also report challenges in their own ability to persuade all parents to purchase books or stationaries for pupils, feed their children properly or send them to school regularly. SMC Chairman, Chiefs, and Community Volunteers, on the other hand, who are literate, are very effective in raising issues at the school level (e.g., need for electricity, computers for ICT, tutoring, resources for KG, maintaining logbooks), increasing awareness among the community because they themselves are role models, in incorporating needs into the SPIP, and in tracking pupil performance. Head Teacher Performance The evaluation team observed that five schools had a vision supporting literacy and SPIP displayed. In five schools, five HTs also reported that they provided INSET focused on English, reading, or language arts, and provided feedback to teachers following classroom observations or review of lesson notes. DDEs, CSs, and teachers were asked to rate the effectiveness of the HT in all 11 of the districts visited. The mean score was 60%. The DDEs stated that the HT training has been very effective. It has helped them to be more effective in their management and record keeping. Other DEO respondents stressed that the HT is the most important actor but has the lowest performance. HTs have a vision and mission but lack the ability to effectively implement them to improve student and teacher performance. CSs affirm that many HTs have improved upon their management functions and are enforcing effective teaching and learning, but there are still a good number who are ineffective. According to CSs interviewed, there are countless instances in which they perform duties on behalf of the HT when they arrive at the school and find the HT absent. HTs were often cited as the least improved actor among other DEO stakeholders interviewed. While nearly 50% are applying effective leadership strategies, many are ineffective at monitoring teachers, as they themselves are teachers and lack motivation and incentives. The majority of pupils interviewed reported that HTs often substitute teach when classroom teachers are absent. Therefore, HTs are overburdened with a double workload, making it difficult for them to supervise and support teachers. Final Evaluation of the USAID/Ghana PAGE Project Final Report 14 Question 2 Conclusions 1. The most significant contribution of the PAGE project towards improved governance and supervision has been the empowerment of the SMC/PTA to work collaboratively with the DEO to monitor school performance. 2. DEOC/DEO and CS monitoring has been highly effective in improving teacher attendance/performance and increasing accountability, but effective monitoring strategies, such as use of work plans, checklists, and performance indicators, are not being systematically implemented across all districts (only three districts). 3. CSs have increased frequency and quality of monitoring visits but have not yet fully exercised their role in providing pedagogical support to teachers in the classroom, tracking SPIP and capitation grant activities, and enforcing sanctions. 4. HTs have shown the least improvement according to stakeholders interviewed. They lack motivation and support to fully achieve reading targets developed under the PAGE project. Question 2 Recommendations 1. Continue supporting SMC/PTAs to effectively carry out their duties. To sustain the enthusiastic and active participation of the SMC/PTA over the long term, the GES and USAID should continue to engage the SMC/PTA in refresher trainings and activities that solidify relationships between communities, schools, and the DEO, such as town hall meetings, SPAMs, and the SPIP. As recommended by a DEO staff person, the GES may also recognize hardworking SMCs (e.g., those that develop infrastructure, provide teacher accommodations, recruit volunteer teachers, etc.) with awards and/or other incentives. 2. Standardize reporting structure for monitoring visits. The GES should consider adopting effective monitoring strategies identified in the three districts—namely, the use of work plans, performance indicators, and monitoring checklists to capture standard data in all districts on school performance. The Municipal Director of Education (MDE) in Savelugu has developed a comprehensive checklist (20 items for monitoring) and was asked to distribute copies to other districts at a United Nations Children’s Fund (UNICEF) meeting. This could be a potential template for other districts to adapt. 3. Increase capacity of CS and HT to exercise their pedagogical and managerial roles. USAID should support the GES in current or future projects in developing incentives/sanctions for increasing teacher accountability and provide training on methods for appraising teachers, delivering sanctions, and rewarding good teacher performance. They should also receive training on conducting classroom demonstrations, providing constructive feedback, setting teacher development objectives, and using appropriate reading methodologies (e.g., NALAP) to improve literacy outcomes. If the CS and HT work together and empower teachers, it will lessen the burden on each individual. Evaluation Question 3: Lessons Learned and Best Practices Question 3 Findings The factor that has largely contributed to achievement of results, as described among the findings for the previous evaluation question, is increased awareness of responsibilities and capacity of all stakeholders, especially the SMC/PTA, to effectively carry out their monitoring and management roles. Despite this significant achievement, the project has faced numerous challenges at the national, district, and school levels. This section highlights the key challenges and best practices frequently cited by project stakeholders and beneficiaries. National-Level Challenges A. Sub-granting delays: According to the 2013 USAID audit report findings, although the cooperative agreement between USAID and CARE International was signed in June 2010, project activities in the districts did not start until November 2011 (17 months after the agreement was signed for this three-year Project). According to project officials, this was because CARE had difficulty finalizing its sub-agreement with GES. The late release of funds interrupted GES work schedules and caused activities to be rushed. The delay in project start-up also significantly Final Evaluation of the USAID/Ghana PAGE Project Final Report 15 hindered achievement of targets in the first two years. Targets for some activities were later reduced in Year 3 and the project was extended for another year to account for the delay. The revision of targets was a recommendation from the USAID audit, and it was done in collaboration with USAID. B. Record-keeping and documentation: A third challenge the project faced was related to documentation and record-keeping. The first annual assessment found that the districts did not keep accurate records of town hall meetings and joint partnership planning meetings and therefore could not provide evidence of participation and issues discussed. As a result, some of the town hall meetings (THM) and joint partnership and planning meetings (JPPMs) needed to be repeated in order to meet USAID indicator requirements. DEO staff interviewed noted that activity guidelines were delayed and so they were not aware of the requirements. Providing adequate guidelines prior to the start of activities and maintaining proper documentation became a lesson learned for the DEO on USAID reporting requirements. District-Level Challenges A. Monitoring challenges: Four DDEs and 11 DEOCs stated they lack means of transportation to conduct regular school visits. They face difficulty reaching remote villages due to poor road conditions and high transport costs. CSs (75 total) unanimously reported their greatest challenge was frequent breakdowns of the motorbike supplied by the PAGE project even when they are well maintained, citing poor quality of the motorbikes, bad road conditions, and high fuel consumption. They claim the bike was not the original Yamaha AG100, as parts bought locally did not fit this model and had to be imported from Togo. These CSs stated they are frustrated with the level of maintenance required and the allowance is insufficient to cover the frequent repairs, causing high out-of-pocket expenses. The DDEs are also dissatisfied with the maintenance costs and the number of stranded CSs at the community level needing transportation back to the DEO. B. M&E system challenges: The evaluation team observed that although respondents in all 11 districts comprising DEO and school staff in both northern and southern areas were aware of the M&E system, it is still in the nascent stages. DEOC and CS monitoring reports state challenges with HT and teacher record-keeping. Teachers interviewed in five schools reported not having the Continuous Assessment booklet to record grades. For instance, the Nkoranza DEOC monitoring report stated that HTs were asked to use part of their capitation grant to purchase SBA booklets for teachers in order to replace the outdated Continuous Assessment booklets. This implies that teachers and HTs may have been unaware of the proper tools for recording pupil performance and lack competence in requesting the necessary resources through the SPIP. The Bongo District DDE commented that HTs have received training and have data, but they need help with preparing the school report, interpreting the SRC and disseminating the information. C. Inadequate training: All districts report that the PAGE training did not reach all schools in the district and that both initial and refresher trainings are needed for each stakeholder. D. Insufficient funding from Government of Ghana (GoG): Numerous districts reported not having received any government allocation for the past three years or, in some cases, since 2008. DEOC and DDE staff interviewed frequently reported that the DA is underfunded. This affected the government’s ability to address critical infrastructural shortages (e.g., teacher accommodations, classrooms, sanitation facilities) and teaching and learning materials required for quality education service delivery. The project also struggled with obtaining commitments from the DA to achieve indicator targets and sustain monitoring visits and trainings. Due to the fact that the Final Evaluation of the USAID/Ghana PAGE Project Final Report 16 GES is still not fully decentralized, it is difficult for education costs, including DEOC monitoring lines, to be integrated into DA budgets. PAGE has achieved 41% of this indicator and, even with a reduced target, is striving to reach 45% by the end of Year 46 . E. DEO attrition: According to PAGE/CARE staff interviewed, high attrition of DDEs and other GES staff slowed down the pace of activity implementation in Tolon, Bawku West, Ketu North, and Akatsi. Out of five districts in the Northern and Upper East regions visited, three had frequent transfers; Bawku West, Bongo and Tolon had three or four DDEs over the life of the project. Many PAGE activities depend on the availability of high-level DEOC and DEO government officials with competing priorities. Demanding schedules and massive transfer at the DEO level affected the number of core staff available to support activity implementation. CARE learned through the process that engaging multiple stakeholders breeds continuity and sustainability. With frequent changes in leadership, smooth implementation also requires very close relationship building between the PO and DEO as well as consistent monitoring support. School-Level Challenges Teachers in 10 schools visited reported that one or more of the following challenges hindered student achievement (listed in order of most frequent response): overcrowded classrooms (exceeding 60 pupils); absence of stationery; lack of supplementary materials for children; lack of TLMs (including chalk); and parents’ failure to nourish, supervise, and support their children’s education. School personnel (teachers in five districts and HTs in four districts) report that parents are a major challenge in helping to educate children in the school system because they do not fulfill their role of ensuring that children have required learning materials, are prepared to learn, and arrive to school punctually and regularly. The most significant challenge impeding student achievement according to CS and DEO staff is teachers’ lack of commitment to attend school regularly. According to DEO staff interviewed, teacher absenteeism is caused by a number of factors. One is distance. As most teachers commute to school from distant towns due to lack of teacher accommodations in rural areas, they are not punctual and have irregular attendance. They also miss school to attend distance education courses. Still others simply lack commitment and discipline despite sanctions, as perceived by CS and DEO staff. Teacher absenteeism affects students’ attendance and persistence. Often, when students find the teacher is absent, they do not return to school until they are notified that the teacher has returned. Based on focus groups with primary school teachers in all ten schools visited, teachers’ lack of commitment to attend school regularly or perform at the highest standards is related to insufficient government support to address issues. Teachers interviewed in three schools visited stated that the government is not responsive to the following: transportation and salary needs of newly posted teachers; teacher accommodations; and meeting class size and textbook standards, making it difficult to manage overcrowded classes with limited resources. The SMC/PTA confirmed that teacher accommodations, lack of teaching and learning materials, and large class sizes were challenges. In several districts, the SMC/PTA was able to provide electricity, teacher accommodations, and classrooms, but the quality of the infrastructure was not according to government standards. The majority of teachers interviewed asserted that there is improvement in government responsiveness to requests for furniture or classroom construction, but to a lesser extent for teaching and learning essentials. 6 Annual Review Exercise. Semi-Annual Report no. 7, p. 28. Final Evaluation of the USAID/Ghana PAGE Project Final Report 17 All teachers, HTs, and CSs interviewed, related challenges impeding reading achievement to overcrowded classrooms; a high proportion of untrained teachers in primary and KG unqualified to teach reading; high attrition of NALAP-trained teachers; lack of foundational skills due to not having attended KG or over-age enrollment; inadequate use and availability of NALAP materials and English textbooks; insufficient electricity and reading materials at home; and parents’ illiteracy, which limits their capacity to monitor student performance. Finally, respondents noted that although they had proposed reading competitions as an effective strategy for improving reading performance, they were considered outside of the scope of the PAGE project and deemed an unallowable expense7 . In some districts, the GES funded reading competitions, but PAGE budget constraints prevented other districts from pursuing this activity. National-Level PAGE Best Practices Reported by CARE and GES HQ  Rewarding good district performance with an extra motorbike.  Training multiple GES staff at all levels developed a cadre of trainers within the GES and promoted the exchange of experiences and institutional memory.  Equal distribution of resources. PAGE provided five motorbikes and for exceptional districts one extra motorbike. Most districts had more than five circuits and hence more than five CSs. One strategy that was effective in fairly allocating resources was placing the bikes in a pool with each CS having an opportunity to use the motorbike according to an agreed-upon schedule. This allowed them to reach more circuits and schools. Sharing fuel allocations so that each CS received an equal share regardless of whether they use the PAGE motorbike or a personal one was generally viewed as a best practice. District-Level PAGE Best Practices Reported by DDE/DEO Officials  Use of CVs to collect M&E household data. Five districts stated that CVs assisting with data collection through conducting household and school-based surveys was a best practice. This data enabled the DEO to examine external factors (e.g., health, teenage pregnancy, child labor) that affect student achievement in respective PAGE schools and to develop effective interventions to overcome the barriers. M&E officers stated that CVs are able to reach remote villages and obtain data that they would not be able to collect otherwise. Unfortunately, when PAGE ends, they fear that the CVs may no longer be supported.  Sanctioning non-performing HT/teachers through demotions, withholding salary, blocking promotions, and requiring teachers to apply for and earn promotions. For example, in Savelugu district, the DEO has placed stop payments on salaries of teachers who do not report to school; they have demoted non-performing HTs; and now require new HTs to pass a screening process and one-year probationary period prior to receiving tenure.  Re-assigning teachers and staff based on capabilities. The MDE in Savelugu has re-assigned teachers according to their areas of specialty and qualifications. Teachers who have degrees in early childhood education (ECE) and were teaching in the JHS were posted to KG. Those in senior high school with general degrees were placed in JHS or appointed to the DEO. He also moved trained teachers from upper primary to lower primary, where they are most needed, and those in lower primary to upper primary. Others who were not capable as HTs were removed and returned to the classroom to teach. He does not tamper with their rank or salary, only re-posting 7 In one district the CS indicated that PAGE had imposed activities where reading competitions were not identified in the current tranche. Final Evaluation of the USAID/Ghana PAGE Project Final Report 18 them to where they will perform best according to their capabilities. Teachers protested at first, but understand the rationale and agree it is best for the students to have trained, capable teachers.  Intensifying visits to low-performing schools to improve achievement.  Rotating Regional Director Meetings between districts and linking them with school monitoring visits to transfer knowledge and experiences (Bawku West).  Requiring teachers to sign pay slips via the CS and reporting vacant teaching posts. When the salary arrives, all pay slips are sent to the DEO. In Savelugu district, the CS signs and takes the pay slips to the schools and asks teachers to sign. If the teacher is absent or the post is vacant, they return the pay slip to the MDE. The MDE then stops payment of salaries of teachers who are absent frequently or have transferred out and have not reported it. In such cases, a letter is sent to the bank requesting a stop payment and the teacher is contacted and requested to return money to the government. As a result of such sanctions, GHS 158,000 have been returned to the consolidated fund. The DDE for Tolon implemented the same practice and found that teacher absenteeism cost Tolon almost GHS 200,000, which has since been returned to the GES.  Fuel allowance contingent on submission of CS monthly monitoring reports.  Best practices for sustainability: Bongo provides an excellent example of PAGE sustainability. The team found that there is a budget line of GHS 13,000 for monitoring. SPAMs are part of their annual work plan. There are plans to purchase an engine for a dilapidated vehicle to support monitoring visits. The DA approved items (laptops, fridges, etc.) to reward teachers for good performance and they have leveraged other funding sources from the World Bank (GPEG), the Department for International Development (DFID), and GoG, and plan to involve many more schools, communities, and SMC/PTAs in PAGE activities. They expect PAGE to be fully institutionalized by the end of the project.  DEOC practice of “adopting schools.” The Nkoranza DEOC has adopted 10 schools for regular monitoring. DEOC members asked staff and students to consider them as “adopted parents concerned with their well-being.” Frequent monitoring has allowed the DEOC to establish close relationships with school staff and to promptly respond to immediate needs, such as chalk or exercise books.  Motivating HTs through recognizing good performance (e.g., best HT award) was mentioned as a best practice in Abura Asebu, Asuogyaman, and Savelugu. According to DEOC members in Savelugu, the Best Teacher award was instituted in Tolon a few years ago and it was very effective. They are currently working on an award for the HT proposed by GPEG. The CSs invited high-performing HTs to come to the office for interviews (through a process structured by the GPEG) and they selected those who demonstrated notable accomplishments.  Conducting regular, unannounced visits from DEO staff is a best practice that allows them to learn about the real situation in the school. Best practices identified through interviews with CS are:  CS monitoring work plans and CS review meetings. The evaluators observed in several districts that the DEO had instituted CS monitoring work plans. The work plans set specific objectives for school visits according to the priorities for the corresponding term. Following visits, CS and the AD of Supervision held review meetings in which they discussed the outcomes of the visits, any challenges faced, and the action plan for addressing the issues identified. In some cases, the findings of the visits would inform the objectives set for monitoring schools in subsequent terms.  Use of CS monitoring checklists. Six districts stated that this was an effective management strategy that helped focus visits on monitoring teaching and learning.  Training SMC/PTA on roles, responsibilities, and conflict management helped them to resolve issues and work collaboratively towards a common vision. School-Level PAGE Best Practices Reported by HTs, Teachers, and SMC/PTAs  Reading Competitions. Respondents stated the reading competitions were highly effective in motivating pupils to read and in encouraging parents to buy their children books. Also, given that the reading competitions in some schools are conducted on a monthly basis at the school level, Final Evaluation of the USAID/Ghana PAGE Project Final Report 19 circuit level, and district level, there are many opportunities for students to earn extra marks, recognition, or awards.  Currently, there are inadequate teaching and learning materials for teachers to become very effective in teaching reading. Therefore, use of TLMs, including local resources, the NALAP big book, and textbooks, was considered the primary best practice in 10 of 11 districts visited. Respondents, especially teachers in KG and the early grades, insisted that not all teachers have access to TLMs; teachers that do have access or have observed others state that TLMs have significantly facilitated teaching and learning in the classroom. They note that the visual aids, particularly the Big Books, pictures and note cards, take the burden off the teacher to explain the lesson and stimulates student learning.  Mandatory reading time in school. Several HTs and one DDE mentioned this as a best practice. After observing during school monitoring rounds that children could not read or write, the Savelugu DDE instituted compulsory reading in all subjects of primary school. The first 10 minutes of each lesson must be devoted to reading a topic in relation to the subject. He also advised HTs to include reading materials and other supportive activities in the SPIP.  Use of reading charts in class for students to monitor their progress. In Ga East, the language teacher posts a chart demonstrating the reading stages in the classroom. Students are then responsible for tracking their own reading progress and physically moving their names from one stage to another when they meet the benchmark. Non-readers receive extra help from the teacher during after school 40 minute tutoring sessions.  After-school study sessions/classes for students with reading difficulties (described above).  Teacher durbars.  Community bylaws enforced by town chiefs, HTs, and SMC/PTA members requiring children to stay home at night and study.  SMC/PTA establishing school farms as a means of generating school funds.  Use of radio messaging and community drama performances for community awareness of education, teen pregnancy, role of parents in pupil achievement.  Training illiterate parents how to check exercise books and inquire about their children’s performance. Question 3 Conclusions Conclusions on Lessons Learned 1. Persistent support and engagement of multiple GES staff is required for continuity and sustainability. 2. Having appropriate financial mechanisms, documentation, staffing, and implementation guides in place are critical for adhering to project schedules. 3. According to teachers’ interviewed, teacher absenteeism and poor performance is largely caused by inadequate government responsiveness to address issues. In order to address issues, HTs and teachers require greater competence and awareness in utilizing the capitation grant, SPIP and other channels (e.g., process for lobbying the DA, private sector) for accessing government and donor support. 4. Provision of durable motorbikes and sufficient fuel allowance is critical for the effectiveness of the Circuit Supervisors to carry out their monitoring and supervision roles. Conclusions on Best Practices 1. Rewarding good district performance, training multiple GES staff and providing continuous capacity building, and equally distributing resources has improved the efficiency and internal capacity of the system within the GES and has benefited other donors. 2. Best practices for improving student performance, particularly in reading, have included use of TLMs, reading competitions, use of reading progress charts in the classrooms for pupils to self￾monitor improvement, intensified training and support provided to low-performing schools, and encouraging the inclusion of reading materials and other supportive activities in the SPIP. Final Evaluation of the USAID/Ghana PAGE Project Final Report 20 3. Reading competitions have been a best practice in motivating students to read, in persuading parents to purchase textbooks and reading materials for use at home, and in creating collaboration between schools and communities to improve literacy. 4. Incentives (e.g., best teacher/HT award), stringent promotion practices, sanctions, and re￾assigning teachers based on capabilities have been effective practices for improving teacher performance. 5. To increase the quality and outcomes of CS monitoring visits, best practices have included CS monitoring work plans, use of CS monitoring checklists, CS review meetings and provision of fuel allowance contingent on submission of CS monitoring reports. Regular unannounced visits by the DEOC and DEO are best practices for increasing accountability and effectiveness of HTs and CS. 6. Use of radio messaging and community drama to increase awareness of the parents’ role in education and training illiterate parents on how to interpret marks have been effective in engaging parents to monitor pupil performance. 7. Engaging community volunteers in conducting household surveys is a best practice for identifying and resolving issues in the home or community affecting student performance. Question 3 Recommendations 1. The GES and USAID should incorporate best practices from the PAGE project in future programming, as relevant to the scope of the project. o For instance, to improve student achievement, use of TLMs, reading competitions, reading progress classroom monitoring charts, and inclusion of reading materials and reading activities in the SPIP. o To improve teacher performance, best teacher award, teacher durbars, competitive promotions, probation and sanctions, as well as recognizing high performing HTs are recommended. o In order to sustain and improve the quality of CS monitoring visits within the DEO, the following are highly recommended: CS work plans, checklists, review meetings and reporting requirements. o Use of appropriate media and training to increase parents’ awareness of their role in encouraging academic achievement, such as through training parents on how to interpret marks or using radio messaging and drama, to report on student test results are effective ways to engage parents. 2. In the near future, the GES and USAID should provide HTs with training on how to maximize the use of the capitation grant and SPIP to address school resource constraints, such as to increase availability of TLMs in the classroom. Evaluation Question 4: Ownership, Engagement, and Sustainability Question 4 Findings Measuring results and tracking progress has three major parts: 1. Results measurement and tracking (current and after project); 2. How the process of results measurement and tracking ensures there is ownership and active participation by key stakeholders; 3. How can these results measurement activities and practices be sustained? 1. Results measurement and tracking (current and after project) Results need to be measured against objectives using indicators to assess progress at all levels of implementation. Apart from key performance indicators defined in the PAGE PMP, system-level data and results involve stakeholders in the education system. At the school level, for instance, the evaluation team discovered that pupils’ attendance and performance data in terms of class and exam scores were appropriately recorded in attendance registers and pupils’ continuous assessment books, respectively. The HTs collate these sets of data, add teachers’ attendance, then submit the data to the Final Evaluation of the USAID/Ghana PAGE Project Final Report 21 CS, who then verifies and subsequently submits it all to the DEO for the development of the SRCs for the school on the one hand and the district SRC on the other. To ensure compliance, and as a transparency check, HTs keeps track of teachers’ attendance on a chart that is displayed in the Head Teacher’s office. With the support of PAGE, most Education Management Information System (EMIS) offices of the DEO have also been strengthened and the key statistics personnel had their capacity built in M&E. Invariably, this has resulted in the development and deployment of a set of standardized data collection forms/instruments that contribute data to the development of PAGE monthly and quarterly reports. These consist of a standard PAGE activity monitoring form, a checklist for use during DEOC monitoring visits in a few districts, a household survey for CVs to monitor community indicators, and a few others. The team observed that PAGE ensured the collation, analysis, and dissemination of BECE results in all 46 project districts to inform stakeholder appreciation of students performance to set the stage for the necessary deliberations on an improvement plan. This led to several schools that had been perennial 0% BECE achievers to improve their performance. For example, at Offinso Municipal, CSs intensified their support visits to schools that were scoring 0% in BECE results. They provided INSET for teachers to improve pupils’ performance. The SMC/PTA ensured that children studied at home after school hours. Table 5 shows how some of these schools moved from 0% BECE achievement to better results as informed by CARE. 2. How the process of results measurement and tracking ensures there is ownership and active participation by key stakeholders Building ownership involves processes of review, action planning, and developing individual and collective responsibility. Bringing this concept of ownership to bear on results measurement introduces inclusiveness in the review and action planning process. This is evident in the manner in which PAGE conducted the measurement of results. The DEO, as the main facilitator of the various processes and the entity held accountable for delivery of policy implementation, projects and ultimate results, also has ownership of its data to facilitate planning and decision making. PAGE supported the DEO with tracking results through facilitating DEO-DEOC joint partnership planning meetings at the district level, which provided an opportunity for DEOs and DEOCs to share results of monitoring visits and engage key decision-makers in responding to the issues identified at the school level. During these meetings, issues were discussed as well as follow-up actions. To ensure that decisions at the school-level reflect evidence-based data, PAGE deliberately built the capacity of HTs and CSs in effective leadership and management strategies so that they would be able to interpret student performance and utilize the data to design school visions and missions that are performance-oriented. The evaluation found five schools had developed visions and they were clearly displayed. SPIP is designed to support schools and SMC/PTAs with setting school performance targets, implementing school improvement activities and monitoring performance. The school capitation grant has been tied to the SPIP in order to fund these activities. All SMC/PTAs reported Table 5: Improvements in School Performance through Results-Oriented Planning Source: interview with CARE staff School BECE Performance for Various Years 2011 2012 2013 Kwapanin R/C JHS 0% 62.5% 94.1% Kokote L/A 0% 100% 100% Asuboi M/A JHS 0% 16.7% 54.6% Final Evaluation of the USAID/Ghana PAGE Project Final Report 22 they are actively engaged in implementing activities related to teaching and learning and infrastructural improvements. To ensure accountability and adequate oversight of the capitation grant, SMC/PTA has been given the mandate for approving release of capitation grants to schools and assuring remediation for better results and they have been working directly with DEOs to ensure financial accountability and compliance. With a wide variety of stakeholders, including community, government and education, results of the BECE, school-based assessments and more recently, the SRC, are being discussed at various fora (such as SPAMs, Town hall meetings, etc.) and action steps decided with key stakeholders assigned to various responsibilities. The DEO system is managed by the EMIS unit which produces the District level SRC after it has received verified data from the CSs. In some cases, the DEOC serves as an source of verification and validation of activities implemented, immediate results and challenges faced on the ground. A list of systems, processes and tools that comprise the M&E system can be found in Annex VII. 3. How can these results measurement activities and practices be sustained? Sustainability implies enforced government regulation and provision of services (strictly following defined roles) or compelling collective interest and demand for education from the larger population. Interview responses from DEOs and DEOCs indicate that they will continue to organize THMs to discuss student performance at the BECE level. These THMs will continue to provide the necessary platform for stakeholders to take remedial actions in order to improve upon performance. At the SMC/PTA level, more than six Districts have indicated that they would continue to organize SPAMs and ensure that these SPAMs focus on students performance and issues related to education quality and improvement. Many of these SMC/PTAs posit that though it requires funding to organize better SPAMs, the benefits derived from the discussions alone is enough to encourage them to look for their own resources to organize the SPAMs. When asked which activities they will continue to implement or derive benefits from apart from THMs and SPAMs, 5 out of 11 SMC/PTAs stated that they would continue to use their new skills to sustain their school monitoring activities which involve monitoring pupils and teachers attendance, examining pupils exercises to gauge their performance and providing the needed support in terms of infrastructure maintenance and funds mobilization for other school needs. At the DEOC level, evidence from the interviews of growing commitment amongst Municipal and DCE for the need to fund DEOC activities that are student performance-based. The constraint to seeing this growing commitment come into practice is the continued starving of the Districts of statutory funds from Central Government. Even though many districts (50%) have indicated they will continue to monitor, collect data and utilize the data for decision making at the school, community and DEO level, there is also a good number (50%) of the Districts that think that performance monitoring and measurement might not continue when the PAGE project closes out. In Nkoranza district for instance, the SMC/PTA indicated that, the opportunity cost of neglecting their farms to monitoring attendance and thus ensuring quality data collection on attendance is so huge that, school monitoring might not be done with the vigor during the rainy season as it is done during the dry season when farming activities are minimal. In the Tolon district, the CSs indicated that efforts in monitoring schools and collection performance data might slow down with reduced funding as a result of PAGE close out. They argued that but for PAGE’s intervention, they hardly received any funds for fuel purchases and for maintenance of their motorbikes. Question 4 Conclusion 1. The PAGE project’s systems designed for measuring results, tracking change and using these for planning and decision making were in place and also built the capacity of requisite actors to use these systems effectively. As much as most actors are happy playing these roles currently, they Final Evaluation of the USAID/Ghana PAGE Project Final Report 23 may require still require additional and enhanced capacity to sustain what they are doing now beyond the life of PAGE. 2. Even though all DEOs and School communities were made to understand that the PAGE project was going come to an end at the end of the third year, and they will have to take responsibility for running the education governance system instituted by PAGE, the DEOs and their stakeholders were quite reluctant to accept that reality and continued to advocate for continued funding support from PAGE and USAID. 3. Post PAGE, the responsibility to internalize the coordination and reviewer role played by the PAGE staff to ensure that feedback on immediate results is disseminated to all actors to sustain interest and momentum is best held at the district level. Question 4 Recommendation 1. The best ways to ensure that progress and results are captured and/or continued and which also promote ownership, engagement and sustainability of interventions is to develop an exit strategy with the GES which will: a. Ensure that the DEO takes up the roles currently played by PAGE staff to ensure that the necessary feedback on immediate results is channeled to all actors to sustain the current interest and momentum; and b. Identify specific activities to prepare the DEOs and DEOCs to plan for raising funds from public, private or international sources to support their activities. c. Intensify the process of formalizing all identified best practices as part of an exit plan and build the capacity of supervisors to own the outcome of these practices. This will ensure smooth transfer of roles and responsibilities from PAGE to the GES staff and ownership will be automatic. Supervisors at various levels would have developed the confidence to demand reports and also use the information gleaned from such reports to inform decisions and enhance their reportage. Evaluation Question 5: Performance Monitoring Processes, Systems, and Tools Question 5 Findings PAGE Contributions PAGE contributed to the GES monitoring and evaluation system through training all DEO education staff on roles and responsibilities, which included training on documentation, data collection, and reporting, and through providing activity implementation guidelines for tracking performance and report templates. According to DEO staff interviewed, provision of motorbikes and the recruitment of community volunteers greatly assisted with timely data collection and accurate reporting. According to CARE Program Officers interviewed, PAGE also provided the DEO with technical support for work planning and grants management. As a result of PAGE assistance, the quality of HT, CS and DEOC monitoring reports improved and the SRC system was revived. Monitoring and Reporting Nine of the 11 districts sampled reported having a M&E system in place. Six districts are using the reporting format assigned by PAGE to produce PAGE activity reports, but they have yet to integrate this into the GES system. In terms of reporting mechanisms, eight districts now generate monthly CS monitoring reports as a result of the PAGE project; five develop situational/ad hoc reports to inform the DEO of critical issues requiring DDE attention; and one district generates a SMC/PTA report. Increased Mobilization of Staff According to DEO support staff interviewed, transportations funds and capacity building from the PAGE project mobilized GES staff (DEOC, CS, M&E, CPC, and Girls’ Education Officers (GEO) to visit the school and collect data that directly contributes to decision-making. During monitoring visits, they observe teaching and learning, discuss community concerns and gender issues with girls’ education facilitators and SMC/PTA members, and examine the conditions of the school. The findings Final Evaluation of the USAID/Ghana PAGE Project Final Report 24 are reported at debriefing meetings with DEO staff and are shared with decision-makers at joint partnership planning meetings, district and school community SPAMs, and town hall meetings. Improved data collection and collaboration have helped stakeholders to understand the issues at the school level and support decision-making. Community Volunteers Working in collaboration with the GES District Community Participation Coordinator and District M&E Officer, PAGE trained community volunteers to conduct a household survey that gathered information on wealth, poverty, vulnerability and parental status. This data enabled the DEO to examine external factors that affect student achievement (e.g., hunger, child labor, teenage pregnancy) in respective PAGE schools and in some cases to develop effective interventions to overcome the barriers. One benefit from their survey work revealed the extent to which children come to school hungry every day – some children only eat once a day; many are malnourished because they feed themselves. Other insights into family life led to confirmation that children spend evenings observing cultural obligations, which affect their performance in school. Five districts stated that the use of CVs to collect data was a best practice. Strengthening Education Management Information System (EMIS) and School Report Card System At the inception of the project, PAGE visited 9 districts to assess the implementation of the SRC and found that it was not utilized by any of the DEOs. Reasons varied, but data collection bottlenecks existed at each level of the system, from the HT not receiving the SRC data collection form (or completing it incorrectly) to the DEO failing to produce SRC outputs for schools. Despite the former USAID-Grants and Reporting Accounts to Improve Literacy (GRAIL) project, which introduced the SRC, the system was basically non-functional. To strengthen the educational system and promote transparency and accountability at the school and district levels, PAGE provided training to all actors on documentation, data collection, and reporting. Working through the existing system, PAGE incorporated the SRC into its SMC/PTA trainings, HT trainings, CS capacity building and SPAMs. Within the current GES system, ten districts have SRC forms and are beginning to use it to generate data. In these districts, the HTs now collect data on attendance, pupil performance, teacher absenteeism and other school-level indicators, and submit it to the CS who, in turn verifies the data and submits it to the district M&E/planning unit. The M&E Officer, with support from PAGE, participated in assessment activities to verify the data, and subsequently generated the SRC. PAGE supported the DEO to utilize the SRC for its planning purposes and to share SRC results through town hall meetings and district-level SPAMs. SRCs provide comprehensive school performance data such as pupil attendance/performance, teacher absenteeism, and community involvement. The EMIS unit also produces other reports such as enrolment data and reports from the community volunteer surveys. The data is intended to support schools with developing SPIPs and school-based reports. The above processes and tools are beginning to contribute to the timeliness, accuracy, reliability, and validity of data generated by the system on the performance of district schools. Among all 11 districts interviewed, four districts stated they are using the SRC and other field reports from various stakeholders including school monitoring reports in the development of their Annual District Education Operational Plans (ADEOPs). Ten districts have SRC forms and are beginning to use them to generate data. However, the evaluation team did not find evidence of the SRC being utilized for SPIP development, and it had only begun to be used in Town Hall Meetings and SPAMs, since the SRC was not available until the end of 2012 due to software challenges. Among all DDEs interviewed, only one district stated they are using the SRC in decision-making, distributing the SRC to HTs, and disseminating the SRC via SPAMs. DEOs commonly remarked that the BECE results were more often used to discuss pupil performance and guide decision-making. School/pupil performance is disseminated through displaying the SRC on noticeboards in two districts, through sharing BECE/SBA results in SPAMs in eight districts, via Town Hall meetings in two districts, and radio programs in one district. Final Evaluation of the USAID/Ghana PAGE Project Final Report 25 Additionally, involvement of the SMC/PTA in the SPIP is limited. Two SMC/PTAs interviewed in Akatsi and Tolon districts stated that it was their role to establish the vision, mission and strategies for school improvement. SMC/PTAs in two districts contributed to the development of the SPIP and its implementation process, namely Abura and Akatsi districts. However, four SMC/PTAs described their role to mobilize financial resources to implement the SPIP. Only one district (Ketu North) reviews the school’s progress in implementing the SPIP with the HT. Strengthened Financial Reporting PAGE developed activity definitions and guidelines to assist the DEO with implementing, monitoring, documenting and reporting on USAID grant-funded activities. This occurred after the first tranche when PAGE learned that District Education Offices carried out activities that did not meet USAID reporting requirements and they needed further guidance. The GES submitted monthly and quarterly activity reports and expense reports after the completion of each activity. Funds for subsequent tranches were released when 75 percent of activities had been accounted for and approved by the GES and PAGE finance departments. The reporting requirement enabled PAGE to resolve any issues prior to the release of subsequent tranches. PAGE and national GES accountants held at least four regional meetings with district-level accountants to discuss issues with financial reporting and provide additional training and guidance. Ongoing capacity building of district GES accountants and staff has led to improved efficiency within GES administration and timely release of funds to support project implementation. The quality of activity and financial reports has improved as a result of PAGE assistance, and this has benefited all donors and sectors within the GES. The fact that GES finance accountants received the same training as the M&E Officers also contributed to reinforcing the norms of accuracy, timing and validity for all data processing. CARE International reports that the M&E system is inconsistent in its performance among its target districts and still encounters problems with reporting on funds spent for planned activities. The findings of the evaluation team’s inspection of the M&E system in each of the 11 districts visited are consistent with the CARE implementation team assessments (from an introductory interview with COP and DCOP at PAGE headquarters). Question 5 Conclusions 1. While there has been progress made in the data collection, processing and analysis processes at the district level, dissemination and feedback processes at the school level are slowly being initiated. SMC/PTAs are not yet fully engaged in tracking schools’ and pupils’ performance using DEO monitoring tools. 2. DEO frontline staff and CVs have been actively participating in data collection activities, monitoring performance at the community level, and reporting on issues. Improved data collection, analyses and dissemination processes plus collaboration among GES frontline staff have helped stakeholders to make annual plans and inform constituencies about key issues in education. 3. The SRC is still relatively new and not yet fully institutionalized at all levels of the system. Question 5 Recommendations 1. The GES and USAID should continue building capacity in the following processes, systems, and tools that enhance the M&E system within the DEO in the medium to long term. a. Data collection tools and formats—for example, to generate measures for gender inclusiveness, developing tools to conduct more extensive school audits and inventories that estimate needs for physical infrastructure, text books, and teacher and staff requirements. b. Training in how to develop indicators for educational planning such as teacher/pupil ratio and for project monitoring and evaluation. c. Data collection for tracking individual student performance and student report cards. d. Training on using demographic projection to estimate school-age cohorts entering KG and first grades. Final Evaluation of the USAID/Ghana PAGE Project Final Report 26 e. Training on using net and gross enrollment ratios and deriving promotions, repetitions, and dropouts for estimating school classroom demand and teacher supply. f. Data processing and analysis using household data for use in aligning education policy with conditions in the communities. g. Data verification and data quality techniques. h. Processing SRC data into the computer and using programs to analyze the data. i. Report writing with SRC or other statistical software to produce reports and graphics. j. Training on working with decision makers on how best to display data analyses and specific findings relating to costs and cost optimization. k. Training on the use of data for policy analysis using trend analysis and geographic dispersion. l. Data analysis for use in planning, reporting, and decision-making m. Sub-sector analysis for areas such as school supervision, combining DEOC and CS results from school visits, or analysis of SPIP results across the district and districts. n. Training for regional analysis of educational data. 2. USAID should offer training on the interpretation and utilization of the SRC, particularly for the SMC/PTA, HT, and teachers, so that it is fully integrated at all levels of decision-making. Final Evaluation of the USAID/Ghana PAGE Project Final Report 27 GENERAL CONCLUSION FOR PAGE PERFORMANCE EVALUATION This evaluation exercise has made some important discoveries. The team learned that while its attempt at student achievement testing was not as rigorous as a formal assessment, the pace of learning among students is still slow and the depth of learning inadequate. While PAGE is not responsible for student achievement, the theory of change behind the project is that if governance and accountability improve within a district education system, student learning should also improve. The team confirmed without a doubt that the education systems in the interviewed areas are showing significant signs of renewal and greater capacity in delivering education. In the evaluation team’s view, the system is ready for a step up in achieving greater competence and sophistication in how it delivers education from the community to the school to the District Education Office to the District Assembly and its standing committees. In its evolution, the next phase of funding may shift away from the tranche mechanism to one that recognizes the decentralization process and where districts are now more autonomous than they were previously. The major threats to the school system at the regional and district levels are the uncertainty of funding from the center and the absence of leadership at the DDE level. The encouraging piece to this new era of education is a newfound willingness of the District Assemblies to cooperate and play their roles in supporting education with the means they have. Both DAs and DEOs have learned the power of the town hall meeting and other public events to bolster the cause of education and to engender a new spirit of engagement with the sector. These entities are surprised that parents and communities could be this interested in education. At the school level, teachers also need to feel that they are a greater part of their professional system, which encourages technical links to teacher training institutions or to networks that are willing to support them. As communities mobilize resources to improve their schools, the system will experience rising expectations. These will have to be managed and accompanied by a corresponding willingness on the part of the system to also contribute meaningfully to the investment. The next generation of education in the districts will thus call for new skills and an openness to face the outside world in the quest for resources, either from donors or from the private sector. The sustainability of the education system depends on officials developing some of these new skills in order to keep their systems going and to maintain a sense of vibrancy about learning and preparing youth for a rapidly changing world. Final Evaluation of the USAID/Ghana PAGE Project Final Report 28 ANNEX I: WORKPLAN INTRODUCTION This introduction presents a background of the Ghana Partnership for Accountable Governance in Education (PAGE) Project, the evaluation purpose and questions, and a description of the Social Impact, Inc. (SI) Evaluation Team (Team) as an introduction to the work plan. PROJECT BACKGROUND PAGE was designed to establish governance in the primary school education sector in Ghana. The project’s goal is to improve student achievement through strengthened educational governance, supervision, and accountability, with the following intended results: 1. Improved capacity of District Education Oversight Committees (DEOCs) to contribute more effectively to improving school governance and supervision activities within their districts; 2. Strengthened DEOC and District Education Office (DEO) collaboration and responsiveness to school management issues in their respective districts; 3. Improved competence of circuit supervisors, Head Teachers, and School Management Committees to monitor, manage, and report on school management performance in order to increase student achievement; 4. Appropriately resourced circuit supervisors having essential logistics and equipment to implement an improved and focused supervision program for schools; and 5. Active and engaged Parent Teacher Associations and School Management Committees trained to seek and interpret school performance reports with the view to developing and implementing strategies for improving school management systems and student achievement. The project goal and results was set within the larger context of Ghana’s education decentralization policy that mandates that District Assemblies have executive responsibility for the provision and management of basic and second cycle schools. With a decentralized education system, schools will attain the autonomy needed for effective education delivery. PAGE provided assistance through a Cooperative Agreement (CA) (No. 641-A-00-10-00036-00) on a competitive basis to support the strengthening of governance, supervision and accountability in the education system at the district levels in Ghana. PAGE was implemented by CARE International under a CA beginning in June 2010 and lasting until June 2013. The project was reviewed by USAID and provided an extension until June 2014. Project activities in the districts did not start until November, 2011, 17 months after the agreement was signed for this 3 year program. This, according to project officials, was because of difficulties finalizing a sub-agreement with Ghana Education Service (GES). GES was to provide the mechanism for disbursing funds to the 46 districts where activities are implemented. EVALUATION PURPOSE AND QUESTIONS USAID/Ghana has contracted SI to conduct a final performance evaluation of the PAGE project as outlined in the evaluation Scope of Work (SOW), reproduced in Annex I, Part A. While the PAGE project has five objectives, the SOW defines five questions which will assess the effectiveness of the total project level of effort. The specific questions to be addressed in this evaluation are: 1. Have governance and supervision interventions at the district level resulted in improved student achievement? If so, to what extent and why? If not, why not? 2. What intended and unintended contributions, results and/or impact has the PAGE approach and activities achieved relative to improving governance and supervision in schools? Final Evaluation of the USAID/Ghana PAGE Project Final Report 29 3. What factors affected the achievements and results of the project? What are the lessons learned and best practices, and from which stakeholders or beneficiaries? 4. What are the best ways to ensure that progress and results are captured and/or continued that also promote ownership, engagement and sustainability of interventions after the project ends? 5. What performance monitoring processes, systems and tools were used to ensure accurate, timely, reliable and valid performance and indicator tracking, reporting and feedback at all levels? How did this contribute to logical implementation of activities, data validation, documentation and review of work plans, indicators and/or activities? In answering all the evaluation questions the Team will highlight gender specific approaches, outcomes, participation and engagement to better understand how the PAGE approach has impacted student achievement and school performance. The Team intends to use an evidence matrix of the USAID Domains of Gender Analysis framework to identify any gender-based gaps between males and females and examine how PAGE addressed them. SI EVALUATION TEAM ROLES AND RESPONSIBILITIES The PAGE evaluation team consists of four personnel who will serve as chief implementers of the evaluation work plan including all data collection and analysis methodologies, communications with USAID/Ghana in Accra, and the completion of final deliverables. 1. The Evaluation Team Leader, Dr. S. Dunham Rowley, has 40 years of experience managing and evaluating education and governance development programs with extensive experience in West Africa. 2. The Evaluation Technical Advisor, Ms. Brenda Sinclair has over 10 years of experience in international education practice, analysis and program design and has substantive evaluation experience in Ghana. 3. The two Evaluation Specialists, Mr. Saaka Adams and Mr. Charles Nornoo each have over 10 years of experience leading monitoring and evaluation (M&E) projects for USAID program evaluations within Ghana. The Team will be provided with sufficient support from a team of SI home office (HQ) personnel and a locally-based logistician as follows: 1. Home Office Program Manager (PM), Ms. Rajwantie Sahai will provide continuous technical and managerial support to the field team. She is an international education specialist and has served as a PM on multiple USAID evaluations at Social Impact. 2. Dr. Sarah Tisch, Senior Technical Advisor (STA), has 24 years’ experience as a gender specialist, with a background in governance, civil society, education. She will is a technical expert in evaluation and will assure that the gender analysis fully reflects the situation in Ghana. As STA, Dr. Tisch will work with Ms. Sahai to conduct quality assurance checks (see Report Review Checklists in Annex B) of deliverables and provide guidance to the technical Team. 3. Home Office Program Assistant (PA), Ms. Natalie Shemwell will provide ongoing administrative support to the field team by working in conjunction with the local logistician to guide field coordination. Ms. Shemwell will also provide additional project backstopping as needed throughout the project. 4. A local Logistician based in Accra, Ghana, will provide in-country logistical support to the team. She or he will meet with and work in close consultation with the field team to schedule meetings and reach key stakeholders to collect relevant information for the evaluation. This individual will be selected based on prior experience providing logistical support to evaluation and implementation teams. Final Evaluation of the USAID/Ghana PAGE Project Final Report 30 EVALUATION METHODOLOGIES DATA COLLECTION APPROACHES The Team will use a mixed-methods approach to complete a thorough and effective performance evaluation with the timeline and budget parameters of this contract. Starting with a review of project documents, the Team will identify and plan to meet with select stakeholders in the U.S., and then travel to Ghana. While in Ghana, the team will conduct semi-structured key informant interviews and several focus groups with selected PAGE stakeholders. Additionally, PAGE partners and stakeholders will be invited to participate in a voluntary survey using a paper-based survey to the extent that time permits. These data collection methods will yield both quantitative and qualitative data and are described in greater detail below. An Evaluation Matrix in Annex C consolidates details on the data collection approach for further clarity. Document Review – The Team will review semi-annual and annual reports, monitoring data, relevant assessments and evaluations, and appropriate contextual data and information from government sources and reports. The Team will also look at reports and policy documents produced by the education system and reports written by other program implementers and independent researchers. In addition to documents provided directly by the Mission, other USAID implementing partners, and donor-funded projects in the education sector, the Team will consult secondary sources to verify the information presented in project documents. The document review will be considered a first iteration toward answering the evaluation questions. Semi-Structured Key Informant Interviews (SKII) – The Team will use a semi-structured approach to interviews, allowing for a level of standardization across interviews, as well as flexibility and adaptability to context and the particular person or group of people being interviewed. The SKII protocols will be finalized by the Team after in-depth discussion with USAID/Ghana and experts within SI. Each SKII grouping will differ depending on the key informant’s role and “causal distance” from activities, as well as the extent of the key informant’s involvement in PAGE activities and the time available for interviewing. SKIIS will be held with the following groups: 1. PAGE major partners such as CARE International, key MOE and GES officials at the national level, and other institutions including the National Steering Committee for PAGE activities 2. Ghanaian officials serving roles in the District Assemblies, the DEOCs, the DEOs and other officials within the local government 3. Community Service Organizations (CSOs) and Non-Governmental Organizations (NGOs) working at the district level in the education sector The field visits will focus on semi-structured interviews with local government officials, in districts where PAGE activities have been concentrated. While the Team aims to conduct at least 50 individual interviews, from a non-random sample and based primarily on availability, the final count will depend largely on access to necessary stakeholders and time available in the field. The Team conservatively estimates conducting SKIIs of 10 of the identified officials at the capital city level and 20 interviews in local governments at the district level. Combined with the estimated 10 interviews with CARE International and other major partners and 10 with external experts and USG personnel, we estimate completing an estimated 50 interviews while in Ghana. However, these should be received as estimates, and the final interview count will vary at the end of data collection depending on several limitations including accessibility, The SKII protocol can be reviewed in Annex D and a Key Informants (KI) draft list in Annex E. (Mini-Survey – While the Team plans to conduct site visits in and outside the capital city of Accra, the vast number and geographic distribution of PAGE activities preclude a more extensive effort given the in-country timeframe. The value of a mini-survey is that it allows for the gathering Final Evaluation of the USAID/Ghana PAGE Project Final Report 31 of evidence from otherwise unreachable KIs. First, the Team will contact PAGE partners asking questions based on question 3. The Likert-scale and open-ended questions will be sent to a selection of MOE, GES and district level officials to solicit their impressions of key challenges, achievements, strengths and weaknesses of the PAGE approach. These will be sent by surface mail, and request that they voluntarily complete the mini-survey. A draft version of the mini￾survey to PAGE partners are found in Annex G.) Focus Groups (FGs) – The Team plans to conduct FG interviews with key informants who have been trained through PAGE and who utilize the tools and practices that improve governance, supervision and accountability in the target school systems at the district level. FG participants may be held with Circuit Supervisors, Head Teachers, and members of School Management Committees and parents. SMCs/PTAs. This service delivery component is where the greatest concentration of project implementation activities is focused, and where it is possible to gather a relevant group of males and females who use the training provided by CARE International. Through the FG interviews the Team will get an overview of how systems perform at an enhanced level for improved student achievement. The FG interviews will engage participants using rapid appraisal techniques and use organizational assessment tools to gauge system capacity and performance. The qualitative responses of FG participants will be entered into an evidence table and will be used as supplemental information. These will also be used in the triangulation process described below. The FG protocol can be found in Annex H. Site Visits – Given the extensive geographic scope of PAGE, the number of key PAGE partners, and the concentration of activities in the north and southern parts of the country, the Team must use time efficiently. After arrival in Accra, the Team will work as a single unit refining its research and interview instruments based on feedback during the in-brief with USAID/Ghana’s education team. When research begins, two teams will divide into two sub-groups, in order to reach the selected districts that are covered by PAGE interventions. Please see Annex E for PAGE sample districts. According to the literature given to the Team, PAGE activities have been launched and on-going in at least 42 districts in Ghana, both in the north and the south. The team expects to finalize a sampling framework representing an equal number of districts in each part of the country after discussion with USAID. During the site visits, the Team will conduct SKIIs and FGs with project beneficiaries as described above. To the extent feasible, the Team will seek out comparative data among sites where PAGE activities were not carried out. However, given the scope of this evaluation it will not be possible to establish absolute causality or attribution. Instead, the team will use data analysis techniques to identify relevant trends toward anticipated (and unanticipated) program outcomes and results. Data Triangulation – Information gained from the SKIIs will be triangulated through use of a Team review of PAGE reports and the other-sourced documents mentioned above. Next, data from the voluntary mini-survey of PAGE partners will be reviewed. Finally, the Team will analyze potential findings across these data sources to generate synthesized findings for that activity. The PAGE performance monitoring plan (PMP) will be reviewed to analyze this data and incorporate relevant elements into interview questions and mini-surveys. As mentioned, to the extent feasible, the Team will gather comparative information in all locations.  Other education donors and or implementers established in Accra, Ghana  Visits to districts not served by the PAGE project Final Evaluation of the USAID/Ghana PAGE Project Final Report 32 Comparison groups would be selected from among NGOs and CSOs who did not work with PAGE in any significant way and who were involved with providing program services to the education sector. The final selection will be determined in consultation with USAID/Ghana and CARE. To identify any gaps between males and females in their experience with the project activities, and any implementation gaps that may have prevented PAGE from responding effectively to gender issues, the Team may conduct group interviews and/or FGs separately with males and females. All data from PAGE participants and the counterfactual sources will be triangulated against the gender analysis evidence matrix. See Annex H for illustrative Gender Analysis tools. DATA ANALYSIS For each evaluation question, the Team will organize and disaggregate data obtained from different methods and sources, review it for reliability and validity, and triangulate it to compare findings based on multiple methods, forms of data, sources of data and levels of data or respondents. The Team will use sex-disaggregated data to identify any differences between males and females. The Team will use the ‘domains of analysis’ approach to identify and review how PAGE tools and approaches helped the district of education partners identify and address the different voices of male and female actors in their roles to improve governance, supervision and accountability in education. USAID recommends the domains approach as it covers laws, policies, and institutional practices; cultural beliefs and norms; gender roles, responsibilities and time used; assets and resources’ and patterns of power and decision-making.8 The Team will use an evidence matrix and gender findings table that Social Impact has adapted from two USAID publications and successfully used similar evaluations.9 Data analysis approaches relevant to each evaluation question are identified in the Evaluation Matrix and further explained in Annex J. LIMITATIONS The mixed-methods approach outlined in ‘Data Collection Approaches’ section is designed to overcome several limitations identified by the Team during the proposal and work planning stages. These limitations are indicated below with an explanation of the Team’s plan to address each. Bias – The Team will address bias by using multiple sources of data to triangulate on an evaluation issue, with the assistance of qualitative evidence matrices. By combining information from multiple sources, i.e., documents, interviews, and mini-survey data, the effect of biases on the analysis will be mitigated as much as possible. Another approach that pertains particularly to interviews will be the use of questions about specific examples of knowledge use. This will help the Team probe general responses more thoroughly. Finally, the inclusion of key informants with different ‘causal distances’ from the activity, i.e., indirect beneficiaries and external experts, will provide evidence from different perspectives. If possible, the Team will gather information from non-participating 8 See ADS Chapter 205 Gender Equality and Female Empowerment In USAID’s Project Cycle, p. 8-9 http://www.usaid.gov/ads/policy/200/205. 9 The matrix and findings tables are derived from Deborah Caro (2009) A Manual for Integrating Gender into Reproductive health and HIV Programs: From Commitment to Action. USAID/Interagency Gender Working Group, p. 22 and 27. http://www.prb.org/igwg_media/manualintegrgendr09_eng.pdf; and Final Evaluation of the USAID/Ghana PAGE Project Final Report 33 individuals and organizations. Specific limitations are listed below, along with proposed strategies for mitigation. It is possible that some interviews will be conducted by an individual Team member instead of two. The Team asserts that in this evaluation approach such scenarios will not introduce a significant source of bias. First, the SKIIs will be based on a standard protocol for use in all interviews by every Team member. These will be checked by other Team members for completeness and clarity. Second, any particular finding of fact or conclusion must be based on multiple points of evidence for any data source type. Finally, the analysis is conducted by the Team as a whole, using summary note templates and group-based discussions. As would be the case anywhere in the world, it is possible that translation mistakes may distort evidence. The Team will hold discussions about SKIIs and focus groups conducted each day. The Team feels this potential risk is mitigated to a large extent by the presence of Ghanaian speakers on the Team, the use of professional translators as needed, the application of best practices when engaging with professional translators, and as noted above, a Team-based approach to analyzing evidence. The Team will disaggregate data by sex to pay particular attention to equality gaps between males and female project participants and beneficiaries, specific project sites, and other key factors. To the extent possible, the Team will code the qualitative data collected so the frequency of responses can be measured. Inability to explore causality – The M&E plan for the PAGE project did not include random selection of treatment and control groups, and therefore it is not possible to attribute impact to this project alone. This evaluation will explore causal links, but will be unable to authoritatively ascribe impact or a direct causal relationship between observed outcomes and PAGE. The evaluation will focus less on causality than on assessing the project’s contribution to intended outcomes for Components I through V, taking into consideration other events and development actors at work during the same time period. Qualitative research – While providing rich, in-depth information, the qualitative data will be subject to the standard limitations of qualitative research. First, evaluators often encounter recall bias. Male and female district education personnel including teachers and members from School Management Committees (SMCs) and Parent Teacher Associations (PTAs) may respond to questions with answers blending their experiences into a composite memory, or from past trainings by other donor-funded organizations. For Oversight Committees or DEOs that engaged with PAGE, there may have been a number of other initiatives from other donors providing training or project funding at the same time. Additionally, training may have taken place sometime in the past, so respondents are not able to provide the level of detail needed in an evaluation. Second, response bias is a common problem for evaluations. For example, education personnel such as Circuit Supervisors may provide the interviewer with positive remarks because they hope that the resources and training they received will continue in the future, regardless of the effect of that activity. We fully expect that these district level educational staff, parents, local government officials, and other stakeholders may understand that a negative evaluation could mean the end of a project that provided them with needed training, materials, or other benefits. Third, selection bias in the form of contacts provided by CARE International could mean that the Team may only hear from people with positive experiences. With USAID assistance it is working with CARE to identify 46 target district into three zones – Northern, Middle and Southern; and into 3 categories – high, satisfactory and low performing districts. SI will then select its sample from this. To mitigate these risks the Team will triangulate qualitative data gathered from interviews and focus groups with quantitative data provided by PAGE and collected during the evaluation from secondary sources. The Team will seek USAID and PAGE guidance regarding which individuals to interview and will also seek out individuals beyond those suggested by project staff to broaden the sample. The Final Evaluation of the USAID/Ghana PAGE Project Final Report 34 interview protocols are presented in the Annex and will be pre-tested in either Accra or a nearby town in which grantees are located to mitigate risk of response bias and identify potential instrumentation issues. Baseline Limitation – The Team will work with the existing base line data and if needed work with education data generated by the Government of Ghana or from other sources. The Team will find other secondary sources to permit comparison with evaluation findings. Critical Dependencies – The team has identified several key actions and information that it will depend on receiving or having completed by the USAID/Ghana COR (and education team) and the implementing partner, CARE. These are as follows:  Regarding the document review: The Team has received some documents from the Mission to begin a thorough literature review. However, it will be critical to receive any additional data and documents from the implementing partner that will support planning and the refinement of evaluation tools such as a list of project sites and any pre-selected/identified comparative sites.  Regarding accessibility to stakeholders for data collection needs: As early as possible and prior to fieldwork, it will be important that the team receive contact and location information of government and district officials, school level stakeholders such as Head Teachers and PTA/SMC leaders, and targeted schools from the implementing partner. This will support the finalization of a sampling framework, allow the team to pilot and disseminate the electronic survey ahead of field work, and finalize a travel and meeting schedule. Early receipt of this information will increase accessibility and maximize time needed for logistical planning and data collection.  Regarding data analysis and triangulation: The Team’s ability to identify and meet with viable comparison groups in the field will depend on early consultations with the USAID/Ghana Education Team and the CARE COP (or relevant staff person). The extent to which true comparison groups/sites are identified will also depend on geographic accessibility and scheduling feasibility as well as the responsiveness of relevant points of contact. MANAGEMENT AND IMPLEMENTATION STRATEGY SUMMARY OF SCOPE AND METHODOLOGIES The USAID SOW specified that this evaluation focus exclusively on PAGE Components I through V with emphasis on the specific questions stated above. The northern region is the most disadvantaged and challenged by its climate and lack of access to resources. The Team is carefully considering, with the help of USAID and CARE International, the selection of sites with their ability to yield the information needed by USAID. The Team has carefully addressed requirements indicated in the evaluation SOW and understands USAID’s main interests in knowing: 1. The capacity of circuit supervisors, head-teachers and School Management Committees (SMCs) to monitor, manage and report on school management performance and the degree to which these strengthen educational governance and supervision; and 2. The capacity of DEO and DEOCs to support and enable the competencies of circuit supervisors, head-teachers and SMCs for school improvement and improved student achievement. Each of these issue areas will be addressed through the methodologies associated with the five evaluation questions listed in the ‘Evaluation Purpose and Questions’ Section. As aforementioned, an Evaluation Matrix that organizes each data collection method alongside each question area can be found in Annex C. The Final Evaluation Report will be informed by methodological and analysis inputs included in the matrix. When possible, FG interviews and KIIs will be designed to reflect the Final Evaluation of the USAID/Ghana PAGE Project Final Report 35 perspective of both PAGE partners and beneficiaries. While direct attribution will not be possible to measure, the Team will strive to make causal linkages wherever possible, taking into account the development actors and their circumstances. IMPLEMENTATION APPROACH The Team’s methodological approach and other requirements of the SOW will be implemented according to three implementation phases as follows: Phase I – Field Work Preparation and Initial Data Collection10 1. Review and organize project reports to better portray and summarize actual project activities by type, region, and issue/subject. 2. Prepare and disseminate an e-survey (and as needed paper survey) to government level personnel in the education sector who use computers and email, completed by January 28, 2014. 3. Identify initial list of Ghana-based stakeholders, partners and informed experts. 4. Begin scheduling interview appointments for: third week of January 2014 a. Meet with stakeholders located in Accra, Ghana identified by USAID and CARE International b. Contact CARE International Regional Centers to request assistance in assembling membership/ beneficiary focus groups. 5. Draft initial structured SKII and FG questions. 6. Begin contacting PAGE stakeholders and school level workers in different site visit locations for interviews. 7. Finalize preliminary report outline and assign major analysis and writing assignments among team members. Phase II – Field Work: Ghana 1. Data Collection: Accra a. Meet with USAID (January 20) and CARE International (January 21). b. Refine work plan, data collection instruments, and gender analysis tables based on briefing discussion and results of e-survey. c. Prepare Domains of Gender Analysis tables and populate based on pre-arrival information. d. Conduct Accra-based SKIIs (see expected numbers discussed in Section 2.1 under the “semi-structured key informant interview” method), remaining open to additional recommendations from local sources. The Evaluation Specialist will continue with Accra-based data collection as needed as the rest of the Team travels to the northern districts. e. Identify additional site visits outside of Accra and request support from CARE International regional centers in organizing focus groups and SKIIs. 2. Site Visits Outside Accra a. All four Team members will conduct initial data collection activities in Accra. Dr. Rowley, Ms. Sinclair, Mr. Adams and Mr. Nornoo. Once the protocols have been agreed upon and the Team is confident of their quality the northern team, Ms. Sinclair 10 Note that gender-focused questions/data will be an integral part of all data collection formats used by the Team. Final Evaluation of the USAID/Ghana PAGE Project Final Report 36 and Mr. Adams will depart for their destinations. The teams will further be deployed depending on the zones set up by USAID in consultation with CARE International. b. Regional resource centers will help to organize meetings of local members/beneficiaries for mini-surveys, SKIIs and focus group discussions. The Team is currently organizing these meetings. c. Additional stakeholders, including district authorities, educational leaders, and other PAGE stakeholders, will be interviewed to assess PAGE partner effectiveness and impact. 3. De-brief and Presentation of Preliminary Findings and Analysis a. Responses to the survey will be used to triangulate and refine findings from the SKIIs and project document and secondary data review. b. Team conducts preliminary analysis of data from document review, surveys, SKIIs, and focus group discussions. c. Team provides USAID and then CARE International with preliminary findings, conclusions and recommendations February 20 and 21, 2014. Team provides presentation of findings to key GoG officials in Accra. Team departs Accra on week beginning February 22, 2014. Phase III – Reporting and Dissemination 1. Team prepares and submits draft evaluation report following guidelines indicated in SOW. 2. USAID to review, provide additional information/clarifications, and comments. 3. Team addresses USAID input and submits final draft for submission. 4. USAID approves and accepts final draft. Once final report is accepted by USAID, Social Impact will submit all data used and collected for the evaluation. 5. SI submits final report to Development Experience Clearinghouse (DEC). 6. USAID prepares a Contractor Performance Assessment Report (CPAR). Table 1: Schedule of Activities presents the dates during which evaluation activities and tasks will be conducted/ completed. Table 1: Schedule of Activities Task/ Deliverable Period of Performance Kickoff meeting with USAID Upon Award Review background documents; preparation work (offshore) December 16 to 20 Team Planning Meeting hosted by SI December 16, 2013 Evaluation team members travel to Ghana January 18 In-brief with USAID/Ghana January 21 Submit revised final work plan January 21 Data collection January 22 to February 21 Preparation for presentation February 17-18 Presentation and debriefing with USAID/Ghana February 20 Debriefing with key stakeholders February 21 Expatriate team members depart Accra February 22 SI submits draft report to USAID/Ghana February 21 USAID/Ghana comments on draft report February 27 Evaluation Team revises draft report February 27 SI delivers final report March 11 Final Evaluation of the USAID/Ghana PAGE Project Final Report 37 MANAGEMENT APPROACH Communication and Scope Management The Team will manage its evaluation approach through the following actions: 1. Maintain close communication with USAID/Ghana with calls to the COR at least once per week. 2. Involve PAGE and Ghanaian partners in a collaborative effort. 3. Seek alternative perspectives and views from relevant non-stakeholders. 4. Integrate a gender framework in all data collection and analysis formats. 5. Retain focus on SOW questions, but be alert to un-anticipated consequences. 6. Strive to capture and present instances of “probable causality” with respect to PAGE impact11 . 7. Triangulate qualitative and quantitative data during analysis to verify key findings and conclusions. 8. Provide USAID with evidence-based conclusions and actionable recommendations at strategic, programmatic and project levels. 9. Fulfill Team roles and responsibilities as discussed in the ‘SI Evaluation Team Roles and Responsibilities’ Section of Work Plan. The evaluation management strategy outlined above will maximize the time and efficiency of the Team during data collection and ongoing data analysis while in Ghana. Quality Assurance Procedures During each of the three evaluation phases, the Social Impact quality assurance (QA) process is used to ensure high quality, evidence-based results that are useful for program improvement and learning. This feedback is provided to the Team Leader who then discusses it with the team and if needed, USAID. Phase I: Evaluation Planning – The STA will specifically review the feasibility and rigor of the proposed evaluation design, work plan, data collection tools and protocols. Special attention is given to ensuring that analytic tools are used to identify gaps between males and females and how vulnerable groups, such as ethnic and social minorities are incorporated into the evaluation design. The STA and PM will review the evaluation tools and plan using SI’s 25-point gender check list to ensure that gaps in the experiences between males and females identifiable and addressed. Phase II: Field Work – The PM and Program Assistant (PA) will work closely with the four temporarily, field-based key personnel and the Accra-based local logistician to coordinate logistical needs to and respond to team needs in a timely and efficient manner. A schedule of meetings and activities will be updated and circulated with the team by the logistician on a daily basis. The PM will oversee that evaluation activities abide by the final, USAID/Ghana-approved Work Plan and the awarded budget in a cost-conservative manner. Phase III: Reporting –Reports are comprehensively assessed by the STA and PM to ensure structure and logical linkages between the findings, analysis, conclusions, presentation of qualitative and quantitative data, and practical recommendations. The PM assists in this process by verifying that data is accurately calculated and presented, and in copy-editing and formatting the report. For the final 11 It is not within the scope of this evaluation to attribute absolute causal linkages, but it is possible to analyze data such that potential causal linkages are highlighted. Final Evaluation of the USAID/Ghana PAGE Project Final Report 38 report the STA and PM will use a 45-point quality check of the executive summary, program and methodology description; adequacy of findings, analysis, conclusions, and final recommendations; compliance with the USAID Evaluation Policy; and overall report presentation. In addition, the HQ team will use SI’s 25-point gender check list for evaluating gender impacts again to review the final report. Final Evaluation of the USAID/Ghana PAGE Project Final Report 39 ANNEX I, PART A: EVALUATION SCOPE OF WORK Background At the time of the award of the PAGE Project (June 2010), USAID Ghana’s Education Assistance Objective 08 sought “to improve quality of, and access to, basic education.” This goal was set within the larger context of Ghana’s education decentralization policy that mandates that District Assemblies have executive responsibility for the provision and management of basic and second cycle schools. With a decentralized education system, schools will attain the autonomy needed for effective education delivery. To improve student achievement and school performance in basic schools, district and community level require good governance to strengthen accountability, enhance participation and break down inequalities in education. In Ghana, improved education governance through decentralized authority has been identified as a key to academic achievement. The Ministry of Education (MOE) is committed to decentralization, but is concerned that districts lack the capacity to properly impact learning outcomes and that communities lack the ability to hold them accountable. It is to meet these strategic goals and evolving needs that the PAGE project was developed; using lessons learned from previous USAID funded programs such as Government Accountability Improves Trust (GAIT II) and the Quality Improvement in Primary Schools (QUIPS) programs. On June 3, 2010, USAID/Ghana signed a three-year $7,417,030 Cooperative Agreement with CARE International to implement the PAGE project, with an expected end time of June 2, 2013. The PAGE Project approach engenders sustainable improvements in education management that will directly impact student learning achievements by working collaboratively with district assembly (DA) and education officials, head-teachers, School Management Committees and Parent Teacher Associations to strengthen school governance and supervision activities in 46 targeted districts. The project’s goal is “improved student achievement through strengthened educational governance and supervision”, with the following intended results: 1. Improved capacity of District Education Oversight Committees (DEOCs) to contribute more effectively to improving school governance and supervision activities within their districts; 2. Strengthened DEOC and District Education Office (DEO) collaboration and responsiveness to school management issues in their respective districts; 3. Improved competence of circuit supervisors, head-teachers and School Management Committees to monitor, manage and report on school management performance in order to increase student achievement; 4. Appropriately resourced circuit supervisors having essential logistics and equipment to implement an improved and focused supervision program for schools; and 5. Active and engaged Parent Teacher Associations and School Management Committees trained to seek and interpret school performance reports with the view to developing and implementing strategies for improving school management systems and student achievement. Objective The objective of the contract is to conduct a final performance evaluation of the PAGE project. Findings and recommendations will be incorporated into USAID/Ghana’s short, medium and long term program planning and execution. The purpose of the evaluation is to test the USAID/Ghana Education Office’s hypothesis that without effective decentralized systems with functional governance and supervision at the district and community levels, interventions to support learning outcomes, specifically reading, cannot be implemented effectively at a national scale. Therefore, underscoring the conviction that if instruction is enhanced, if it is supported through strengthened basic education management systems, and if education sector governance institutions are more accountable and transparent to citizens who can engage in school oversight, then children’s learning Final Evaluation of the USAID/Ghana PAGE Project Final Report 40 outcomes, specifically children’s reading performance in primary schools, will improve (see Annex B for the DO4: Education Results Framework). The final performance evaluation will focus on how PAGE has been implemented, what it has achieved, whether expected results have occurred according to the project’s design and implementation. Following a performance audit conducted on the PAGE project in February, this Final Performance Evaluation will also provide an opportunity to do an in-depth investigation and analyses of the issues, document project performance, results, impacts, lessons learned and best practices from various stakeholder perspectives as a critical contribution to USAID’s knowledge management efforts. The results will be used to inform USAID Ghana in the short, medium and long term on areas that require strengthening or possible reductions of scope of investments specifically in relation to how activities are perceived, valued, and sustained. The evaluation must comply with the January 2011 USAID Evaluation Policy, available at http://www.usaid.gov/evaluation. In order to achieve the results of the program, the following four activity components are directly interlinked with the objectives of the program. The PAGE program works with district and school community level partners in the Ghana education system to improve student achievement through the strengthening of their education governance and supervision functions. Component One: Stakeholder outreach Participation and engagement of all stakeholders in the district is critical for ensuring ownership and the sustainability of interventions. Component Two: DEOC-DEO collaboration building and development of district specific framework for education governance and supervision This component seeks to strengthen district education oversight structures through supporting DEOCs and DEOs to create an enabling environment for improved collaboration and responsiveness to school management issues; work with the DEOs to improve school supervision, and promote the institutionalization of participatory processes in governance activities in school management for the purpose of improving education delivery and student achievement in schools. Component Three: Training and resourcing of circuit supervisors and head-teachers and school supervision program Effective school supervision has a direct impact on quality and student achievement. Circuit supervisors are responsible for the supervision of all schools in their assigned circuits. Major challenges facing the circuit supervisors in the conduct of this function include lack of training in supervisory approaches and mobility limitations. Component Four: SMC/PTA capacity building School Management Committees in a significant number of school communities are defunct. SMCs in a number of schools have been absorbed into the Parent Teacher Associations (PTAs) to keep the institution alive in name. Evaluation Design and Methodology This is a final performance evaluation and is focused on PAGE implementation, results, outcomes in relation to the development hypothesis, and how activities are perceived, valued, and sustained by stakeholders. Evaluators will use a mix of quantitative and qualitative data collection and analysis methods to arrive at findings and recommendations based on the provided evaluation questions. The evaluation Team should consider a range of possible methods and approaches for collecting and analyzing the information required to achieve the evaluation objectives. The evaluation Team shall share data collection tools with USAID for review, feedback and/or discussions with sufficient time for USAID’s review before they are applied in the field. Audience and Intended Users Final Evaluation of the USAID/Ghana PAGE Project Final Report 41 The audience of the evaluation report will be the USAID/Ghana Mission, specifically the Education Office team, the Ministry of Education (MOE), the Ghana Education Service (GES), development partners and the implementing partner, CARE. It is expected that GES, District Education Offices, school head teachers and community stakeholders will have the opportunity to discuss how USAID’s support through PAGE assisted them and how future efforts could better assist them. Evaluation Questions The evaluation will focus on four primary areas of interest regarding the performance of the PAGE project. Via the evaluation questions, USAID Ghana is interested in learning about the performance, results and impact of PAGE. • Have governance and supervision interventions at the district level resulted in improved student achievement? If so, to what extent and why? If not, why not? What intended and unintended contributions, results and/or impact has the PAGE approach and activities achieved relative to improving governance and supervision in schools? How? • What factors affected the achievements and results of the project? What can be identified as lessons learned and best practices and from which stakeholders or beneficiaries? • What are the best ways to ensure that progress and results are captured and/or continued which also promote ownership, engagement and sustainability of interventions after it has ended? • What performance monitoring processes, systems and tools were used to ensure accurate, timely, reliable and valid performance and indicator tracking, reporting and feedback at all levels? How did this contribute to logical implementation of activities, data validation, documentation and review of work plans, indicators and /or activities? Final Evaluation of the USAID/Ghana PAGE Project Final Report 42 ANNEX I, PART B: SOCIAL IMPACT’S EVALUATION REPORT CHECKLISTS SENIOR TECHNICAL ADVISOR’S CHECKLIST FOR ASSESSING FINAL EVALUATION REPORTS EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments STRUCTURE OF THE REPORT 1. Is the report well-organized (each topic is clearly delineated, subheadings used for easy reading)? 2. Is the report well written (clear sentences, reasonable length paragraphs, no typos, acceptable for dissemination to potential users)? 3. Does the report adequately address all the evaluation questions in the SOW? 4. Does the evaluation report discuss any issues of conflict of interest, including the lack thereof? 5. As applicable, does the evaluation report include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team? EXECUTIVE SUMMARY 6. Does the evaluation report begin with a 3- to 5-page stand￾alone summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if applicable) of the evaluation? 7. Does the Executive Summary concisely state the main points of the evaluation? 8. Does the Executive Summary follow the rule of only saying what the evaluation itself says and not introducing new material? INTRODUCTION 9. Does the report introduction adequately describe the project? 10. Is the purpose of the evaluation clearly stated? 11. Is there a clear statement of how the evaluation will be used and who the intended users are? 12. Are the priority evaluation questions presented in the introduction? SCOPE AND METHODOLOGY 13. Does the report provide a clear description of the evaluation’s design? 14. Does the report state the period over which the evaluation was conducted? 15. Does the evaluation address all evaluation questions included in the Statement of Work (SOW)? 16. Does the methodology include gender analysis? 17. In answering the questions, does the report appropriately use comparisons made against baseline data? 18. If the evaluation is expected to influence resource Final Evaluation of the USAID/Ghana PAGE Project Final Report 43 EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments allocation, does it address cost structure and scalability of the intervention, as well as its effectiveness? 19. Is there a clear description of the evaluation’s data collection methods (summarized in the text with the full description presented in an annex)? 20. Does the evaluation report contain a section describing the limitations associated with the evaluation methodology (e.g. selection bias, recall bias, unobservable differences between comparator groups, small samples, only went to villages near the road, implementer insisted on picking who the team met with, etc.)? 21. Does the evaluation scope and methodology section address generalizability of the findings? FINDINGS 22. Are FINDINGS specific, concise and supported by strong quantitative and qualitative evidence? 22.1. As appropriate, does the report indicate confirmatory evidence for FINDINGS from multiple sources, data collection methods, and analytic procedures? 23. Are adequate data provided to address the validity of the “theory of change” or development hypothesis underlying the project, i.e., cause and effect relationships? 24. Are alternative explanations of any observed results discussed, if found? 25. Are unplanned results the team discovered adequately described? 26. Are opinions, conclusions, and recommendations kept out of the description of FINDINGS? ANALYSIS AND CONCLUSIONS 27. Are charts and graphs used to present or summarize data, where relevant? 27.1. Are the graphics easy to read and simple enough to communicate the message without much text? 28. Is there a clear distinction between CONCLUSIONS and FINDINGS? 29. Is every CONCLUSION in the report supported by a specific or clearly defined set of FINDINGS? 30. Are the CONCLUSIONS credible, given the FINDINGS the report presents? 31. Can the reader tell what CONCLUSIONS the evaluation team reached on each evaluation question? RECOMMENDATIONS 32. Are RECOMMENDATIONS separated from CONCLUSIONS? (Are they highlighted, presented in a separate section or otherwise marked so that the reader sees them as being distinct?) 33. Are all RECOMMENDATIONS supported by a specific or clearly defined set of FINDINGS and CONCLUSIONS? (Clearly derived from what the evaluation team learned?) 34. Are the RECOMMENDATIONS practical and specific? 35. Are the RECOMMENDATIONS responsive to the purpose of the evaluation? Final Evaluation of the USAID/Ghana PAGE Project Final Report 44 EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments 36. Are the RECOMMENDATIONS action-oriented? 37. Is it clear who is responsible for each action? 38. Are the RECOMMENDATIONS limited/grouped into a reasonable number? LESSONS LEARNED 39. Did this evaluation include lessons that would be useful for future projects or programs, on the same thematic or in the same country, etc.? 40. Are the LESSONS LEARNED highlighted and presented in a clear way? 41. Does the report indicate who the lessons are for? (e.g., project implementation team, future project, USAID and implementing partners, etc.) BOTTOM LINE 42. Does the evaluation report give the appearance of a thoughtful, evidence-based, and well organized effort to objectively evaluate what worked in the project, what did not and why? 43. Is the evaluation report structured in a way that will promote its utilization? 44. Does the evaluation report explicitly link the evaluation questions to specific future decisions to be made by USAID leadership, partner governments and/or other key stakeholders? 45. Does the evaluation report convey the sense that the evaluation was undertaken in a manner to ensure credibility, objectivity, transparency, and the generation of high quality information and knowledge? REPORT DISSEMINATION 46. Has a dissemination plan been developed for this report? Final Evaluation of the USAID/Ghana PAGE Project Final Report 45 SOCIAL IMPACT CHECKLIST FOR GENDER INTEGRATION IN EVALUATIONS PAGE Evaluation December 2013 to March 2014 S. Dunham Rowley, Team Leader; Brenda Sinclair, Saaka Adams, Charles Nornoo Good Adequate addressed Poor or not applicable Not 1. Conceptual framework: and research design 1-1 Evaluation includes a gender analysis framework 1-2 Evaluation identifies constraints between males and females, where appropriate 1-3 Stakeholder consultations with all key groups, including groups of females 1-4 Use of rapid assessment/diagnostic studies during evaluation design, where appropriate 1-5 Ensure focus on closing gaps between males and females not just women and not just men. 2. Organization of the research 2-1 Both sexes included at all levels of research team 2-2 Male and female local language speakers involved 3. Sample design 3-1 Both male and female household members interviewed of different ages, as appropriate 3-2 Special modules to interview other (non-household head) male and female members of the household 3-3 Monitoring who participates (both attends and speaks) in community meetings 3-4 Follow-up sample if key and under-represented male and female groups missing 3-5 Focus groups selected to ensure all key and under-represented groups with male and female members represented 3-6 Follow-up sample for missing males and females who identify with targeted groups 4. Data collection methods 4-1 Data collected (where appropriate) on both sexes 4-2 Key development gaps between males and females are covered 4-3 Information on division of labor between males and females, of all ages as appropriate 4-4 Time use patterns of males and females of all ages, as appropriate 4-5 Control of resources of males and females in a household, group or organization 4-6 Information collected about, and from, different male and female household members 4-7 Use of qualitative data collection methods where required. 4-8 Mixed method data collection strategy 4-9 Systematic use of triangulation to verify what males say about females and females say about males 4-10 Focus groups are held in locations accessible to women 4-11 If talking about sensitive topics, focus groups of exclusively females and exclusively males are held Final Evaluation of the USAID/Ghana PAGE Project Final Report 46 4-12 Data collected from both married and unmarried males and females 5. Data analysis and presentation 5-1 Ensure sex-disaggregation of data. 5-2 Follow-up (if possible in the field) when triangulation reveals inconsistencies between information gathered from males and from females. 5-3 Ensure findings reach, and are commented on, by all key groups (including groups representing both men and women) Final Evaluation of the USAID/Ghana PAGE Project Final Report 47 ANNEX I, PART C: EVALUATION MATRIX PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal Question 1: Have governance and supervision interventions at the district level resulted improved student achievement? If so, to what extent and why? If not, why not? Objective 1 Stakeholder Outreach  Activity materials, baseline assessment; Functional Organizational Assessment Tool (FOAT); Audit Report; DEO reports   District Assemblies; District Education Oversight Committees; Community members and Parents GoG Education policy Government workers with Internet; CBO and International NGOs Overarching Research Hypothesis Goal: If educational governance and supervision in basic schools is improved, then student achievement will also improve. H1: If the capacity of circuit supervisors, head-teachers, and SCMs to monitor, manage, and report on school management performance is strengthened, then educational governance and supervision will improve. H2: If the capacity of DEOCs to support and enable the competencies of circuit supervisors, head-teachers, and SMCs is strengthened, then educational governance and supervision will improve. Final Evaluation of the USAID/Ghana PAGE Project Final Report 48 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal PMP; performance indicator results SMC members Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision Training materials; and reports; District annual plans; Manuals; procedures; DA or DEOC policies   DA, DEOC, DEO Community members and Parents Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program Manuals, training materials and reports DEO procedure revisions on Supervision   DEO and Circuit Supervisors and Head Teachers Pupils, teachers Parents and community members Objective 4 SMC/PTA capacity building SMC manuals that were distributed; Training plans and reports SMC and School Plans and sample meeting reports and minutes   SMC/PTA members Parents of children at the school NGOs working in Education in the District Question 2: What intended and unintended contributions, results and/or impact has the PAGE approach and activities achieved relative to improving governance and supervision in schools? Objective 1 Stakeholder Outreach  USAID Audit Report; Performance indicator Public surveys;  DA and DEOC Parents of children at the Final Evaluation of the USAID/Ghana PAGE Project Final Report 49 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal results; PMP; Quarterly and Semi￾annual reports records/meeti ng minutes; school Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DEO reports  DEO; DEOC DA; parents of students attending school NGOs working in Education in the District Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports CS and Head Teacher reports; DEO reports  CS and Head Teachers; DEO personnel DA; parents of students attending school NGOs working in Education in the District Objective 4 SMC/PTA capacity building USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports School level reports; DEO reports  CS and Head Teachers; DEO personnel; Parents and SCM members parents of students attending school NGOs working in Education in the District Question 3: What factors affected the achievement and results of the project? What are the lessons learned and best practices, and from which stakeholders or beneficiaries? Objective 1 Stakeholder Outreach  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi-   DA; DEOC Schools and Parents NGOs working in Education in the DEO and GES personnel with Internet access Final Evaluation of the USAID/Ghana PAGE Project Final Report 50 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal annual reports District Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports Evaluations of training/traini ng reports   DA; DEOC; GES; MOE; DEO Schools and Parents NGOs and CSOs working in Education in the District DEO and GES personnel with Internet access Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports Evaluations of training/ training reports DEOC; DEO; SCs; and HT Schools and Parents NGOs and CSOs working in Education in the District DEO and GES personnel with Internet access Objective 4 SMC/PTA capacity building USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports SMC/PTA meeting minutes SMC/PTAs; school personnel; DEO; DEOC Schools and Parents NGOs and CSOs working in Education in the District DEO and GES personnel with Internet access Question 4: What are the best ways to ensure that progress and results are captured and/or continued that also promote ownership, engagement and sustainability of interventions after the project ends? Objective 1 Stakeholder Outreach  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾DEO reports   DA, DEOC, DEO Schools and Parents NGOs and CSOs working in Education Final Evaluation of the USAID/Ghana PAGE Project Final Report 51 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal annual reports in the District Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DEO reports   DA, DEOC, DEO Schools and Parents NGOs and CSOs working in Education in the District Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DEO reports DA, DEOC, DEO Schools and Parents NGOs and CSOs working in Education in the District Objective 4 SMC/PTA capacity building USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports School reports DA, DEOC, DEO Schools and Parents NGOs and CSOs working in Education in the District Question 5: What performance monitoring processes, systems and tools were used to ensure accurate, timely, reliable and valid performance and indicator tracking reporting and feedback at all levels? How did this contribute to logical implementation of activities, data validation, documentation and review of work plans, indicators and/or activities? Objective 1 Stakeholder Outreach Quarterly and Semi￾annual reports DA, DEOC, DEO Schools and Parents Objective 2 USAID Audit Report; DA, DEOC, Schools and Final Evaluation of the USAID/Ghana PAGE Project Final Report 52 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal DEOC-DEO collaboration building and development of district specific framework for education governance and supervision Performance indicator results; PMP; Quarterly and Semi￾annual reports DEO Parents Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DA, DEOC, DEO Schools and Parents Objective 4 SMC/PTA capacity building USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DA, DEOC, DEO Schools and Parents Final Evaluation of the USAID/Ghana PAGE Project Final Report 53 ANNEX I, PART D: DRAFT SEMI-STRUCTURED KEY INFORMANT INTERVIEW PROTOCOL CARE International and their Implementing Partners at Central Level MOE, GES CARE International and their Implementing Partners at District Level DA, DEOC, DEO, School Supervisors CARE International Implementing Partners at School/Community Level School Heads, School Management Committee and PTA Heads External Experts, Other Donor Staff, USG Personnel Question 1: Have governance and supervision interventions at the district level resulted in improved student achievement? If so, to what extent and why? If not, why not? Objective 1 Stakeholder Outreach Are stakeholders aware of governance and supervision interventions at the district level? Are they aware of the impact of improved supervision on student achievement? Are stakeholders at the district level aware of governance and supervision interventions at the district level? Are they aware of the impact of improved supervision on student achievement? Have the project activities in your areas resulted in improved student achievement? If so, what measures are stakeholders aware of that convinces them of the improvements? In the last three years have you seen improvements in the school system such that student achievement is higher now? If so, why, or why not? Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision Are stakeholders aware of the existence of a DEOC-DEO collaborative framework for education governance at the district level? Are they aware of the impact of the framework on education governance and supervision at the district level and its effects on student achievement? Are district stakeholders aware of the existence of a collaborative relationship between the DEO and the DEOC in the district? If yes, could Distract stakeholders describe the relationship? If not, why not? Has the framework resulted in any way in the improvement of students' achievements? What is the relationship between the DA, DEOC, the DEO and the Schools and their SMCs? Are school communities now more involved in the operations of the school than in the past? If yes, please explain the nature of involvement, and if no, why not? Has this relationship resulted in improved school supervision, and how has that resulted in improved Are you aware of any collaborative framework for education governance and supervision of the district and how has this contributed to student achievement? Final Evaluation of the USAID/Ghana PAGE Project Final Report 54 student achievement in your school? Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program What role(s) do circuit supervisors play in the improvement of how the DEO now operates? Did circuit supervisors and head-teachers receive training, and have they imparted their new skills in any way that has improved quality of teaching and learning at the school level? What role(s) do the Head￾Teachers play in the improvement of how the school delivers education at the school level now? Are you aware (point to a time) Head-teachers received training that led to new practices to improve instruction and organization at the school level/ Have you seen any improvement in the capacities of Circuit Supervisors and Head-Teachers to supervise and manage schools effectively, and has that contributed in any way to student achievement? Objective 4 SMC/PTA capacity building Are District stakeholders aware that SMCs/PTAs have received capacity building assistance? Is there evidence that they have utilized their new capacity to influence education supervision at the school level? What role does the SMC/PTA play at the school now? How has this contributed to student achievement? Is the community substantially more involved now? If yes, in what ways, please describe? Question 2: What intended and unintended contributions, results and/or impact has the PAGE approach and activities achieved relative to improving governance and supervision in schools? Objective 1 Stakeholder Outreach What were the most important contributions (intended and unintended) of the PAGE approach to improving governance and supervision at the National and District levels? What were the most important contributions (intended and unintended) of the PAGE approach to improving governance and supervision at District levels? What were the most important contributions (intended and unintended) of the PAGE approach to improving school management and supervision? What were the most important contributions (intended and unintended) of the PAGE approach to improving governance and supervision at the National and District levels? Final Evaluation of the USAID/Ghana PAGE Project Final Report 55 Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program What expected or unexpected results/impacts were created as a result of increased capacities of Circuit Supervisors and Head￾teachers in performing their roles effectively at the District and school level. What results and or impacts has the PAGE activities achieved relative to improving capacities of Circuit Supervisors and Head￾teachers to perform their roles effectively at the district level? Which of the results/impacts were less important, and which of the results/impacts were important but unexpected? Have the PAGE activities in your areas resulted in improved supervision and school management by the Circuit Supervisors and School Head￾teachers? Which of the results do you think were important but unexpected and which ones were achieved but less important? What were the most important contributions, results and impacts (intended and unintended) of the PAGE activities to improving the capacities of Circuit Supervisors and Head-teachers in supervising and managing schools in their jurisdictions? Objective 4 SMC/PTA capacity building Are there any intended or unintended results/impacts created at the SMC/PTA level as a result of the PAGE program’s capacity building activities? Are there evidence that the PAGE activities have resulted in improved capacities of SMCs/PTAs in your Districts? Which of the results/impacts by the PAGE activities were intended and which ones were unintended? What results and or impacts has the PAGE activities achieved relative to improving capacities of SMCs/PTAs to supervise teaching and learning at the school? Which of the results/impacts were less important, and which of the results/impacts were important but unexpected? What were the most important contributions, results and impacts (intended and unintended) of the PAGE activities to improving the capacities of SMCs/PTAs in supervising teaching and learning in schools at the community level? Question 3: What factors affected the achievements and results of the project? What are the lessons learned and best practices, and from which Final Evaluation of the USAID/Ghana PAGE Project Final Report 56 stakeholders or beneficiaries? Objective 1 Stakeholder Outreach What factors do you think affected the coordination of stakeholder engagement at the District level? What factors do you think accounted for the non￾achievement of broad project objectives? What lessons have you learnt relative to the project’s facilitation of processes in identifying and engaging stakeholders in education governance? Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program Are there any best practices you have observed in the areas of capacity building activities for Circuit Supervisors and Head-teachers at the District level? Which of these best practices do you think are worth replicating, and at what level do you think this could be replicated? What factors do you think affected achievements of results relative to training and resourcing of Circuit Supervisors and Head￾teachers at the District level? Are there any best practices that could be replicated, and at what level do you think these should be replicated? What significant lesson have you learnt whilst participating in the development of the education governance framework? Are there any best practices you have observed in the area of training and resourcing of Circuit Supervisors and Head￾Teachers, and how have these best practices contributed to students achievement at the school level? Objective 4 What factors do you think Final Evaluation of the USAID/Ghana PAGE Project Final Report 57 SMC/PTA capacity building affected achievements of results relative SMCs/PTAs capacity building activities by the PAGE project? Are there any best practices that could be replicated, and at what level do you think these should be replicated? What significant lesson have you learnt whilst participating in the development of the education governance framework? Question 4: What are the best ways to ensure that progress and results are captured and/or continued that also promote ownership, engagement and sustainability of interventions after the project ends? Objective 1 Stakeholder Outreach Is education well considered by the DA? Does the DEOC bring issues before the DA consideration? How much involved is the DA in the formulation of policy for education in the Districts? How much involved are you in the formation of policy for education in your District? Does the population, in general, know that the DEOC is in place and playing a role? What is the perception of the DEOC in the eyes of the community? Of parents of students who go to school? In the eyes of those who work at the school? What assistance should the outsiders (donors) provide to education stakeholders in Ghana to make the education system work better? Objective 2 DEOC-DEO collaboration Does the DEO have an annual plan for education in the District? Do you have an annual plan for education for your community school? Is there an overall framework for educational management in the district now that was not Final Evaluation of the USAID/Ghana PAGE Project Final Report 58 building and development of district specific framework for education governance and supervision If so, who contributed to the planning effort, and what planning process was adapted by the DEO? If so, who contributed to the planning efforts? there before? If so, can you state what is working with it and what is not? Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program How does the DEO know if education is achieving progress in the District and attaining its goals? Does education data from the schools play a role in making decisions about how education is delivered in the District? How do you (SMC/PTA) know if Head-teachers and teachers are doing their work and students are learning effectively and achieving education goals in your community? What do you think will make the education system work better in your community? Objective 4 SMC/PTA capacity building What new things do you think if you (SMCs/PTAs) are assisted will lead to education working better in your community? In your work with communities, is there any influence from the school system and how members of the community ? Question 5: What performance monitoring processes, systems and tools were used to ensure accurate, timely, reliable and valid performance and indicator tracking reporting and feedback at all levels? How did this contribute to logical implementation of activities, data validation, documentation and review of work plans, indicators and/or activities? Objective 1 Stakeholder Outreach What systems tools were used for performance and indicator tracking? Have stakeholders been introduced to these tools and do they use them? What systems tools were used for performance and indicator tracking? Have stakeholders been introduced to these tools and do they use them? Objective 2 DEOC-DEO collaboration Does the MOE and GES know about the tools and systems tools being introduced and Are you using the systems and tools that were provided to build the framework for Have you received training and information about the use of systems and planning tools for the Are you aware of systems tools being used in the education sector where stakeholders are Final Evaluation of the USAID/Ghana PAGE Project Final Report 59 building and development of district specific framework for education governance and supervision used at the District level? planning in the district for education? Are educational indicators and EMIS being used as tools for planning? management of your school? If so, what tools and why are they effective? also working? Do you see these tools and systems as contributors to children achieving better grades and learning more in school? Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program Are you aware of the training that has been received at the district level to build capacity for tracking educational data and planning in education? If so, what have you heard about it and do you think it is useful? What is your impression about the tools and systems provided in training to build the framework and track educational progress in your district? Do you as schools use the systems and planning tools that you received in training provided by CARE International? If so, what tools and why are they effective? Objective 4 SMC/PTA capacity building What has been the reception of the SMC and PTAs on the acquisition of systems and planning tools in education? Which of these are working at the school level? Do you as SMC and PTAs use the systems and planning tools that you received in training provided by CARE International? If so, what tools and why are they effective? If working at community level in education, do you see any differences in the planning or systems that members of the community are using that you have not seen before? Final Evaluation of the USAID/Ghana PAGE Project Final Report 60 ANNEX I, PART E: PAGE SAMPLE DISTRICTS FINAL PERFORMANCE EVALUATION OF PAGE PROJECT PAGE PROGRAM SAMPLE DISTRICTS – ZONED Northern Ghana 1. Northern Region a. Savelu/Nanton District - Savelugu town b. Tolon District - Chirifoyili 2. Upper East Region a. Bongo District - Tingre b. Bawku West - Sakom 3. Brong Ahafo a. Nokoranza District - Beboano Southern Ghana 1. Central Region a. Mfantseman District – Saltpond Catholic Boys Primary b. Abura Asebu Kwamankese District - Abura Dunkwa Methodist Basic School 2. Greater Accra Region a. Ga East District- Abokobi Presby Kg/Primary Central Ghana 3. Volta Region a. Ketu North District - Penyi Anglican Basic School (J.H.S Kg/Primary) b. Akatsi District - Avenorpeme D/A R/C Basic School 4. Eastern Region a. Asuogyaman District – Atimpoku RC Primary School Final Evaluation of the USAID/Ghana PAGE Project Final Report 61 ANNEX I, PART F: DRAFT LIST OF KEY INFORMANTS USAID IMPLEMENTERS Care project staff NATIONAL LEVEL PAGE Steering Committee members comprised of MOE/GES STATE INSTITUTIONS District Assembly members District Directors of Education Front Line ADEs Accountants Finance Office Girl-Child Officer M&E Officer Community Participation Coordinator Circuit Supervisors SCHOOL LEVEL Head Teachers Assistant Head Teachers School Management Committee/Parent Teacher Association members Final Evaluation of the USAID/Ghana PAGE Project Final Report 62 ANNEX I, PART G: PAPER SURVEY (OPTIONAL) PAGE Perception Survey Questionnaire Illustrative Draft 1. To what extent has the PAGE approach and activities led to improved governance and supervision in schools? ______ 1 2 3 4 5 None To a limited extent To some extent To a large extent Achieved Results or Exceeded Target 2. Has student achievement increased in the schools that have received governance and supervision interventions from the PAGE project? If yes, to what extent? (If not, check none.) ____ 1 2 3 4 5 None To a limited extent To some extent To a large extent Achieved Results or Exceeded Target 3. What factors, if any, have affected/inhibited the achievement of project results and could be considered lessons learned? 4. What factors, if any, have significantly enhanced/accelerated the achievement of project results and could be considered best practices? _______ 5. To what extent will the PAGE project processes, systems, and interventions be sustained by the MOE? _____ 1 2 3 4 5 None To a limited extent To some extent To a large extent Achieved Results or Exceeded Target 6. (If applicable) What are some examples that demonstrate the achievement of sustainable interventions and results? 7. What are the best ways to ensure that effective teaching interventions, management practices, and governance structures are sustained after the project ends? 8. How would you rate the capacity of DEOCs to contribute more effectively to improved school governance and supervision activities? Final Evaluation of the USAID/Ghana PAGE Project Final Report 63 1 2 3 4 Limited Has capacity in some areas of the functional areas required, but many functions still not performed Has most of the skills needed to perform functional role and monitor school management and student performance Is fully capable of carrying out functions to monitor and improve school management and student performance 9. How would you rate the competence of the following stakeholders to monitor and manage school performance to increase student achievement: Circuit Supervisors ___ Head Teachers __ SMCs __ 1 2 3 4 Limited Has capacity in some areas of the functional areas required, but many functions still not performed Has most of the skills needed to perform functional role and monitor school management and student performance Is fully capable of carrying out functions to monitor and improve school management and student performance 10. Do you have any other comments that you would like to share about the strengths or weaknesses of the PAGE project? Final Evaluation of the USAID/Ghana PAGE Project Final Report 64 ANNEX I, PART H: DRAFT FOCUS GROUP PROTOCOL Focus Group Guidelines The PAGE evaluation team will hire and train a local note taker and interpreter-moderators that will conduct FG discussions with males and females who are circuit supervisors, head-teachers and School Management Committee members. The senior Team member of the evaluation team will facilitate the discussion. The note taker will transcribe the discussion from Ghanaian language into English. If an appropriate group of school level workers can be organized who speak English, then notes will be taken by one of the Team members. The focus group will involve an odd number of individuals, preferably between 7-11 individuals with a preference for more women than men. The FG facilitators will ask primary leading questions. The facilitator will make sure all topics are covered before moving to the next set of questions with the FGD participants. Each FG will start with an introduction of why the participants have been asked to speak with the team, and why they are being asked three questions about: 1. Extent of their knowledge of collaboration and planning from DEOC and DEO 2. The tools they use in data collection and its use in maintaining education quality in the district 3. How the tools they use contribute to student achievement. Moderator Instructions 1. Make sure all topics under the 3 questions are covered 2. Do not give your opinion to direct how the participants should respond to questions. 3. Allow participants to volunteer their own answers and in their own ways. 4. Make sure that everyone in the group gets an equal opportunity to discuss the question. Note-taker Instructions 1. Write down exactly what people say. 2. If there is a good quote, record the statement verbatim. 3. When the FGD is completed, transcribe electronically and directly translate the full record of the group discussion. Do not summarize. Final Evaluation of the USAID/Ghana PAGE Project Final Report 65 CARE International Implementing Partners at School/Community Level School Heads, Circuit Supervisors, School Management Committees and PTAs Overarching Research Hypothesis Goal: If educational governance and supervision in basic schools is improved, then student achievement will also improve. H1: If the capacity of circuit supervisors, head-teachers, and SCMs to monitor, manage, and report on school management performance is strengthened, then educational governance and supervision will improve. H2: If the capacity of DEOCs to support and enable the competencies of circuit supervisors, head-teachers, and SMCs is strengthened, then educational governance and supervision will improve. Question 1: Have governance and supervision interventions at the district level resulted in improved student achievement? If so, to what extent and why? If not, why not? Objective 1 Stakeholder Outreach What kinds of changes have you noticed in your school since the beginning of this project? Have you seen improved functioning of the school system? Have you seen improved learning outcomes or report cards from you children when they come home from school? Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision Have you developed an educational plan for your district? How long did it take to get consensus for the plan? Have you developed plans and policies with the District Assembly and DEOC collaboration? Have the plans and policies worked to improve education in the district? What ways can you describe that demonstrate collaboration and a willingness to play a role? Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program Has training and capacity building met the needs of building education governance in your district? How did the district play a role in making sure that the training received met the need of the district? How much are you using the training that you received? Can you site examples of tools learned that you are using now? Does the district use the data you collect and are they interested in whether or not you are using the training you received? Final Evaluation of the USAID/Ghana PAGE Project Final Report 66 Objective 4 SMC/PTA capacity building Did the training you received meet the need to make improvements to teaching and learning in your community? Did the training you received help you work more effectively at the school? What things are you doing now that you were not doing before? Are you able to ask more important questions of the school system officials with the training you received? Question 2: What intended and unintended contributions, results and/or impact has the PAGE approach and activities achieved relative to improving governance and supervision in schools? Objective 1 Stakeholder Outreach What contributions have stakeholders made through PAGE in improving the achievement of students at your school? Is the community aware that changes and improvements are being implemented and planned for schools in the District? Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision What contributions has the specific framework for educational governance made toward improving school performance in your community/school? What contributions, intended or unintended, have been made through better supervision of schools? What contributions, intended or unintended, have been made through improved governance of the school system? Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program What contributions have been made, either intentional or unintentional, through the training of Circuit Supervisors? What contributions have been made, either intentional or unintentional, through the training of School Heads? What contributions, either intentional or unintentional, have been made in improving schooling through additional resourcing to the school head teachers and circuit supervisors? Objective 4 SMC/PTA capacity building What contributions, either intentional or unintentional, have been made that improves the school system through the capacity building of the School Management Committees and the PTAs? Are the SMC/PTAs able to plan and become a more important force for change within the school? How has community participation improved the quality of the school? Question 3: What factors affected the achievements and results of the project? What are the lessons learned and best practices, and from which stakeholders or beneficiaries? Final Evaluation of the USAID/Ghana PAGE Project Final Report 67 Objective 1 Stakeholder Outreach What factors affected the achievement and results obtained from the project through stakeholder involvement? Did you see stakeholder involvement in the project? If so, who were they, and what were the factors of their engagement? Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision Was collaboration a factor in the success of this project? Were there any clear lessons learned from the collaboration building effort of the project? If collaboration was a success factor, then which collaborators were the most active or committed? Was the development of a framework for education a factor in the success of the project? Were there any lessons learned from the establishment of the framework? What were the most important elements of governance that seemed to work and what still needs to be done in governing and regulating the education system? Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program Was training a factor in making the project a success? What did training actually achieve and how was it a factor in improving student achievement? Was the receipt of resources a factor in the project success? Are there any lessons learned in how resources are distributed and managed once in place? What were the most useful elements of training for the Circuit Supervisors and for Head Teachers? And the least useful parts of the training? Objective 4 SMC/PTA capacity building Are the SMC and PTAs factors of success for this project? If so, how are they operating to make it a success? What lessons have been learned from the SMC and PTA part of the project? Do the SMC people find other areas where the project has been a success? Question 4: What are the best ways to ensure that progress and results are captured and/or continued that also promote ownership, engagement and sustainability of interventions after the project ends? Objective 1 Stakeholder Outreach How are stakeholders responsible to ensure that progress and results are captured and which promote ownership, engagement and sustainability of the project? Has the PAGE project assisted the stakeholders to ensure that progress and results are captured to promote proper planning and ownership of the school system by the community? Objective 2 DEOC-DEO collaboration building Does the collaboration between the DEOC and the DEO and the development of the specific framework for education been a good way to ensure that progress and results are captured and continued to promote ownership, engagement and sustainability of the interventions of the project? If so, how will this continue after the project ends? Final Evaluation of the USAID/Ghana PAGE Project Final Report 68 and development of district specific framework for education governance and supervision Does the new emphasis on supervision and governance ensure that progress and results are captured and continued to promote ownership, engagement and sustainability of the interventions of the project? What is more important, Supervision or Governance that will ensure that progress and results are captured and continued to promote ownership, engagement and sustainability of the interventions of the project? Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program Is training and resourcing been a good way to ensure that progress and results are captured and continued to promote ownership, engagement and sustainability of the interventions of the project? If so, how? Objective 4 SMC/PTA capacity building Is building the capacity of the SMC and PTA a way to ensure that progress and results are captured and continued to promote ownership, engagement and sustainability of the interventions of the project? How would community participation play a role in promoting ownership, engagement and sustainability of the school? Question 5: What performance monitoring processes, systems and tools were used to ensure accurate, timely, reliable and valid performance and indicator tracking, reporting and feedback at all levels? How did this contribute to logical implementation of the activities, data validation, documentation and review of work plans, indicators and/or activities? Objective 1 Stakeholder Outreach What stakeholders used monitoring processes, systems and tools to ensure accurate, timely, reliable and valid performance and indicator tracking, reporting and feedback at all levels? Do you know if stakeholders used information for the purpose of making plans for education in the District? Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision Did the collaborators use monitoring processes, systems and tools to ensure accurate, timely, reliable and valid performance and indicator tracking, reporting and feedback at all levels? Was there collaboration in using these tools? Does the framework for education governance and supervision use monitoring processes, systems and tools to ensure accurate, timely, reliable and valid performance and indicator tracking, reporting and feedback at all levels? Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program Does training that you received as Circuit Supervisors and School Heads introduce you to monitoring processes, systems and tools to ensure accurate, timely, reliable and valid performance and indicator tracking, reporting and feedback at all levels? Did you use these tools to validate and ensure accurate data collection for the District and for the schools you serve? What tools do you use now that you did not use before? Are they helpful? Final Evaluation of the USAID/Ghana PAGE Project Final Report 69 Objective 4 SMC/PTA capacity building In the capacity training you received were you introduced to monitoring processes, systems and tools to ensure accurate, timely, reliable and valid performance and indicator tracking, reporting and feedback at the school level? What information tools do you use and how are they useful? Final Evaluation of the USAID/Ghana PAGE Project Final Report 70 ANNEX I, PART I: DRAFT GENDER ANALYSIS MATRICES The evaluation team has chosen to use the “domains of analysis” framework to present the data. For each domain, the evaluation team has listed key questions that should be asked when conducting gender analysis, suggested sources of information and areas for further inquiry, the gender-based constraints that need to be considered, and opportunities for future project (in other words, recommendations). The domains are described in greater detail in the table below, which also appears in USAID’s Toward Gender Equality in Europe and Eurasia: A Toolkit for Analysis (2012): Domain Key Issue Access to assets Who has access to which particular assets? What constraints do they face? Knowledge, beliefs, perceptions Who knows what? What beliefs and perceptions shape gender identities and norms? Practices and participation Who does what? What are the gender roles and responsibilities that dictate the activities in which men and women participate? How do men and women engage in development activities? Time and space How do men and women spend their time, as well as where and when? Legal rights and status How are women and men regarded and treated by customary and formal legal codes? Balance of power and decision making Who has control over the power to make decisions about one’s body, household, community, municipality, and state? Are such decisions made freely? Table 1. Framework for Gender Relations, Constraints, and Opportunities What are the key gender relations related to each domain that affect male and female participation in PAGE, or projects similar to PAGE? What other information about gender relations is needed? What were the gender-based constraints hindering achievement of project objectives? What were the gender-based opportunities in design for future projects? Laws, policies, regulations, and institutional practices Cultural norms and beliefs Gender roles, responsibilities, Final Evaluation of the USAID/Ghana PAGE Project Final Report 71 and time used Access to and control over resources Patterns of power and decision￾making Adapted from: Elisabeth Duban and Catherine Cozzarelli (2012). Toward Gender Equality in Europe and Eurasia: A Toolkit for Analysis. USAID p.16 http://pdf.usaid.gov/pdf_docs/pnaea292.pdf, and USAID ADS Chapter 205, p.12. http://www.usaid.gov/sites/default/files/documents/1870/205.pdf Table 2. Gender Evaluation Questions Summary Table PAGE Evaluation: Gender Analysis Project Performance Evaluation Questions How the Project Responded How the Project Worked Adapted from Deborah Caro (2009) A Manual for Integrating Gender into Reproductive health and HIV Programs: From Commitment to Action. USAID/Interagency Gender Working Group, p. 27. http://www.prb.org/igwg_media/manualintegrgendr09_eng.pdf Final Evaluation of the USAID/Ghana PAGE Project Final Report 72 ANNEX I, PART J: APPROACH TO DATA ANALYSIS Data Analysis Plan This section of the work plan outlines basic assumptions that informed the structure of the performance evaluation to be conducted on PAGE, identifies the needed data sources, and describes the methodology of conducting the analysis. An attempt is also made to describe charts and other data presentation methods that will help report findings in a manner that can be appreciated by different audiences. Assumptions/Conditions Precedent  The RFP for the PAGE evaluation is for the conduct of a Performance Evaluation and not an Impact Evaluation;  The PAGE project design did not plan for an Impact Evaluation using time-series data; Data Expected to Be Available for Analysis A review of available documentation on the PAGE PE reveals that data could be obtained mainly from the following sources:  Data from Baseline Survey conducted by PAGE.  Collated Data and information on Performance Indicators available in PAGE MIS and also reported in: o CARE Monthly/Activity Reports; o Quarterly Progress Reports; and o Annual Reports, If any  Data from a review of available records at the school, district, regional and national levels.  Data from Key Informant Interviews to be undertaken by SI Team.  Data from FGD to be undertaken by SI Team.  Data and information from February 2013 Audit Report.  Data and information from any special studies undertaken by PAGE. Proposed Analysis Approaches  Conduct a trend analysis to assess student improvement over time including a number of years prior to the Project.  Conduct a contribution analysis to determine the extent to which the governance and supervision interventions directly resulted in improved student achievement. This will also indicate other possible contributors to the same result, as it will be likely that other education interventions might have influenced student performance during the period of PAGE implementation.  Use a mix of quantitative and qualitative data to respond to this evaluation question, although it will not be able to establish an absolute causal statement or an impact analysis within the scope of this performance evaluation. Analytical Tools to Be Used Two main analytical tools will be used for performance evaluation of PAGE. These are Trend Analysis and a modified form of the McKinsey Capacity Assessment Grid. These are elaborated in the sections that follow. 1. Trend Analysis A series of conceptual issues would be addressed before analyzing and interpreting trend data available for the PE. These include: Final Evaluation of the USAID/Ghana PAGE Project Final Report 73  Sample size (for periods) —In this case from base year to 2013  Examine all data collected/collated for the presence of extreme observations or outliers to determine whether these are due to random variability or whether they reflect a real departure from the general trend.  Availability of accurate numerator and denominator data for 2011 and 2013  Are there any (Confounding) changes over time in factors related to the indicator of interest that must be looked at? The SI team will use trends analysis to do any or all of the following:  Determine the overall pattern of change in an indicator over time.  Compare baseline data to current evaluation data.  Compare data across regions or regional averages with national averages.  Compare both absolute and relative differences of one population to another.  Make future projections. Whenever feasible, graphs will be used to enable an easy appreciation of the results being discussed. 2. A Modified McKinsey Capacity Assessment Grid For each respondent institution that the Instrument Framework is used to assess, the first step would be to have a column for inputting the accomplishment rating of the various issues that require an answer. This is provided in Annex I in Table 1. Table 2: Expanded Instrument Framework to enable a recording of capacity ratings *Red lettering using only first sub-category under Governance to illustrate use. Category (1) Start-up Stage (2) Developing Stage (3) Consolidating Stage (4) Maturing Stage Input accomplishment rating in this column A. GOVERNANCE Oversight Committee or other governing body Existence of the Oversight Committee Advisory committee or other pre￾Oversight Committee structure in place Oversight Committee members identified but not yet meeting regularly Oversight Committee is meeting regularly but attendance is erratic Oversight Committee in place and meeting regularly with full participation 1 Differentiation of oversight and management roles No recognition of the need for such differentiation. Oversight Committee is involved in daily management or management assumes task of formulating Oversight Committee and management understand respective roles, but not yet following their designated mandates Differentiation between Oversight Committee and Management improving Oversight Committee fully focused on playing its role of providing vision, overall policy direction, fundraising, public 4 Final Evaluation of the USAID/Ghana PAGE Project Final Report 74 Category (1) Start-up Stage (2) Developing Stage (3) Consolidating Stage (4) Maturing Stage Input accomplishment rating in this column policy direction relations, lobbying and financial oversight Managing transitions Oversight Committee function vested on political appointments Frequent Oversight Committee member resignations; replacement process is ad hoc Majority of members filling all terms; process defined for selecting replacements but not always followed Most members filling all terms; replacement process fully respected 3 Oversight Committee composition/div ersity Oversight Committee function defined by policy, but not yet in place Oversight Committee is relatively homogeneous, limited representation on basis of gender, class, stakeholder interest, etc. Includes few leaders in the field of education More represented of constituency and includes several leaders in education Fully representati ve of constituency and includes many leaders and women in the field of education 2 Managing transitions Oversight Committee function vested on political appointments Frequent Oversight Committee member resignations; replacement process is ad hoc Majority of members filling all terms; process defined for selecting replacements but not always followed Most members filling all terms; replacement process fully respected 3 Sub-Category/Section Average (for Oversight Committee or other governing body) 2.6 The second step is to collate all the sub-category or section average accomplishments to help establish category averages. Table 3 illustrates how this could be accomplished. Final Evaluation of the USAID/Ghana PAGE Project Final Report 75 Table 3: Establishing the Average Category Rating of Accomplishments Category Accomplishment Rating Levels Sub-Category/ Section Accomplishment Dimension Averages A. GOVERNANCE Oversight Committee or other governing body 2.6 Mission Legal Status Leadership B. OPERATIONS AND MANAGEMENT SYSTEMS Management of Information Administration Planning Communications Program Development & Implementation C. HUMAN RESOURCES All these and other data generated by the data collection can be developed as a spreadsheet (MS Excel) that will ensure that once Table 1 is completed manually, it automatically collates the summary sheets in Table 2 and even the summary graph provided below as Figure 1. The final step would be to plot a graph that will enable a pictorial representation of the various levels accomplished for each category by PAGE. This is illustrated in Figure 1 below. Comparison of Baseline to Evaluation is for illustrative purposes only. The evaluation will generate comparative and descriptive results based on the types of data collected. Figure 1: Illustrative Summary Chart showing a comparison with baseline ratings Final Evaluation of the USAID/Ghana PAGE Project Final Report 76 ANNEX II: PAGE PROJECT BACKGROUND AND PARTNER ROLES In 2010, USAID/Ghana committed to achieving the Education Assistance Objective of “improving the quality of, and access to, basic education.” One of the ways the Mission envisioned achieving this objective was through the implementation of the Partnership for Accountable Governance in Education (PAGE) program. To help support the Government of Ghana (GoG) and USAID/Ghana strengthen the country’s educational system, the Cooperative for Assistance and Relief Everywhere (CARE) entered a three –year agreement with the Mission for the implementation of PAGE across a sample of districts. CARE hypothesizes that if education leadership and management improve through accountable, transparent governance systems, and school supervision is strengthened, then the performance of students at the basic level will improve. Overview of GES Structure and PAGE Interventions PAGE is working at all levels of the GES governance structure and building the capacity of each actor to perform their functions well. From the highest levels of government to the community-based school committees, PAGE works with the National-Steering Committee, the District Education Oversight Committee, Circuit Supervisors, Head Teachers and the SMC/PTA committees as described below. The National Steering Committee (NSC) was designed to ensure the smooth implementation of project activities and close alignment of the project with the Education Strategic Plan (ESP). The NSC includes the Director General, Directors from six GES Divisions (Basic, Inspectorate, Teacher Education, CRDD, HRMD, and S&L), the GES PAGE Coordinator, the USAID AOTR, and a senior official from the Ministry of Local Government and Rural Development (Director of the Decentralization Secretariat). The District Education Oversight Committee (DEOC) is the highest body within the decentralized, district-level structure of the Ghana Education System. The DEOC plays a crucial role in education planning, supervision and accountability by overseeing the management of the District Education Office (DEO), which is responsible for education service delivery. Under the 2008 Education Act, the DEOC is responsible for overseeing the provision of teachers and the regular and punctual attendance of teachers and pupils; the proper performance of functions by school staff; complaints relating to teachers, non￾teaching personnel and learning materials; and complaints from teachers. Through joint partnership planning meetings, PAGE is encouraging better collaboration between DEOCs and DEOs so that these two bodies will collaborate in planning, budgeting, and monitoring school performance. Circuit Supervisors: Within the District Education Office, the key actors are the Circuit Supervisors (CS). Carrying a triple role of education management and quality improvement, the CS serve as an evaluator of teaching and learning in the schools; a curriculum advisor; and, liaison agent between schools in the circuit and the DEO. PAGE has trained CS to conduct focused school visits to encourage better teaching and management practices and has facilitated CS monitoring through provision of Final Evaluation of the USAID/Ghana PAGE Project Final Report 77 motorbikes so they can conduct regular and highly-visible supervisory visits to schools. The GES has supported Circuit Supervisors with funding for fuel and motorbike repair in some districts, and has established performance targets and reporting mechanisms to maximize the impact of the supervisory visits. SMC/PTAs: At the school-level, the School Management Committee (SMC) is the governing body for basic schools. The SMC operates directly under the DEOC and in collaboration with the District Education Directorates and schools. The main role of the SMC is to help the Head Teacher (HT) of a school to organize and manage the school’s activities in an effective and efficient manner. Among other duties, the SMC plays a significant role in contributing to the development of the School Performance Improvement Plan (SPIP); in establishing priorities and setting goals for school improvement; and in implementing mechanisms for holding HT and staff accountable for progress set out in the SPIP. PAGE has developed handbooks in collaboration with the GES describing supervisory roles and responsibilities of each oversight body and has provided training to the DEOC, Circuit Supervisors, Head Teachers and the School Management Committees. As part of its efforts to improve the quality of teaching and learning, the GES introduced the School Report Card (SRC) in February 2010. The SRC is the basic tool for tracking school and student performance at the district and community levels. The DEO utilizes the school performance data for setting targets, addressing performance issues, and allocating resources to address quality or equity gaps. At the community level, through School Performance Appraisal Meetings (SPAM), the School Report Card is utilized to discuss the performance of schools with all stakeholders and to give them the opportunity to participate meaningfully in how to improve school outcomes. PAGE is building the capacity of SMC and PTA members in 3,680 schools to interpret school performance data (School Report Cards) and to advocate for better management, better teaching and higher levels of achievement among boys and girls. Furthermore, PAGE is strengthening community capacities to hold schools and district authorities accountable for targets indicated in the School Performance Improvement Plan (SPIP). Final Evaluation of the USAID/Ghana PAGE Project Final Report 78 ANNEX III: CORE INTERVIEW QUESTIONNAIRE INTERVIEW PROTOCOL FOR SKII *modified for focus groups DISTRICT: _____________ DATE: ________________ Name: ___________________________________ Position: _________________________________ Length of Time on project: _________________________________________________ 1. What is your understanding of the PAGE Project? What are its goals and objectives? 2. How would you describe your role? 3. What specific types of contributions (e.g., materials, equipment, training, funding) have you received from the PAGE project? a. How have these contributed to improving governance and supervision at the district level? 4. Have you received any training or capacity building from the PAGE project in the last academic year? 5. What types of effective management and governance practices have you applied as a result of the PAGE project trainings? 6. Have you participated in PAGE capacity buildings for other key actors within the education management and governance system (e.g., PTA/SMC/DEOC/ head teachers)? a. If yes, what is your opinion of the trainings? Has it contributed to improved governance and supervision? b. Is there evidence that they have utilized their new capacity to influence teaching and learning at the school level? c. Which actors have had the most impact on improving educational governance? Final Evaluation of the USAID/Ghana PAGE Project Final Report 79 7. Has educational governance and supervision at the school level improved as a result of PAGE interventions? a. If yes, which interventions have had the greatest impact? 8. Has student achievement (general literacy/early grade reading) increased in the schools that have received PAGE’s interventions? a. If yes, could you share some examples of how it has improved and what you used to measure the achievement? b. If not, why not? 9. Have there been any challenges impeding your ability or the project’s ability to achieve PAGE results? If yes, what? 10. Who is, or who are the most important actor(s) in the school system to increase student achievement? How effectively are they carrying out their roles? a. DEOCs b. Circuit Supervisors or Head Teachers c. Teachers d. SMCs/PTAs. 11. Are there any best practices you have observed or applied that have contributed to achievement of project results (e.g., improved management, governance or student achievement in reading)? 12. In your opinion, what do you perceive as the benefits of the PAGE projects, if any? 13. Do you think the benefits of the PAGE project will continue after the project ends? Why or why not? 14. Is there a monitoring and evaluation system in place to ensure data on educational outcomes is used for planning and decision-making? a. What are the mechanisms and tools used to support data collection? b. What reporting structures are in place? (e.g., SRC) c. How is the data used to inform decision-making? (e.g., SPIP) d. How has PAGE contributed to the system? 15. In your opinion, has the PAGE project achieved its intended results? 16. Were there any unexpected outcomes of the PAGE project activities? Final Evaluation of the USAID/Ghana PAGE Project Final Report 80 17. Do you have any other comments or observations that you would like to share about the strength or weaknesses of the PAGE project? 18. What would you emphasize in a future education project? Final Evaluation of the USAID/Ghana PAGE Project Final Report 81 ANNEX IV: PUPIL QUESTIONNAIRE DISTRICT: ______________________SCHOOL: _________DATE: _________ CLASS NAMES CLASS ONE 1. 2. CLASS TWO 3. 4. CLASS THREE 5. 6. Proposed Procedure  Seek permission from school Head Teacher and visit class One, Two and Three  In each class, exchange greetings as a way to seek pupils' attention  Randomly select two pupils (a boy and a girl) from each class and take them to a central location for interaction.  Request the Head Teacher to provide you with English readers for all three classes (classes One, Two and Three).  Explain to the selected pupils that you are in the school to learn from them about the school, their class and what they learn Pupils Questions 1. Ask the following questions to get a sense of how the school time is utilized: a. What time do you come to school in the morning? b. What time do classes start? c. What time do teachers report for classes? d. When do you normally close from school? e. DEOs the Head Teacher visit your classrooms? f. What DEOs the head teacher do when she/he visits your classroom? 2. What did you not like about your school a year ago? 3. What do you like about your school now? 4. Have you observed any improvement in how your teachers are teaching now? 5. In which language do you communicate with your teacher in the classroom? English? Ghanaian? a. In which language do you understand better when she/he uses it to teach? 6. Can you read in the Ghanaian language? 7. Can you read in English? 8. Ask each pupil which topics they have already read in class with their teachers. Select a paragraph from each topic they are acquainted with and let them read. Also select a non-familiar topic from the same grade/class book and make them read a paragraph at random. Grade the pupils according to the following criteria below: a. Pupil can read fluently (reads without difficulty and at the pace of a child of his/her age) b. Pupil reads with difficulty (i.e. Reads with paucity and some degree of difficulty) c. Cannot read at all. Final Evaluation of the USAID/Ghana PAGE Project Final Report 82 9. Do you take class work home as homework? a. Who helps you complete the homework? b. DEOs the teacher “mark” your homework? c. DEOs the teacher discuss your homework with you and your classmates after marking your scripts? 10. Do your parents visit the school? a. What do your parents do when they visit the school? b. Do they look at your exercise books? c. Do they encourage you to speak English at home? d. Do they assist you to read in English? e. Who else apart from your parents assist you at home to complete your homework or read in English? 11. Who else apart from your parents visit the school? a. What do they do when they visit the school? Note to Evaluator: Please tick the box with the correct level of the student tested. Student 1 Name: __________________ Level of Reading Ability Tick ✔ correct response Pupil can read fluently (reads without difficulty and at the pace of a child of his/her age) Pupil reads with difficulty (i.e. Reads with paucity and some degree of difficulty) Cannot read at all Student 2 Name: __________________ Level of Reading Ability Tick ✔ correct response Pupil can read fluently (reads without difficulty and at the pace of a child of his/her age) Pupil reads with difficulty (i.e. Reads with paucity and some degree of difficulty) Final Evaluation of the USAID/Ghana PAGE Project Final Report 83 Cannot read at all Final Evaluation of the USAID/Ghana PAGE Project Final Report 84 ANNEX V: GENDER ANALYSIS The evaluation team has chosen to use the “domains of analysis” framework to present the data. For each domain, the evaluation team has listed key questions that should be asked when conducting gender analysis, suggested sources of information and areas for further inquiry, the gender-based constraints that need to be considered, and opportunities for future project (in other words, recommendations). The domains are described in greater detail in the table below, which also appears in USAID’s Toward Gender Equality in Europe and Eurasia: A Toolkit for Analysis (2012) Gender in the PAGE Project PAGE has opted for a gender inclusiveness approach, which means enabling District Girls’ Education Officers (DGEO) to increase initiatives and visits to schools in order to address obstacles to girls’ education. The initiative was started in the second reporting period between October 2010 and March 2011. At first it conducted workshops and visioning exercises and eventually provided financial support to all 46 districts to undertake or actively participate in community mobilization and school monitoring in order to increase school and community awareness on issues affecting girls in schools. Findings PAGE’s inclusiveness approach is an on-going activity. According to GES policy, each school should have a girls’ club. PAGE helped to establish girls’ education facilitators in order to ensure the clubs are active and to build the capacity for girls’ education at the school level. As a result, the community and government have become more aware of and responsive to addressing teenage pregnancy, early marriages, and other harmful traditional practices interfering with girls’ educational attainment. For example, the DEO of Nkoranza has worked to overcome the barriers to teenage pregnancy. It has established a policy enabling teenage mothers to continue their education. The District Girls’ Education Officers were given opportunities to undertake monitoring visits in order to track girls’ enrollment and girl dropout rates as well as to overcome barriers to retention such as teenage pregnancy. Semi-annual Report 7 cites several cases where the DGEOs have made interventions and have met the issues of teen pregnancy and school girl participation in reading competitions. The M&E Office database has data on schools with the highest girl dropout rates. Those schools were selected for the PAGE intervention. PAGE leveraged additional funding to support a number of schools with the highest dropout rates due to teenage pregnancy. The GEO has received training in Information, Education, Communication (IEC) activities and often coordinate the radio shows and other IEC activities supported by CARE in the district. Some districts more readily address gender issues than others. The evaluation team paid a visit to a classroom in one Savelugu school in which the students stood up and repeated “Good morning, madam, good morning, sir,” whereupon the teacher corrected the class and said that “Good morning, sir” is sufficient. The PAGE Gender Analysis Summary Table, located below in this Annex, indicates that PAGE reports gender breakdowns with only a few exceptions. It pays attention to reporting gender accurately and it is committed to seeking gender parity within its own structure. There is gender disparity in the number of SMC/PTA members, CS, community volunteers, but it actively requests to train more women in these positions. The PMP and Annual World Plans represent gender to the extent possible and recognizes when indicators demand the appropriate distinctions. Conclusions 1. PAGE actively pursues and succeeds in achieving gender inclusiveness in the program. Final Evaluation of the USAID/Ghana PAGE Project Final Report 85 2. At the system level, PAGE strives to offer training for both genders but is not successful with female participation, as it takes what the district and communities assign for the training. 3. PAGE is beginning to address gender inclusiveness at the school and community levels through the Girls’ Education Officer at the DEO. But the GEOs are severely limited due to lack of transportation and the budget to do anything substantive. Recommendations 1. In its representations of gender inclusiveness, USAID should make additional efforts to report gender breakdowns on all activities reported in its Semi-Annual Reports and other publications. 2. PAGE should work with the districts and communities for additional women to be trained in its programs. Including more women in key stakeholder roles ensures a more equal distribution of skills and opportunity for empowerment throughout the system. 3. PAGE, or the next generation of PAGE, should consider a school club activity component in the next generation of USAID-funded programs in education. These could involve programs that offer training in life skills, sexuality, family life, early marriages, and relationships. The US Peace Corps is involved at the JHS level in Ethiopia promoting girls’ clubs, developing curriculum, and encouraging student leaders in schools. The GEOs need a handbook like the ones that exist for other functions of DEO service, such as those for the CSs and the HTs. Final Evaluation of the USAID/Ghana PAGE Project Final Report 86 Table 1: Gender Inclusiveness Domains and Key Issues of the USAID/Ghana PAGE Project Domain Key Issue Access to assets Who has access to which particular assets? What constraints do they face?  Girls have access to schools, but they may be far away, their families may not want them to go and they may feel threatened about going.  Female teachers may not want to work in remote schools for fear of feeling prejudiced or because their husbands are working in a different place.  Women SCs do not want to use motorbikes because they feel self-conscious on them and out of place.  DA has access to budget assets and women are beginning to participate more directly in the discourse. Knowledge, beliefs, perceptions Who knows what? What beliefs and perceptions shape gender identities and norms?  Knowledge, beliefs and perceptions require more in-depth contact and structured research than the results of performance evaluation can produce. Practices and participation Who does what? What are the gender roles and responsibilities that dictate the activities in which men and women participate? How do men and women engage in development activities?  The Evaluation Team found that Joint Collaborative Planning is beginning to take place at the DEO/DEOC/DA level and these involve women. These meetings discuss the performance of girls and boys in school where solutions through the budget are sought to remediate them. Time and space How do men and women spend their time, as well as where and when?  No data on time and space. Legal rights and status How are women and men regarded and treated by customary and formal legal codes?  In some northern districts women are dominated by men in meetings. Either cultural prerogatives or blatant superiority defines the status of men over women.  Girls have the right to go to school. Final Evaluation of the USAID/Ghana PAGE Project Final Report 87 Balance of power and decision making Who has control over the power to make decisions about one’s body, household, community, municipality, and state? Are such decisions made freely?  Female DEOC members holding executive positions on the committee are in force in all 11 Districts interviewed. From our limited interviews a substantive conception of power held by women in DEOCs was not possible. However, the Committee is composed of Assembly women and District Heads of Health and Social Services.  The GES has control over who it hires as DDEs. It needs to hire more women DDEs.  The GES has control over how and who it hires as Head Teachers. It needs to hire more female head teachers.  Government regulations determine who is eligible for SMC and PTA members. DEOs and DEOCs can regulate and supervise the composition of SMC/PTAs.  Pupils: achieving almost parity in the school system; girls often achieve higher results than boys in the early grades. Final Evaluation of the USAID/Ghana PAGE Project Final Report 88 Table 2: Framework for Gender Relations, Constraints, and Opportunities What are the key gender relations related to each domain that affect male and female participation in PAGE, or projects similar to PAGE? What other information about gender relations is needed? “Poverty is one of the major barriers to reaching the goal of universal basic education in Ghana and educational enrollment and attainment for boys and girls is lowest, and the gender gap is the largest, in the three northern regions.” 1. Improved reading instruction 2. Strengthened basic education delivery system 3. Improved accountability and transparency between parents and local government What were the gender-based constraints hindering achievement of project objectives? What were the gender-based opportunities in design for future projects? Laws, policies, regulations, and institutional practices  Constitution of the Republic of Ghana 1992, Art. 25(1) (a) Education Act, 2008 (Act 778), Art. 2(2)  By law there is free and compulsory education for both boy and girls  Not all boys and girls equally go to school. Some families are reluctant to send their girls to school.  The Ghana Shared Growth and Development Agenda (GSGDA 2010-2013) proposes increasing community mobilization and awareness of the importance of girls‘ education (p. 18, Ghana Gender Assessment)  The GSGDA 2010-2013 proposes establishing and enforcing a no tolerance policy for sexual harassment and disciplining perpetrators, especially teachers (p. 18 Ghana Gender Assessment) Final Evaluation of the USAID/Ghana PAGE Project Final Report 89 What are the key gender relations related to each domain that affect male and female participation in PAGE, or projects similar to PAGE? What other information about gender relations is needed? “Poverty is one of the major barriers to reaching the goal of universal basic education in Ghana and educational enrollment and attainment for boys and girls is lowest, and the gender gap is the largest, in the three northern regions.” 1. Improved reading instruction 2. Strengthened basic education delivery system 3. Improved accountability and transparency between parents and local government What were the gender-based constraints hindering achievement of project objectives? What were the gender-based opportunities in design for future projects?  Teachers and sexual harassment  Girls are afraid to go to school. Cultural norms and beliefs  Cultural norms may inhibit girl education enrollment in the North and South  Gender gaps and gender based violence, (p. 74 in Ghana Gender-Assessment Plan)  District Assemblies and traditional authority figures get involved with these issues after the SMC/PTA and DEOC reports make it necessary to act. Local governments are becoming increasingly aware of these problems.  Engage SMC, PTAs, and Final Evaluation of the USAID/Ghana PAGE Project Final Report 90 What are the key gender relations related to each domain that affect male and female participation in PAGE, or projects similar to PAGE? What other information about gender relations is needed? “Poverty is one of the major barriers to reaching the goal of universal basic education in Ghana and educational enrollment and attainment for boys and girls is lowest, and the gender gap is the largest, in the three northern regions.” 1. Improved reading instruction 2. Strengthened basic education delivery system 3. Improved accountability and transparency between parents and local government What were the gender-based constraints hindering achievement of project objectives? What were the gender-based opportunities in design for future projects? traditional authorities in developing codes of conduct and local statutes on gender-based violence.(p. 73, Ghana Gender Assessment) Gender roles, responsibilities, and time used  Female teachers  Not enough role models in schools for girls  The GSGDA 2010-2013 proposes continuation of increasing the numbers of female teachers, especially in deprived areas (p. 18, Ghana Gender Assessment)  Help GOG to identify and create appropriate incentives to attract and retain women teachers, e.g., decent housing. (p. 73 Ghana Final Evaluation of the USAID/Ghana PAGE Project Final Report 91 What are the key gender relations related to each domain that affect male and female participation in PAGE, or projects similar to PAGE? What other information about gender relations is needed? “Poverty is one of the major barriers to reaching the goal of universal basic education in Ghana and educational enrollment and attainment for boys and girls is lowest, and the gender gap is the largest, in the three northern regions.” 1. Improved reading instruction 2. Strengthened basic education delivery system 3. Improved accountability and transparency between parents and local government What were the gender-based constraints hindering achievement of project objectives? What were the gender-based opportunities in design for future projects? Gender Assessment) Access to and control over resources  How can women Circuit Supervisors go the field?  Female agents not able to go to field.  Improve monitoring and performance of Mission’s gender￾related investments in education (p. 75 Ghana Gender Assessment) Patterns of power and decision-making  Girl-child enrolments based on family decision-making  Girls are not enrolling in school.  The GSGDA 2010-2013 proposes expanding the incentive schemes to increase girls‘ enrollment, retention and completion particularly in poor regions(p.18 Ghana Gender Final Evaluation of the USAID/Ghana PAGE Project Final Report 92 What are the key gender relations related to each domain that affect male and female participation in PAGE, or projects similar to PAGE? What other information about gender relations is needed? “Poverty is one of the major barriers to reaching the goal of universal basic education in Ghana and educational enrollment and attainment for boys and girls is lowest, and the gender gap is the largest, in the three northern regions.” 1. Improved reading instruction 2. Strengthened basic education delivery system 3. Improved accountability and transparency between parents and local government What were the gender-based constraints hindering achievement of project objectives? What were the gender-based opportunities in design for future projects? Assessment)  Identify strategies that build the capacity of communities to promote attendance and improved performance of girls and boys.(p. 57 Ghana Gender Assessment  Girls were discouraged from studying subjects like math and science, or technical skills most likely to provide job opportunities in the future. Furthermore, both male and female teachers Final Evaluation of the USAID/Ghana PAGE Project Final Report 93 What are the key gender relations related to each domain that affect male and female participation in PAGE, or projects similar to PAGE? What other information about gender relations is needed? “Poverty is one of the major barriers to reaching the goal of universal basic education in Ghana and educational enrollment and attainment for boys and girls is lowest, and the gender gap is the largest, in the three northern regions.” 1. Improved reading instruction 2. Strengthened basic education delivery system 3. Improved accountability and transparency between parents and local government What were the gender-based constraints hindering achievement of project objectives? What were the gender-based opportunities in design for future projects?  Power over instruction  Girls are deprived in the classroom reinforced gender stereotypes and disparaged women‘s leadership roles. (p.72 Ghana Gender Assessment *Adapted from: Elisabeth Duban and Catherine Cozzarelli (2012). Toward Gender Equality in Europe and Eurasia: A Toolkit for Analysis. USAID p.16 http://pdf.usaid.gov/pdf_docs/pnaea292.pdf, and USAID ADS Chapter 205, p.12. http://www.usaid.gov/sites/default/files/documents/1870/205.pdf Final Evaluation of the USAID/Ghana PAGE Project Final Report 94 USAID requires performance monitoring and evaluation to understand how the differences in the roles and needs of male and female participants improve or detract from the efficiency and overall impact of its programs. According to ADS 203.8, in order to track how effectively USAID assistance contributes to gender equality and female empowerment, performance plans must include gender sensitive indicators and sex disaggregated data. All people level indicators at CDCS, project or activity level must be sex-disaggregated. Table 3: PAGE Evaluation Gender Analysis (Summary Table) Evaluation Questions Project Performance How the Project Responded Project Evidence How the Project Worked 1a. Have governance and supervision interventions at the district level resulted in improved student achievement? 1b. If so to what extent and how many? If not, why not?  Town Hall Meetings: Most recent report, Semi-Annual Report No. 7, did not indicate gender attendance, but IEC training for campaigners were identified by gender  DEOC members trained: males and females trained reported  DEOC joint monitoring visits not broken down by gender  DA-DEOC-DEO 15 Joint Planning meetings: not broken down by gender composition.  SMC/PTA members receive role awareness training and re-training: gender breakdown reported in Semi-annual report no. 7  PAGE gender inclusiveness training in Semi-annual Report No. 7 indicates DGEO  The Evaluation Team found that when asked, most DEOCs could respond with a rough breakdown of male and female representation at Town Hall Meetings.  Evaluation Team found both male and female DEOC members had received training  DEOC Joint Planning meetings reported as completed with women participation in 4 districts  DEOC joint monitoring visits represented simply as broken down by teams and visiting schools with specific circuits, females and males participating within teams.  Evaluation Team found that 60% of SMC/PTA having good awareness of roles; PAGE reports that 430 SMC/PTAs have received training, 3098 males, and 997 females. ET identified 59 males and 69 females interviewed. Final Evaluation of the USAID/Ghana PAGE Project Final Report 95 Evaluation Questions Project Performance How the Project Responded Project Evidence How the Project Worked officers in all 46 districts.  PAGE reported Circuit Supervisors meet with communities and organize leadership effectiveness meetings in 5 schools per district  The GEO in Abura mentioned the training and indicated that more was needed to mobilize the community around girl issues.  Circuit Supervisors meet, but no gender participation breakdown.  On average there are 2 female DEOC members holding executive positions on all of the 11 districts interviewed.  The PAGE Evaluation Team met with 12 females and 63 male CSs  PAGE Evaluation Team met with and “spot checked” 30 male and 30 female pupils for reading competency. 2. What intended and unintended contributions, results and/or impact has the PAGE approach and activities achieved relative to improving governance and supervision in schools?  Unintended outcomes listed in a response from CARE to questions posed by the Evaluation Team.  Document supplied by the Finance Team Unintended outcomes: The DEOs developed a deeper appreciation of the potential of a number of activities to mobilizing communities, parents SMCs for school support.  Accountants developed a deeper understanding of and the ability to effectively account for the use of USAID funds.  Gender balance within CARE itself.  Evaluation Team noted that CARE lists some of its activities by breaking down gender participation but not all. (The last two Semi Annual reported activities not broken down by gender. They need to be broken down more consistently.)  The need to reflect the gender participation in the reports has been a problem along with other elements in the reporting process. (in a communication from the Finance Staff at CARE, International)  CARE is making attempts to become more gender conscious, making the processes non-discriminatory, sensitive to both genders’ profiles. Females and males Final Evaluation of the USAID/Ghana PAGE Project Final Report 96 Evaluation Questions Project Performance How the Project Responded Project Evidence How the Project Worked represent respectively 55 and 45% of PAGE administration, finance and program teams. (Semi-annual report No 5) 3a. What factors affected the achievements and results of the project? 3b. What are the lessons learned and best practices and from which stakeholders and beneficiaries?  Lessons learned: Through inclusiveness approach CARE needed to do more visioning among key stakeholders.  PAGE learned through its inclusiveness approach that it should address girl’s education by supporting DEO Girl Child Officers.  PAGE facilitated a session of stakeholders from CARE, MLGRD, GES Headquarters, Regions and Districts/Municipalities on “Mainstreaming Gender in PAGE”. But in the reporting of this event, no breakdown of participants was provided. (Semi-annual report no. 5)  Semi annual report no. 7 reports on training offered for the GCOs, but only made refererence to 46 districts, no participants were enumerated. 4. What are the best ways to ensure that progress and results are captured and/or continued that also promote ownership, engagement, and sustainability of the interventions after the project ends?  Use of Community Volunteers to collect M&E data from households  CARE has learned that capacity building can be used to strengthen the visioning of gender at all levels  Over the course of the last semester, 594 CVs (47 female and 547 male) were already actively visiting households and schools to interact with parents on students’ attendance to school and the support students that they provide to these children to facilitate their performance at school. A total of 1,279 household surveys have been conducted. The questionnaire used for the interviews is gender sensitive and oriented to capture households’ revenues and expenditures. (Semi-Annual Final Evaluation of the USAID/Ghana PAGE Project Final Report 97 Evaluation Questions Project Performance How the Project Responded Project Evidence How the Project Worked Report, No. 5.)  Gender equity in school governance and leadership is a key communication theme for the effectiveness of SMCs and PTAs in the PAGE developed and printed manual of IEC strategy. SMCs and PTAs were educated to always include women in selections. (Semi Annual Report, No. 5.) 5a. What performance monitoring processes, systems, and tools were used to ensure accurate, timely, reliable, and valid performance and indicator tracking, reporting and feedback at all levels? 5b. How did this contribute to logical implementation of activities, data validation, documentation and review of work plans, indicators, and/or activities?  CARE uses its PMP to monitor and report on deliverable results  PAGE Annual Work Plans for FY 2013  Revised PAGE FY 13-14 targets  PAGE supports the use of the SRP  Annual Review  Evaluation Team has consulted the PMP and verified that gender monitoring is taking place, but not in the detail of its meeting all requirements for particular indicators. For example in Semi-annual reports reporting takes place but without the conditions that qualify for full accounting, such as gender participation for meeting requirements for Joint Partnerships for Planning and gender breakdown for Town Hall meetings. (Semi-annual reports 5, 6, and 7.)  Annual Work Plans and Revised PAGE targets have no reference to gender strategy or targets that emphasize gender parity and/or participation. Anticipated training targets for CS and Standard Indicators are gender targeted.  The School Report Card is a tabulated census of school enrolments and other data collected by the school system. Numbers of female/male teachers, pupils and other Final Evaluation of the USAID/Ghana PAGE Project Final Report 98 Evaluation Questions Project Performance How the Project Responded Project Evidence How the Project Worked appropriate gender disaggregation appears on this form. SRP is used in other settings for planning District education budgets and for planning SPIP at school level.  Evaluation Team has reviewed the Annual Review as summarized in Semi-Annual report no. 7 and results are tabulated as gender specific. *Adapted from Deborah Caro (2009) A Manual for Integrating Gender into Reproductive health and HIV Programs: From Commitment to Action. USAID/Interagency Gender Working Group, p. 27. http://www.prb.org/igwg_media/manualintegrgendr09_eng.pdf Final Evaluation of the USAID/Ghana PAGE Project Final Report 99 ANNEX VI: DESCRIPTION OF FINDINGS BY IR AND EVALUATION QUESTION Evaluation Question 1: Have governance and supervision interventions at the district level resulted in improved student achievement? If so, to what extent and why? If not, why not? Goal: Improved Student Achievement Sub IR 2.1 Improved competence of circuit supervisors, head teachers and SMCs to monitor, manage and report on school management to increase student achievement Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Indicator 2.1.1: % of P1-3 pupils who demonstrate progress toward achieving Ghanaian language literacy 46% 48% 15 50% *(N = 29) Description: Pupils are considered demonstrating progress toward achieving Ghanaian language literacy if they are scoring at “transitioning” or “developing” levels on the NALAP assessment. Transitioning is the highest level of literacy achievement; developing is the middle category; and beginning is the lowest level of literacy. Achievements to date: PAGE administered the EQUALL/NALAP assessment tool in a sample of 100 schools with 2,000 pupils in each of the 46 districts at baseline and found 46% meeting this indicator. Because PAGE has no direct strategy to improve teaching and learning, the target was set at a very low benchmark of 48%. As the targeted percentage increase was only two percentage points between baseline and endline, student performance was not measured on an annual basis, but planned for the endline assessment. Additionally, it would not have been cost-effective or feasible given there was no funding allocated for student assessment. The project encouraged school-based reading assessments to track reading progress. Performance Evaluation Findings: Fifty percent of those tested met the indicator, surpassing the 48% target by two percentage points and the baseline by four. Fifteen students (8 boys and 7 girls) demonstrated progress toward achieving Ghanaian language literacy. Seven pupils (3 boys and 4 girls) read a Ghanaian language book with fluency, 8 pupils (5 boys and 3 girls) read with difficulty, and 14 students (7 boys and 7 girls) could not read at all. Boys and girls performed similarly across all categories. Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Final Evaluation of the USAID/Ghana PAGE Project Final Report 100 Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Indicator 2.1.2: % of P1-3 pupils who demonstrate progress toward achieving English language literacy 51% 53% 31 51% Description: Refer to description in indicator 2.1.1. Achievements to date: At baseline, 51 percent were found to be meeting the English language performance indicator. For more information regarding actuals to date, refer to indicator 2.1.1. Performance Evaluation Findings: The English test took place in all regions of the assessment with a sample size of 51 pupils (25 girls and 26 boys) in P1 – P3. Similar to the local language test results, 51% of pupils achieved the indicator. However, this is equal to the baseline value and two percentage points below the target. A total of 31 pupils (18 boys and 13 girls) showed progress toward achieving English language literacy. Fifteen pupils (6 boys and 9 girls) read an English language book with fluency, 16 pupils (7 boys and 9 girls) read with difficulty, and 20 pupils (13 boys and 7 girls) could not read at all. Girls performed better than boys at all levels. Evaluation Question 2: What intended and unintended contributions, results and/or impact has the PAGE approach and activities achieved relative to improving governance and supervision in schools? How? IR 3: Improved Educational Governance and Supervision Sub IR 3.1 Increase awareness creation, information sharing, and expectation management of stakeholders. (Relates to Component One Activities) Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Indicator 3.1.1: # of education-focused town hall Meetings organized by DEO 3 92 74 8 73% Description: In order to be considered, the town hall meetings must have the following characteristics: include participation of men and women, focus on pupil performance (both boys and girls performance discussed), include participation from a variety of communities within the district, and include stakeholder representation of at least 4 of the 6 following groups: Teachers, Head teachers, Civil Society Organization (CSOs), SMC/PTA members, Traditional Authorities, or DEOs/DEOCs. Final Evaluation of the USAID/Ghana PAGE Project Final Report 101 Achievements to date: PAGE has supported 46 partner districts to conduct town hall meetings as an education governance strategy to strengthen accountability systems and community engagement. This was a relatively new activity for districts evidenced by a baseline value of three. FY13 results indicate PAGE has successfully conducted 74 education-focused town hall meetings thereby achieving 78% of its target. Sub-granting delays and belated protocol guides slowed the achievement of this indicator. Performance Evaluation Findings: Of 11 districts sampled for the PAGE performance evaluation, 8 were found to meet the criteria for this indicator. The overall percentage of districts, averaging 73%, reflects the PAGE monitoring results (78%). According to DDEs interviewed, the town hall meetings provided a platform for dialogue on school-related issues. It enabled the DEO to engage stakeholders in ways that contribute to student achievement. In Nkoranza, the participants of the Town Hall meeting were most interested in discussing school performance, particularly the BECE results. High performing schools were asked to share their successes. Low-performing schools were asked to set a target for performance improvement. Other town hall meetings have centered on enrolment, retention, effective teaching and learning, role of parents, and the importance of investing in children’s, particularly girls’, education. Sub IR 3.2 Improved capacity of DEOCs to contribute effectively to improved school governance and supervision activities within their districts. (Relates to Component Two Activities) Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Indicator 3.2.1: # of DEOC members trained 0 460 544 6 54.5% Description: DEOC members are considered trained once they have received a new DEOC handbook and completed training in their specific roles and responsibilities Achievements to date: PAGE trained 544 DEOC members exceeding the 460-target number of DEOC members trained in specific roles and responsibilities. Performance Evaluation Findings: The evaluation team found that nine districts had indicated they have been trained by the PAGE project, but only six stated they had received handbooks. Therefore, 54.5% met the indicator criteria. Due to a relatively high turnover within the DEOC in several districts visited, some members had participated in refresher trainings, but had not received the handbooks. The majority of DEOC members trained asserted that the training was very useful. Prior to the PAGE training, the DEOC was dormant. Having never been trained, they were unaware of their roles and responsibilities. DEOC members in Bongo stated, “This singular training helped in making the DEOC discussions focus on how to improve teaching and learning at the school level.” Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Indicator 3.2.2: % of DEOCs using effective management strategies 0 50% 46% 6 54.5% Final Evaluation of the USAID/Ghana PAGE Project Final Report 102 Description: DEOCs are considered to be using effective management strategies if they have an operational district education management plan (ADEOP) with two of the following three also taking place: 1.) DEOC visits at least one school in each circuit per term; 2.) DEOC demonstrates responsiveness to school communities’ issues and concerns during visits to schools; 3.) DEOC has a task force to oversee literacy (NALAP) in place. Achievements to date: As the DEOC existed only in name prior to the PAGE project, CARE targeted 50 percent to be using effective management strategies. The actual achievement to date is 46 percent meeting the effectiveness indicator. Performance evaluation findings: Over 50 percent of districts visited appeared to use effective management strategies. Eight districts stated they had an ADEOP in place and visited at least one school per term. Six districts responded to issues identified during monitoring visits. Yet only one district had a NALAP taskforce in place. ADEOP: Many DEOC members stated that the ADEOP existed prior to the PAGE project. What PAGE has contributed has been attention to education issues and assistance with collection of data. Issues that have been discussed during DEOC meetings, monitoring visit debriefings and budget planning meetings have been embedded into the ADEOP in some districts. PAGE has directly assisted the process of developing the ADEOP through provision of motorbikes, which enable CS to collect data from the HTs that feeds into the annual School Report Cards (SRC). Through recruiting Community Volunteers, PAGE has assisted the GES to collect household data on out-of-school children, health, income and other indicators that support educational planning. NALAP activities: Although a NALAP taskforce was not found in 10 of the 11 districts visited, most districts co-opted NALAP coordinators in monitoring visits that review the materials and assess the learning process. The implementation of NALAP has been challenged in Ga East, Bongo, Bawku West and other districts in which the language most widely spoken differs from the approved NALAP language for the specific area/district. In such cases, it was out of place to institute a taskforce to teach children a language different from their mother tongue. Sub IR 3.3 Strengthened DEOC and DEO collaboration and responsiveness to school management issues. (Relates to Component Two Activities) Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Indicator 3.3.1: % of DA budgets that include costs that impact school monitoring plans 0 45% 41% 6 54.5% Description: DA budgets are considered to include costs that impact school monitoring plans if they include a line item for DEOC school monitoring or a similar item. This demonstrates that the DA is examining and considering the DEOC monitoring plans in its budget process. Achievements to date: At the inception of the project, no districts met the indicator for including costs that impact school monitoring plans. The project is facing very important challenges including the fact Final Evaluation of the USAID/Ghana PAGE Project Final Report 103 that GES is still not fully decentralized, making it difficult for education budgets, including DEOC monitoring budgets, to be integrated into District Assembly budgets. PAGE has achieved 41% of this indicator and is striving to reach 45% by end of Year 4. Performance evaluation findings: Evaluation results suggest that PAGE has exceeded its target. Six districts stated that the DA is financially committed to include school monitoring plans in their budgets. For instance, the Bongo district DA has set aside Ghc13, 000 to aid DEOC school monitoring for the 2013/2014 academic year. In other districts, when asked about sustainability, DEOC monitoring was often stated as not likely to continue due to DA funding constraints. Whilst commitment to DEOC monitoring is strong, funding is still a challenge. Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Indicator 3.3.2: Number of DEO/DEOC Joint Partnership Planning Meetings 1 83 48 4 36.4% Description: Joint Partnership Planning Meetings (JPPMs) are considered if they include members of the DEOC and DEO staff (with female participation). During Joint Partnership Planning meetings, the following issues must be discussed: issues that have been raised by school communities, and presentations to the DA regarding budgets. Achievements to date: Due to the initial delay in GES sub-granting and subsequent rollout of DEOC training and monitoring activities, JPPMs did not take effect until Year 2. Then, because of delayed release of activity guidelines, only 1 of the 8 JPPMs conducted in Year 1 and 9 of the 30 carried out in Year 2 were approved and calculated in cumulative totals. By Year 3, PAGE had accomplished 48 of the 92 targeted JPPMs. The end of Year 4 target has been reduced from 92 to 83. The project expects to hold 35 more JPPMs to achieve the target. Performance evaluation findings: Four of the 11 districts visited claimed they had held DEO-DEOC joint partnership meetings during the life of the project. This represents 36.4% of all districts visited, and is significantly lower than actual to date figures reflecting 57% of total PAGE districts. The DA has provided support in the form of funds for mock exams, budget for DEOC monitoring visits, supplying drinking water and septic tanks, providing furniture for KG classes, and lending vehicles to the DEOC/DEO (e.g., Asuogyaman) to support monitoring visits. The evaluators observed that there have not been many contributions directly related to teaching and learning or early grade literacy. Sub IR 3.4: Active and Engaged PTA and SMC trained to seek and interpret school performance reports with the view to developing and implementing strategies for improving school management systems and student achievement. (Relates to Component Four Activities) Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement Target Actual Districts Percent Indicator 3.4.1: Community awareness Little/No Awareness 20% 25% 2 20%* Final Evaluation of the USAID/Ghana PAGE Project Final Report 104 Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement Target Actual Districts Percent level of SMC roles and responsibilities: 47% Some Awareness 8% 15% 12% 2 20% Good Awareness 45% 65% 63% 6 60% *(N=10) SMC/PTAs were interviewed in 10 of the 11 districts, excluding Ga East, which was the pilot district. Hence, districts were measured against a total of 10, not 11, for this indicator. Description: Referring to roles and responsibilities listed in the SMC Handbook, the following scale was used to measure awareness:  Little or No Awareness = identifies 0-2 points unrelated to the role or responsibility  Some Awareness = identifies 3 points – at least one role and one responsibility  Good Awareness = identifies 4 points – at least one role and one responsibility Achievements to date: PAGE is close to achieving its targets for increasing community awareness of SMC roles and responsibilities with 63% achieving good awareness (target: 65%), 12% some awareness (target: 15%) and 25% having little to no awareness (target: 20%). Performance evaluation findings: The evaluation team found 60% of SMC/PTAs interviewed expressed good awareness, 20% (exceeding the target) had some awareness and 20% (meeting the target) had little to no awareness. An overwhelming majority of DDEs and stakeholders interviewed emphatically declared that SMC/PTA involvement has improved dramatically. Most SMCs were dormant or overshadowed by the PTA in the past. Now, respondents claim that SMC monitoring and contributions have had the greatest impact on improved teacher/pupil attendance, enrolment and performance. SMCs are now cooperating with the DEO to manage the schools and have a very close relationship with the DEO, to the extent that they communicate financial malpractice, non-performing HTs, or absentee teachers directly to the DDE. According to the DDE in Ga East, the SMC involvement has benefited school management and oversight of the HT, who is now present at meetings and submits frequent reports, which is much different from the past. She stated, “The capacity building for the SMCs was very helpful. It takes half of the load off our (DEO) shoulders.” Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement Target Actual Districts Percent Indicator 3.4.2: % of SMC/PTA effectively supporting schools 13% 55% 51% 7 63.6% Description of indicator: SMCs are considered to be effectively supporting schools if they are doing at least two of the following: 1.) visiting schools at least twice per term and interacting with head teachers, Final Evaluation of the USAID/Ghana PAGE Project Final Report 105 teachers and pupils; 2.) organizing and conducting planning meetings twice per term; and, developing and implementing at least two activities from SPIP per school year. Achievement to date: The percentage of SMCs effectively supporting schools has increased from 13% at baseline to 51% at the end of Year 3. The Year 4 target has been reduced from 70% to 55% in order to accommodate the delayed rollout of SMC training, which took place in Year 2 instead of Year 1. Based on the evaluation findings, PAGE is anticipated to exceed its target. Performance evaluation findings: Over 60% of districts visited met this indicator. Nine of the 11 districts visited the school at least twice per term and interacted mostly with HTs. Five districts conducted planning meetings at least twice per term. And, seven districts implemented at least two SPIP activities in the last school year. Beboanu Primary School in Nkoranza district exhibited exceptional performance by completing 5 SPIP activities in one academic year. IR 2: Improved Quality of Education Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Indicator 2.3.1: % of CS making effective school visits 4% 85% 85% 10 90% Description: Circuit Supervisors make effective school visits if two of the following three apply: 1.) District records indicate CSs visit each school in their circuit at least twice per term; 2.) INSET training was provided by CSs for HTs in the district at least once per year; 3.) CSs meet with community members, representative of the community’s diversity, at least once per term. Achievements to date: According to PAGE monitoring reports, the target has been achieved, with 85% of CS making effective school visits as compared to only 4% during the baseline. Performance evaluation findings: All 10 Districts indicate CSs visit each school in their circuit at least twice per term. On average, each school was visited thrice last term, as opposed to once per term in the past. Ten districts indicate that CSs provided INSET for head teachers at least once per year. Four districts reported that CSs meet with community members at least once per term. This means that 90% of districts sampled have met the PAGE effectiveness criteria. At least four districts stated one or more of the following roles and responsibilities of the CS: visiting schools to monitor and supervise teaching and learning; delivering INSET for teachers/HT at least once per term in their circuit; providing professional guidance to HT and teachers and developing their capacity; vetting teachers’ lessons; observing classes; interpreting education policy; monitoring pupil/teacher attendance; and liaising between GES and the school. 3 districts stated the following role: In addition to the above, they mentioned organizing SPAM. 2 districts stated the following role of the CS: Reviewing pupil exercises to assess the work output of teachers and pupils; assessing teachers’ performance; carrying out demonstration lessons on challenging topics; guiding NALAP implementation; training SMC and community members/facilitating meetings, and ensuring that the SMC/PTA are working collaboratively with schools; and, inspecting the school environment and sanitation facilities to ensure that it is child-friendly and gender-sensitive. Final Evaluation of the USAID/Ghana PAGE Project Final Report 106 1 district stated the following role of the CS: Monitoring capitation grant activities; crosschecking SPIP activities against budget allocations; enforcing sanctions of absentee teachers, drunken teachers, etc.; educating both parents and pupils on the importance of education; and arbitrating between the community, teacher and HTs when problems arise. CS performance as viewed by stakeholders interviewed: DDEs rated CSs performance at 80% in terms of carrying out their visits, but 65% by way of performing their tasks. One DDE noted, “The DDE goes to the school and sees more than the CS sees. The purpose of the visit should be to help teachers out of their difficulty – to address critical issues. The CS visits, but he DEOs not monitor.” HTs, however, have been known to complain that the CS covers so many pages of the logbook when they come to visit as opposed to before. They claim the CS checks the attendance of teachers, reviews lesson notes, observes the classroom to assess teachers’ performance, and provides INSETs. Teachers confirm that CS’ promote effective teaching and learning by reviewing their lesson notes and pupils exercise books to assess work output and quality. Indicator Baseline Value CARE Monitoring Achievements Performance Evaluation Achievement (Frequencies) Target Actual Districts Percent Indicator 2.2.1: % of head teachers using effective school leadership and management strategies 10% 50% 56% 5 46% Description: A Head Teacher is considered to be demonstrating effective leadership and management strategies if two of the following three apply: 1.) Promotes a shared and focused school vision that supports early literacy development; 2.) Interacts with teachers: visits classrooms and provides feedback to teachers, organizes in-service training on literacy instruction for teachers each term, and displays the SPIP; or, 3.) Has completed at least one Head Teacher training in educational roles and responsibilities in the last academic year. Achievements to date: Compared to 10% of HTs using effective school leadership and management strategies at baseline, 56% are now applying such strategies as a result of PAGE interventions. Actual achievement to date exceeds the end of project target of 50%. Performance evaluation findings: The evaluation results are 10% lower than PAGE monitoring reports, with 46% of HTs meeting the effectiveness indicator versus 56%. The evaluation team observed that five schools had a vision supporting literacy and SPIP displayed. HTs also reported in five schools that they provided INSET focused on English, reading or language arts, and provided feedback to teachers following classroom observations or review of lesson notes. Four districts had completed one head teacher training within the last year. Several HTs were newly transferred so the number of trained HTs may appear lower than actual. Evaluation Question 5: What performance monitoring processes, systems and tools were used to ensure accurate, timely, reliable and valid performance and indicator tracking, reporting and feedback at all levels? How did this contribute to logical implementation of activities, data validation, documentation and review of work plans, indicators and /or activities? PAGE M&E tools, systems and processes used to track project performance Final Evaluation of the USAID/Ghana PAGE Project Final Report 107 During the start-up of the project, PAGE developed a performance monitoring plan (PMP) and tools to support data collection. A baseline survey was conducted to obtain data for monitoring performance indicators. The project set targets for each year of the project and defined roles for the M&E staff within the DEO and CARE. The project took advantage of existing data, data-collection instruments and processes applied under GRAIL and EQUALL to set up its own performance management plans. The M&E Officer developed the following data forms for monitoring PAGE activities: 1) PAGE District Reporting form – to be used on a quarterly basis to collect information on district and school activities; 2) Training (Trai Net) Reporting Form and Training Data Capture Template – used to collect basic information on all participants in PAGE training activities; 3) Community Match Reporting Form – an instrument for tracking in-kind match contributions; and 4) Enrolment form – developed to capture school enrolment data each academic year. PAGE used these forms for collecting data on eleven PMP indicators and reported progress to USAID through semi-annual reports. PAGE also carried out annual assessments and utilized the findings to design trainings and fine-tune implementation strategies Final Evaluation of the USAID/Ghana PAGE Project Final Report 108 ANNEX VII: PAGE M&E SYSTEM – PROCESSES AND TOOLS The M&E system is made up of the following:  Data collection process which is made up of: - A set of predefined data collection tools and formats used by the M&E officer, the CSs, the Girl Education Officer, the Community Volunteers and Head Teachers for data collection - Data collection is decentralized (i.e. HTs collect school based data and submit to CS, CS verifies data during school visits, collect other CS related data and submits data to the EMIS Department  Data Processing and analysis component (EMIS Department) - EMIS verifies field data by randomly visiting schools to cross check the data - EMIS enters data into SRC software and produces the SRCs for schools - EMIS also produces other reports such as enrollment data and other reports from the community volunteer surveys  Data dissemination and usage component - Various reports produced by EMIS are used at the Town Hall meetings, joint partnership planning meetings to produce ADEOP and SPAMs.  Feedback and Planning purposes: - EMIS reports SRC, enrollment data and community volunteer survey results are used by the DEO/DEOC in producing the ADEOP The schools and school communities also develop their SPIPs out of the SRC and other school-based reports they receive from the DEO/EMIS. Final Evaluation of the USAID/Ghana PAGE Project Final Report 109 ANNEX VIII: REFERENCES Cammack, John. Building Capacity through Financial Management: A Practical Guide. Oxfam GB, 2007. CARE International. “Partnership for Accountable Governance in Education (PAGE): Performance Monitoring Plan (PMP)”. February 2011. CARE International in Ghana. “Partnership for Accountable Governance in Education: Semiannual Report June – September 2010”. October 2010. CARE International in Ghana. “Partnership for Accountable Governance in Education: Semiannual Report (October 2010 – March 2011)”. April 2011. CARE International in Ghana. “Partnership for Accountable Governance in Education: Semiannual Report (April 1, 2011 – September 30, 2011)”. October 2011. CARE International in Ghana. “Partnership for Accountable Governance in Education: Semiannual Report (October 1, 2011 – March 31, 2012)”. April 2012. CARE International in Ghana. “Partnership for Accountable Governance in Education: Semiannual Report (April 1, 2012 – September 30, 2012)”. October 2012. CARE International in Ghana. “Partnership for Accountable Governance in Education: Semiannual Progress Report (April 1 – September 30, 2013)”. October 2013. CARE International in Ghana. “PAGE Performance Report.” October 2013. “Cooperative Agreement No. 641-A-00-10-00036-00, Partnership for Accountable Governance in Education Program”. June 2010. Nielson, Eldora Burns and Kirstin McCarthy. “USAID/Ghana Partnership for Accountable Governance in Education (PAGE): Baseline Survey Report”. CARE International. January 2011. Office of the Inspector General. “Audit of USAID/Ghana’s Partnership for Accountable Governance in Education Program.” February 2013. “PAGE Checklist for Activity Implementation.” “PAGE National Steering Committee: 8th Session Agenda.” August 1, 2013. "Republic of Ghana, Ghana Education Service, 2013 Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) Brief Report". October 2013. USAID/GHANA Country Development Cooperation Strategy: 2013-2017. December 2012. Final Evaluation of the USAID/Ghana PAGE Project Final Report 110 ANNEX IX: EVIDENCE MATRIX PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal Question 1: Have governance and supervision interventions at the district level resulted improved student achievement? If so, to what extent and why? If not, why not? Objective 1 Stakeholder Outreach  Activity materials, baseline assessment; Functional Organizational Assessment Tool (FOAT); Audit Report; PMP; performance indicator results DEO reports   District Assemblies; District Education Oversight Committees; SMC members Community members and Parents GoG Education policy Government workers with Internet; CBO and International NGOs Overarching Research Hypothesis Goal: If educational governance and supervision in basic schools is improved, then student achievement will also improve. H1: If the capacity of circuit supervisors, head-teachers, and SCMs to monitor, manage, and report on school management performance is strengthened, then educational governance and supervision will improve. H2: If the capacity of DEOCs to support and enable the competencies of circuit supervisors, head-teachers, and SMCs is strengthened, then educational governance and supervision will improve. Final Evaluation of the USAID/Ghana PAGE Project Final Report 111 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision Training materials; and reports; District annual plans; Manuals; procedures; DA or DEOC policies   DA, DEOC, DEO Community members and Parents Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program Manuals, training materials and reports DEO procedure revisions on Supervision   DEO and Circuit Supervisors and Head Teachers Pupils, teachers Parents and community members Objective 4 SMC/PTA capacity building SMC manuals that were distributed; Training plans and reports SMC and School Plans and sample meeting reports and minutes   SMC/PTA members Parents of children at the school NGOs working in Education in the District Question 2: What intended and unintended contributions, results and/or impact has the PAGE approach and activities achieved relative to improving governance and supervision in schools? Objective 1 Stakeholder Outreach  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports Public surveys; records/meeti ng minutes;  DA and DEOC Parents of children at the school Final Evaluation of the USAID/Ghana PAGE Project Final Report 112 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DEO reports  DEO; DEOC DA; parents of students attending school NGOs working in Education in the District Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports CS and Head Teacher reports; DEO reports  CS and Head Teachers; DEO personnel DA; parents of students attending school NGOs working in Education in the District Objective 4 SMC/PTA capacity building USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports School level reports; DEO reports  CS and Head Teachers; DEO personnel; Parents and SCM members parents of students attending school NGOs working in Education in the District Question 3: What factors affected the achievement and results of the project? What are the lessons learned and best practices, and from which stakeholders or beneficiaries? Objective 1 Stakeholder Outreach  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports   DA; DEOC Schools and Parents NGOs working in Education in the District DEO and GES personnel with Internet access Objective 2 DEOC-DEO collaboration building and development  USAID Audit Report; Performance indicator results; PMP; Evaluations of training/traini   DA; DEOC; GES; MOE; DEO Schools and Parents NGOs and CSOs working in DEO and GES personnel with Internet access Final Evaluation of the USAID/Ghana PAGE Project Final Report 113 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal of district specific framework for education governance and supervision Quarterly and Semi￾annual reports ng reports Education in the District Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports Evaluations of training/ training reports DEOC; DEO; SCs; and HT Schools and Parents NGOs and CSOs working in Education in the District DEO and GES personnel with Internet access Objective 4 SMC/PTA capacity building USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports SMC/PTA meeting minutes SMC/PTAs; school personnel; DEO; DEOC Schools and Parents NGOs and CSOs working in Education in the District DEO and GES personnel with Internet access Question 4: What are the best ways to ensure that progress and results are captured and/or continued that also promote ownership, engagement and sustainability of interventions after the project ends? Objective 1 Stakeholder Outreach  USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DEO reports   DA, DEOC, DEO Schools and Parents NGOs and CSOs working in Education in the District Objective 2 DEOC-DEO collaboration building and development  USAID Audit Report; Performance indicator results; PMP; DEO reports   DA, DEOC, DEO Schools and Parents NGOs and CSOs working in Final Evaluation of the USAID/Ghana PAGE Project Final Report 114 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal of district specific framework for education governance and supervision Quarterly and Semi￾annual reports Education in the District Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DEO reports DA, DEOC, DEO Schools and Parents NGOs and CSOs working in Education in the District Objective 4 SMC/PTA capacity building USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports School reports DA, DEOC, DEO Schools and Parents NGOs and CSOs working in Education in the District Question 5: What performance monitoring processes, systems and tools were used to ensure accurate, timely, reliable and valid performance and indicator tracking reporting and feedback at all levels? How did this contribute to logical implementation of activities, data validation, documentation and review of work plans, indicators and/or activities? Objective 1 Stakeholder Outreach Quarterly and Semi￾annual reports DA, DEOC, DEO Schools and Parents Objective 2 DEOC-DEO collaboration building and development of district specific framework for education governance and supervision USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DA, DEOC, DEO Schools and Parents Final Evaluation of the USAID/Ghana PAGE Project Final Report 115 PAGE Evidence Matrix Activities and data sources are subject to change based on factors in the field Documents/Research Semi-Structured Interviews Mini-Survey, Focus Group USAID Im pleme nter Secondary USG Personnel CARE International Staff Direct Participants (Implementing and Key Partners) Indirect Beneficiaries E xter nal Objective 3 Training and resourcing of circuit supervisors and head-teachers and school supervision program USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DA, DEOC, DEO Schools and Parents Objective 4 SMC/PTA capacity building USAID Audit Report; Performance indicator results; PMP; Quarterly and Semi￾annual reports DA, DEOC, DEO Schools and Parents Final Evaluation of the USAID/Ghana PAGE Project Final Report 116 ANNEX X: USAID EVALUATION REPORT REVIEW PROCESS