HEALTH, EDUCATION AND RECONCILIATION (HEAR) PROJECT MID-TERM EVALUATION MAY, 2010 This publication was produced for review by the United States Agency for International Development. It was prepared by Thomas Tilson, Ph.D. and Andrew Epstein, M.Ed., Management Systems International. HEALTH, EDUCATION AND RECONCILIATION (HEAR) PROJECT MID-TERM EVALUATION Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contracted under DFD-I-00-05-00251-00, Task Order No. 2 USAID/Sudan SUPPORT Project Juba, Sudan HEAR Implementer: Creative Associates International HEAR Evaluation: Management Systems International (MSI) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Health, Education And Reconciliation (HEAR) Project i Mid-Term Evaluation CONTENTS ACKNOWLEDGEMENTS ................................................................................................ii ACRONYMS......................................................................................................................iii 1. PROJECT SUMMARY.................................................................................................. 1 2. EXECUTIVE SUMMARY ............................................................................................ 1 3. BACKGROUND........................................................................................................... 6 4. PURPOSE OF EVALUATION.................................................................................... 7 5. PROJECT DESCRIPTION........................................................................................... 7 6. EVALUATION METHODOLOGY............................................................................. 8 7. EVALUATION FINDINGS ....................................................................................... 10 A. Project Site Context............................................................................................................. 10 B. Project Design:....................................................................................................................... 13 C. Project Activities................................................................................................................... 14 D. Partners................................................................................................................................... 23 E. Management ........................................................................................................................... 24 F. Project Achievements and Impacts..................................................................................... 25 G. External Variables and Sustainability.................................................................................. 30 8. CONCLUSIONS & RECOMMENDATIONS........................................................... 31 General Conclusions .................................................................................................................. 32 Education...................................................................................................................................... 32 Health............................................................................................................................................ 34 Reconciliation & Governance.................................................................................................... 35 Gender Equity.............................................................................................................................. 36 Education Resource Centers..................................................................................................... 37 Partners ........................................................................................................................................ 39 Management................................................................................................................................. 40 Project Achievements and Impacts .......................................................................................... 41 External Variables and Sustainability ........................................................................................ 43 APPENDICES.................................................................................................................. 45 Appendix A Work Plan............................................................................................................. 45 Appendix B Activities Funded by Small Grants..................................................................... 54 Appendix C Schedule................................................................................................................ 55 Appendix D Evaluation Questions.......................................................................................... 60 Appendix E Tables on Deliverables ........................................................................................ 62 Appendix F Possible Success Stories ...................................................................................... 64 Appendix G Interview and group discussion protocols ...................................................... 65 Appendix H Documents Reviewed......................................................................................... 71 Appendix I Classroom Observation Tool.............................................................................. 72 Health, Education And Reconciliation (HEAR) Project ii Mid-Term Evaluation ACKNOWLEDGEMENTS The authors wish to acknowledge the support and cooperation of many people and organizations without whom this evaluation would not have been possible. First and foremost, we would like to express our gratitude and admiration for the people of Sudan, whose endurance, resilience, and strength of spirit inspired the authors to work tirelessly on their behalf. Many people contributed to this evaluation, but special acknowledgment goes to Tassew Zewdie, Ayella Cosmas, Charles Karumba, Mary Mirigo, Japheth Mugumo, Worku Tegegne, Paul Ssali, Salome Musili, Jimmy Yugga, and Santino Samuel of Creative Associates, International. From EDC, Kent Noel and Leonard Odini were especially helpful. We would also like to give a special thanks to the USAID staff who provided guidance and support throughout: Peter Malnick, Leesa Kaplan-Nunes, Pia Philip, and Anyeth Ayen, who was able to travel with the Evaluation Team to Kurmuk. In addition, the support team at Management Systems International, Inc., especially Ami Henson, did an exceptional job providing guidance and logistical support to the team, which was critical to the success of the evaluation. We are also grateful to the staff of the Secretariats of Health and Regional Education Offices in Kurmuk and Kauda for their support during our visit. At all the schools and communities that we visited, the head teachers, teachers, PTA members and parents could not have been more gracious and cooperative. Finally, we would like to express our appreciation to the Lutheran World Federation and Norwegian Church Aid for accommodating the authors while in the field. Health, Education And Reconciliation (HEAR) Project iii Mid-Term Evaluation ACRONYMS BCC Behavioral Change Communication BRIDGE Building Responsibility for the Delivery of Government Services CAII Creative Associates International, Inc. CBO Community Based Organization CHP Community Health Providers CPA Comprehensive Peace Agreement CTO Cognizant Technical Officer EDC Education Development Center ERC Education Resource Center EMIS Education Management Information System FRESH Focusing Resources of Effective School Health GEE Gender Equity through Education GNU Government of National Unity GoSS Government of Southern Sudan HEAR Health, Education and Reconciliation Project IRI Interactive Radio Instruction KTTC Kauda Teacher Training College LEA Language Experience Approach MC Scotland Mercy Corps Scotland MSI Management Systems International NCA Norwegian Church Aid PCI Pupil Centered Instruction PTA Parents Teachers Association Rabea Radio Based Education for All REO Regional Education Office ROOF Relief Organization of Fazugli SGC Small Grants Committee SOH Secretariat of Health SPLA Sudan People’s Liberation Army SPLM Sudan People’s Liberation Movement SSIRI Southern Sudan Interactive Radio Instruction Project SUPPORT Services Under Program and Project Offices for Results Tracking SOW Scope of Work UNICEF United Nations Children’s Fund USAID United States Agency for International Development WHO World Health Organization Health, Education And Reconciliation (HEAR) Project 1 Mid-Term Evaluation 1. PROJECT SUMMARY Project Name: Health, Education and Reconciliation (HEAR) Project Contract No. EDH-I-00-05-00026-00 Implementing Partner: Creative Associates. The consortium includes the Education Development Center (EDC) and John Snow International (JSI). Mechanism: Contract through an Indefinite Quantity Contract funding mechanism, the Assistance to Basic Education/Linkages to Education and Health Initiative, known as ABE￾LINK. Cost Plus Fixed Fee Contract. Contracting Officer Technical Representative (COTR): Pia Philip Start Date: October 11, 2006 Planned End Date: Originally September 30, 2009, extended to September 29, 2011 Total HEAR Est. Original Cost $3,000,000. New ceiling $8,509,098 Geographic Focus: Sudan, The Three Areas: Abyei, Southern Kordofan, Blue Nile 2. EXECUTIVE SUMMARY An Evaluation Team was led by Management Systems International (MSI) under its SUPPORT contract with USAID/Sudan during February 2010 to conduct the mid-term evaluation of the Health, Education and Reconciliation (HEAR) Project. The purpose of this mid-term evaluation was: 1. To assess the mid-term performance of Creative Associates International, Inc. (CAII) and the HEAR Project regarding expected outcomes and contractual deliverables. 2. To assess the efforts of the HEAR Project to coordinate with the BRIDGE Project and other USAID/Sudan projects. 3. To assess the efforts of the HEAR Project to coordinate with non-USG implementing partners and other stakeholders. 4. To make recommendations towards potential scale-up, short-term adjustments to the contract, and potential follow-on programs. In the entrance interview with USAID, the Evaluation Team was also asked to assess the effectiveness of HEAR Project’s integrated model of health, education, and reconciliation/governance. Summary of Key Findings Overall, the Evaluation Team found that the HEAR Project is functioning effectively, especially over the last two years. It is fully functional in Southern Kordofan1 and Blue Nile State and is just restarting activities in Abyei following a destructive military incident which occurred there in May 2008. Due to the security situation in Abyei, the evaluation included only the project sites of Blue Nile and Southern Kordofan. In Kurmuk (Blue Nile State), the HEAR Project has organized and worked with two clusters of schools and communities for a total of 20 schools; in Kauda (Southern Kordofan), the HEAR Project works in four clusters with a total of 44 schools. The HEAR Project is ready to embark on another cluster of approximately ten schools in each state. Education Resource Centers (ERCs) have been established by the 1 Southern Kordofan is also known as Nuba Mountains. Health, Education And Reconciliation (HEAR) Project 2 Mid-Term Evaluation HEAR Project in both Kurmuk and Kauda. The ERCs support the training programs, house a library of books, and provide computer training and Internet service. HEAR also works in roughly 32 schools in Abyei, for a total of 96 schools in the Three Areas. Regarding the integration of health and education, the Evaluation Team found strong evidence of health activities and messages embedded into the Project, although evidence of the education component is not as strong. While the Reconciliation and Governance component lacks clarity in implementation, the focus on school governance through the Parent Teacher Associations (PTA) and head teacher trainings shows promise. Summary of Key Findings and Recommendations The HEAR Project responds to the USAID objective of investing in people under the program areas of education and health. A key justification for the HEAR Project is to support the Comprehensive Peace Agreement (CPA) by bringing peace dividends to the citizens of Sudan in the Three Areas; this is a priority of the United States Government (USG). The HEAR Project activities are effective in enhancing the demand for and overall supply of social services. This is achieved by strengthening capacity at the school and community level through training of teachers, head teachers, Community Health Promoters (CHP), and Parent Teacher Association (PTA) members. The training has helped to increase the demand for education and health services, which is evidenced, for example, by swelling school enrollments. The Project’s work with the Regional Education Offices and the Secretariats of Health and the Project’s support to the schools and communities has helped to build capacity and is resulting in improved delivery of services, increased accountability, and good governance. The distinguishing feature of the HEAR Project is its intersectoral approach which merges support to schools and communities through the training of teachers, head teachers, CHPs, and PTA members with overall coordination by the Regional Education Offices and the Secretariats of Health. As an integrated health and education project, the health activities are progressing especially well, but there has been less impact on teaching behaviors. Health The primary finding of the evaluation is that the HEAR Project has focused on health/hygiene and disease prevention education at the school and community level through an integrated approach, training teachers and community health promoters and distributing vitamin A, deworming pills, and bednets. While the Evaluation Team saw clear evidence of good health messages, it also saw mixed results in the implemetation of actual health practices such as the use of latrines and hand washing. All constituencies reported that the health of the children, and subsequently their attendance in school, had improved over the last two years. A major challenge to improved health behaviors is the lack of access to clean water, as observed by the evaluators and reported by a majority of individuals asked. The Evaluation Team noted that this lack of access to water limits the full realization of health impacts as described in the project activities’ objectives and goals. While it is clear to evaluators that health messages are being disseminated, it remains difficult to confirm actual behavior change. The project has been successful in soliciting donations of vitamin A and in distributing the tablets, and is on track to reach the project target for vitamin A. However, the distribution of bednets has been slow, as procurement has been difficult and 1,500 of the nets procured were destroyed during the 2008 fighting in Abyei. Health, Education And Reconciliation (HEAR) Project 3 Mid-Term Evaluation The evaluators recommend that USAID and implementing partners access organizations with the capacity to drill boreholes in or near the school sites in partnership with the HEAR Project (for example, GOAL in Kurmuk and BRIDGE and PACT in Kauda). Education The Project has not been particularly successful in improving the skills of teachers outside of health education. This outcome stems, in part, from the design of the project. The initial design of the HEAR Project included Interactive Radio Instruction (IRI) activities that taught children mathematics, English, local language literacy, and life skills and incorporated some teacher training and models of good teaching skills and methodology. These activities were transferred from the HEAR Project to the southern Sudan Interactive Radio Instruction (SSIRI) Project in 2008 and were not replaced with another education component. The Evaluation Team found that teacher training materials are heavily weighted toward health messages at the expense of any education or reconciliation topics. Some of the good teaching methods modeled during the training on the health topics could be applied to other topics, but the potential applicability does not presently seem to be included in the training. The evaluators also found that the conditions necessary for teachers to implement many of the teaching methodologies advocated by the HEAR Project do not exist in most schools. Finally, the structure and delivery schedule of the training regimen do not address sustainability issues. CAII recognizes these issues and has plans to strengthen the education component. The evaluators recommend that training materials be revised to reflect the realities of schools and the methodological needs of the teachers. Opportunities for ongoing follow-up and continuing training for HEAR Project teachers should be offered. Training for head teachers should be decoupled from the PTA training and recalibrated as “training-of￾trainers” (TOT) focusing on school leadership, records management, community relations, the completion of a school development plan, and the ability to facilitate HEAR Project teacher training for new or untrained teachers as well as refresher trainings at his/her school. This training should be provided for Regional Education Office staff as well. Reconciliation and School Governance Both PTA and school governance activities support reconciliation. PTAs were in existence at all schools visited. While originally the PTA/Head Teacher training program was to center on the elaboration of School Development Plans, this proved overly ambitious; the actual training focused instead on the development of small grant proposals, which has been very beneficial. The creation of School Development Plans would require more intensive training, and the Evaluation Team did not find evidence that any school had completed a School Development Plan. The development of small grant proposals required a process similar to the one used in the creation of a School Development Plan and provided more immediate and tangible benefits to the schools and communities. Twenty-six projects have been initiated in 25 of the HEAR Project schools and are used to construct classrooms, kitchens, water tanks, latrines, and fences, as well as to sponsor School Open Days. The Evaluation Team noted that there were not enough funds remaining for each school to obtain a small grant. PTAs were primarily composed of men, who frequently dominated the decision-making process. HEAR Project staff voiced concerns about the sustainability of this project component without further training or resources. The evaluators recommend that the HEAR Project create multi-year service plans for each cluster to ensure that support activities are ongoing in the years following the initial training and service regimen, as well as to direct HEAR Project staff to engage in specific activities to meet this objective. Health, Education And Reconciliation (HEAR) Project 4 Mid-Term Evaluation The PTA training should maintain a focus on small grant development. The Project should consider providing additional resources to grants, revising the grant agreement, and designating a small grants manager at each site. Educational Resource Centers The ERCs are the training hub of the project in each state. The centers organize the various trainings, provide a venue for training activities, offer textbooks and other materials in English, and teach computer skills. The ERCs have a computer database for storing information on their books and CDs and are in the process of creating an improved system for documenting the books that are distributed to schools. In both ERC locations, the computer training includes teachers, primary school students, and officials from the education and health offices. Both ERCs have had initial experience with creating simple videos. In general, the ERC activities are progressing well. The vision and strategy for the role of the ERCs, however, seem to be unclear, especially with respect to how best to use its books and computers for strengthening education. While it is likely the computer training and the use of the ERC library will contribute to teachers’ knowledge of subject matter, there is no way to be certain of this at this time. The Project is drafting a new strategy paper for the ERCs. The evaluators recommend that ERCs provide simple orientation or training to schools that receive the books and align appropriate books with the New Sudan curriculum and the Kenyan curriculum. ERCs should also try to obtain more textbooks and teacher's guides from the Kenyan and New Sudan syllabi and more English reading and picture books, especially for lower primary. In response to comments by the evaluators, HEAR is revising the computer training program to place greater emphasis on training teachers and officials and offer more hands-on experience for all trainees, including students. The HEAR computer staff have an initial list of useful websites for teachers and will be expanding this list based on the teacher’s experiences on the Internet. The ERCs need to build upon their initial work with video production and determine how best to use this technology to support HEAR objectives. The Project also needs to consider options for increasing access to the Centers by teachers in remote schools. Model School Proposal The HEAR Project proposes the development of Model Schools (in their year 3 Work Plan) in order to demonstrate how project objectives can be achieved and replicated in other area schools. These Model Schools would be models for all aspects of the HEAR Project training, including all of the teaching techniques, health behaviors, and school governance activities. The evaluators recommend that the Project does not try to implement the Model Schools during the current contract period unless more funding is provided. Should more funding materialize, specific recommendations on how to proceed sustainably with the Model Schools proposal are provided in the next section. Gender Equity Forty-five percent of the students enrolled in the 13 schools visited by the evaluators were girls. Although not parity, this percentage of girls in school is encouraging. Twenty-one percent of the teachers and seven percent of the administrators are female; the evaluators did not encounter a single female head teacher. These low figures represent the reality on the ground. Changing cultural and economic conditions have improved the prospect for gender equity in the region, but cultural and economic barriers remain and retention of girls remains an issue for many schools. In addition, the Evaluation Team found that the great majority of PTA members are male, and they tended to dominate the discussion and decision-making processes. While little explicit training or direct support in Health, Education And Reconciliation (HEAR) Project 5 Mid-Term Evaluation gender equity is currently provided by the Project, there are some opportunities for such activities, which will be discussed below. The evaluators recommend that gender equity issues be added to all training of teachers, PTAs, and CHPs. Also, the PTA selection process should be managed so as to increase the representation and subsequent participation of women. Partners The HEAR Project is coordinating with the BRIDGE Project in Abyei (Abyei was not included in the evaluation SOW) and is beginning to do so in Kauda as Mercy Corps Scotland gets reestablished. In addition, the Project is coordinating with SSIRI, GEE, and the Internews community radio stations. The HEAR Project collaborates with several non-USG implementing partners. For example, the Project has relied heavily on the World Health Organization (WHO), United Nations Children’s Fund (UNICEF), GOAL, and Save the Children for obtaining vitamin A, deworming pills, and bednets for distribution. Some of the other organizations with which the HEAR Project collaborates include Relief Organization of Fazugli (ROOF) and Norwegian Church Aid (NCA). The HEAR Project has also successfully established relationships with key government partners, the Regional Education Offices, and the Secretariats of Health. The evaluators recommend that the Project seek new partnerships, or the strengthening of existing ones, in carrying out a number of the recommendations of this report, particularly on issues pertaining to water, gender access and equity, and model schools. Management Through discussions with HEAR Project staff, the Evaluation Team found that, in spite of the high turnover of Chiefs of Party (three Chiefs of Party and two Interim Chiefs of Party), the project is generally well￾managed. Nevertheless, the turnover of COPs has created difficulties. Additional challenges remain which are beyond the control of the project and stem from limited infrastructure, including transportation, banking, etc. The staff also had several concerns related to internal Human Resources issues. The high turnover of COPs has been offset, in part, by stability among several senior field staff and the strong support for the CAII regional office in Nairobi. The Project seems to have excellent working relationships within the education and health offices in both states, although the Project is just beginning to work with the new Regional Health Director in Kurmuk. The USAID CTO for the HEAR Project reports excellent communications with CAII and strong management in spite of the high turnover of COPs. The evaluators recommend that the project leadership review internal HR issues and continue to address the root causes of COP turnover. Project Achievements and Impact The HEAR Project is on target for meeting the deliverables in the contract. The table on page 27 shows the current status of the Project in terms of meeting the life-of-project targets. However, the evaluators have two additional observations. First, the Project design does not include efforts to measure directly the impact of the Project on health or pupil learning. Second, HEAR has collected little information on the impact of the Project in terms of behavioral changes at the school and community level, and it collects only minimal data on the activities of community health promoters. The evaluators do not think it is worthwhile for the Project to try to assess potential gains in student learning, especially given the limited training on teaching skills. Also, the evaluators do not think it would be useful to Health, Education And Reconciliation (HEAR) Project 6 Mid-Term Evaluation try to measure directly improved health of the children. Rather, the Project should strengthen its monitoring activities and document the desired behavioral changes at schools, such as washing hands and use of latrines. If there were a substantial improvement in these behaviors along with the distribution of vitamin A and bednets, it would be safe to assume that there should be improved health. In addition, the new tracking tool for CHPs—already designed by HEAR Project staff—should be completed as soon as possible and integrated into the regular reporting mechanisms. The Project should consider a comparative study with schools outside of the HEAR Project in terms of measuring health behaviors and teaching practices that relate to HEAR training. Sustainability The HEAR Project is aware of the importance of addressing sustainability and has done so in their 2010 Work Plan. However, there are external factors outside of the control of the Project that will directly affect the sustainability of the Project. Given the fragility of the current political situation and the limited GNU support to the schools and communities supported by HEAR, there should not be high expectations for sustainability in the near term. Higher school enrollment suggests strong evidence of peace dividends. In order to sustain such rapid growth, many complex and deeply political issues, including teacher compensation, training, recruitment, and retention, must be addressed. Most of these issues are outside the control of the HEAR project but will affect potential sustainability. The evaluators make sustainability-related recommendations in the respective sections, particularly those addressing training, water, and local partnerships. 3. BACKGROUND Southern Sudan and the critical border areas (consisting of Abyei and the northern states of Southern Kordofan and Blue Nile2 – commonly referred to as the Three Areas) have endured years of underdevelopment, conflict, drought, and floods, producing a crisis of enormous proportions across the region and resulting in the devastation of economic, political, and social structures. In addition to the loss of lives, opportunities, and infrastructure, the civil strife displaced families and divided communities. As a consequence, the health, education, and infrastructure status of the Sudanese people are among the poorest globally. Sudan’s warring parties signed the Comprehensive Peace Agreement (CPA) in January 2005. Since that time, the country has taken steps toward peace, reconciliation, and good governance—although the pace has been slower than expected or desired. While these three areas come under the jurisdiction of the Government of National Unity (GNU)3 , or “the North,” a large segment of the population—including school teachers and administrators, as well as some holding local education and health ministry positions—are sympathetic to the social and political objectives of the Government of southern Sudan (GoSS) or “the South.” The resulting complexities and conflicts in governance, especially the manner in which health and education services are delivered, present day-to-day challenges for those who live in the three areas, as well as for organizations 2 Under the Comprehensive Peace Agreement, Abyei was established as a special geographic area under the Presidency; Southern Kordofan and Blue Nile are administered by the Government of National Unity (GNU). The ten states in the South are administered by the Government of Southern Sudan (GoSS). 3 Abyei, the only one of the Three Areas, will vote in 2011 whether to become part of the north or the south. Health, Education And Reconciliation (HEAR) Project 7 Mid-Term Evaluation delivering humanitarian aid and development; the way this impacts the HEAR project in particular is detailed later in this report. In Kurmuk, the county offices of the Ministry of Education and the Secretariat of Health are supported by the Ministries in Damazine. The staff is paid, and the activities are seen as part of the Ministry activities for the state (except for English pattern schools, which receive no support). The situation is different in Kauda, where the education and health offices have very little connection with the Ministry offices in Kadugli and receive no financial support from them. The Health, Education, and Reconciliation (HEAR) Project responds to the United States Agency for International Development (USAID) objective of investing in people under the program areas of education and health. When the Project was initially funded in 2006, it was designed to address USAID’s then Strategic Objective (SO) 6: “Improved Equitable Access to Quality Education” which formed part of the USAID Sudan Interim Strategic Plan (ISP). The ISP goal was “foundation established for a just and durable peace with broad participation of the Sudanese people.” Under the Fragile State Strategy, the USAID Sudan Mission later shifted its strategic objectives, and the HEAR Project in turn worked to address SO No. 9 “Avert and Resolve Conflict” and SO No. 10 “Promote Stability, Recovery and Democratic Reform.” A key justification for the HEAR Project is to support the CPA by bringing peace dividends to the citizens of Sudan in the Three Areas, which is a priority of the United States Government (USG). 4. PURPOSE OF EVALUATION An Evaluation Team was deployed by Management Systems International (MSI) in February 2010 to conduct the mid-term evaluation of the HEAR Project. The purpose of this mid-term evaluation was: 1. To assess the mid-term performance of Creative Associates International, Inc. and the HEAR Project regarding expected outcomes and contractual deliverables; 2. To assess the HEAR Project in the context of its coordination with the BRIDGE Project and other USAID/Sudan projects; 3. To assess the HEAR Project in the context of its coordination with non-USG implementing partners and other stakeholders; and 4. To make recommendations towards potential scale-up, short-term adjustments to the contract, and potential follow-on programs. The evaluation team was also asked to assess the effectiveness of HEAR Project’s integrated model of health, education, and reconciliation/governance. (Please see Appendix D for a complete list of questions posed). 5. PROJECT DESCRIPTION The overarching goal of the HEAR Project is to increase access to quality education for healthy girls and boys through community support and action. With increased community involvement, the HEAR Project aspires to promote quality education, support good school governance, and improve the delivery of health services. By the end of the project, USAID will have increased student enrollment in target schools, strengthened health practices, improved the quality of teaching, helped to build effective Parent Teacher Associations (PTAs), and strengthened linkages between communities and local authorities. The HEAR Project trains head teachers, teachers, PTAs, and community health promoters (CHPs) at the school and community levels. The Project also develops resource materials and community-based projects to reinforce student learning, engage service providers in delivering effective health and hygiene messages to community members, and rehabilitate and construct additional facilities at schools. Health, Education And Reconciliation (HEAR) Project 8 Mid-Term Evaluation The overall project goal is being pursued through the realization of three interrelated and interdependent project objectives including: 1) promoting education, 2) promoting health and 3) strengthening school governance through community groups. The Project has designed activities that collectively “increase the access of healthy girls and boys to quality education through community support and action.” The HEAR Project activities emanate from the Educational Resource Centers (ERCs) in each of the Three Areas and provide services to clusters of schools and communities. The HEAR Project envisions that ERCs will: • Manage training activities in supported schools • Establish and maintain a library • Promote education technologies, especially computers and the Internet The HEAR Project has several key activity areas. There is a seven-month cycle for groups of ten schools (one cluster) around which the principal project interventions are organized. Once a group of ten schools has completed the seven-month cycle, another group of ten schools begins its cycle. Two clusters in Kurmuk and four in Kauda totaling 64 schools have already taken part in the following activities: • Five-day teacher training • Two two-day teacher refresher courses • School visits • CHP training • PTA/Head Teacher training • Small grants program • Enhancing and promoting education technology in the ERCs • School open days • Distribution of Vitamin A and bednets • Monitoring and evaluation (M&E) activities The HEAR Project has supplied seven computers to each ERC and approximately 11,000 textbooks and other materials to be divided among the Three Areas, some of which have already been distributed to schools. Eventually, there are to be 20,000 learning resource manuals and 12,000 donated supplementary reading materials for teachers and administrators to have a well-stocked resource site. 6. EVALUATION METHODOLOGY The mid-term evaluation of the HEAR Project was undertaken between February 6 and March 6, 2010. The Evaluation Team consisted of two external evaluators (Thomas Tilson, Ph.D. and Andrew Epstein, M.Ed.), a USAID representative to Kurmuk, and representatives from Creative Associates International, Inc. (CAII). As mandated by the Scope of Work (SOW), this evaluation was conducted in a collaborative manner and included representatives from key stakeholder entities. The evaluation was contracted under the MSI/USAID SUPPORT Project (Project number 3903-001). The Evaluation Team conducted evaluation activities in Nairobi, Kenya, and in Juba, Rumbek, Kurmuk, and Kauda in Sudan. The Evaluation Team held briefings with MSI and USAID representatives in Juba and the CAII HEAR Acting Chief of Party (COP) and the Educational Development Center (EDC) Regional Director in Nairobi prior to commencing the evaluation. Based on these briefings, the Evaluation Team developed a work plan (Appendix A) and observation and interview protocols (Appendix G) designed to provide answers to the evaluation questions posed in the SOW. Health, Education And Reconciliation (HEAR) Project 9 Mid-Term Evaluation Methods employed by the Evaluation Team to answer the primary research questions involved five information gathering approaches: document review, interviews, group discussions, site observations, and school surveys. Document Review The Evaluation Team prepared for the evaluation by reviewing project documents provided by MSI, CAII, and EDC including: • All contract agreements and subsequent modifications • Project reports, including annual reports, quarterly reports, bi-weekly reports, and other reports as deemed appropriate • Training manuals and educational resources • M&E data, including individuals trained, school monitoring reports, etc. Interviews and group discussions4 The Evaluation Team conducted interviews with the following persons: • Relevant HEAR Project staff in Kauda and Kurmuk, including the Team Leaders/ERC Managers, M&E Coordinator, education and health staff, and the Education Technology Specialist. • Local/regional education and health ministry officials in Kauda and Kurmuk • SRRC directors in Kauda and Kurmuk • Representatives from partner organizations at each site, including Internews Radio, MC Scotland, GOAL, PACT, and ROOF. • Health workers at local clinics and hospitals • USAID COTR and other officials The Evaluation Team conducted group discussions with teachers, PTA members, Community Health Promoters (CHPs), and other community members at the following schools and training sites: Kurmuk Zeriba Primary School Panamayo Primary School Chali Rasha Primary School Borfa Primary School Kurmuk Model Primary School Comboni Primary School Jorot East Primary School Jorot West Primary School HEAR Project Teacher Refresher Course Kauda Alhilu Primary School Ardikanan Primary School Badura Primary School Karindi Primary School Kudi A Primary School Lower Kumo Primary School Kauda Teacher Training Center Yousif Kuwa Teacher Training Institute Observations Observations were conducted on the grounds and in classrooms of all the schools listed above. The Evaluation Team used a classroom observation tool (Appendix I) to assess if and how teachers were using 4 Some of the discussions could be properly defined as “focus group” discussions based on a definition such as the following: "carefully planned discussion designed to obtain perceptions in a defined area of interest in a permissive, non-threatening environment." Other discussions were less structured. For the sake of consistency, we will refer to all such discussions as just “group discussions.” Health, Education And Reconciliation (HEAR) Project 10 Mid-Term Evaluation the skills and knowledge disseminated in the HEAR Project training. Observations of computer training courses and uses of the library were conducted in the ERCs. Finally, the team observed a teacher training refresher course and a CHP training refresher course in Kurmuk. School Survey Data at each school site visited by the team were collected using a survey tool. Information on enrollment, faculty, physical structures, food and water, and other resources were recorded with the help of the HEAR Project Monitoring and Evaluation (M&E) specialist who accompanied the team on their school visits at each site. Comments on methodology This evaluation is primarily a qualitative, rather than quantitative, study. No research was done to statistically confirm whether children are healthier or learning more, nor were observations conducted in a manner that allowed the Evaluation Team to verify that children and their families had adopted the behaviors advocated by teachers, PTA members, and CHPs. Conclusions as to project impact on health and achievement are based solely on the perceptions of those participating in interviews and group discussion. The Evaluation Team has, however, employed quantitative data measuring the contractually-based targets against the actual numbers obtained, e.g., number of teachers trained, number of pupils in the associated schools, and the number of vitamin A pills distributed. 7. EVALUATION FINDINGS A. Project Site Context The two sites visited by the Evaluation Team have important similarities and differences that are significant to the design, implementation, and impact of the HEAR Project. Kurmuk The Kurmuk region is primarily an Arabic-speaking environment and is mostly integrated into the northern government structures. A preliminary inquiry to the Education Ministry and HEAR Project staff revealed that 13 of 64 schools are English pattern schools; the remaining schools are Arabic pattern. Teachers in English pattern schools are unpaid, whereas those in Arabic pattern schools receive salaries, as well as teachers’ guides and textbooks, from the Government of National Unity (GNU) in northern Sudan. This discrepancy is reasoned, according to Ministry officials interviewed, because teachers in English pattern schools are not formally trained and/or certified. Indeed, most of the teachers in the English pattern schools are returnees or internally displaced persons (IDPs) who attended schools in Ethiopian refugee camps where they were able to learn English. Their English, however, remains mediocre to poor, and evaluation interviews required Arabic translation in almost every case. The New Sudan curriculum developed by the Ministry of Education, Science and Technology of the Government of Southern Sudan is used in these schools, even though it is not supported by the Regional Education Office (REO). Consequently, there is a severe shortage of textbooks and curriculum materials. A teacher training institute recently opened that primarily supports in-service training of teachers working in Arabic pattern schools; there are no English language teacher training institutes in the region. Ironically, many teachers in the English pattern schools—some whom were trained by the HEAR Project—were recently recruited by the REO to teach English in Arabic pattern schools (which is taught as a subject starting in grade five), lured away by the promise of salaries. The education officials in Kurmuk take advantage of HEAR Project resources, including the ERC computers, computer training (the officials have two computers of their Health, Education And Reconciliation (HEAR) Project 11 Mid-Term Evaluation own, but very limited skills), and internet connections. In addition, these Ministry officials regularly hitch rides on HEAR Project vehicles in order to visit remote schools. Teachers in Arabic schools, on the other hand, have better qualifications—usually a secondary school certificate—and better access to teacher's guides and student textbooks. Half of the schools currently participating in the Kurmuk HEAR project are Arabic pattern, while the other half are English pattern. The County Minister of Health in Kurmuk is more supportive of English as a language of instruction, as illustrated by his leadership of the post-primary Health Sciences Academy. The Academy provides instruction and certification in Medical Assistance, Nursing, Pharmacy, Midwifery, and Lab technology. There is strong potential for partnership between the Health Ministry and English pattern schools in Kurmuk. Kauda In Kauda, by contrast, the Regional Education Office (REO) has responsibility for a much larger number of schools and teachers—approximately 228 English pattern schools and 1500 teachers. Also in contrast to Kurmuk, the REO receives no support from the GNU. The officials, as well as all staff and teachers, are volunteers. Typically, the teachers receive some support from their communities through the collection of school fees. The English language skills of teachers in Kauda are stronger than in Kurmuk: when interviewing teachers, a translator was rarely necessary. There are two teacher training institutions in Southern Kordofan - the Yousif Kuwa Teacher Training Institute (TTI) and the Kauda Teacher Training Center (KTTC). Most of the teachers encountered by the evaluation team had attended one of these institutions or had attended the southern Sudan “Fast Track” teacher training regimen. Although most teachers did not have more than a grade eight education, some had attended secondary schools, and two teachers that the Team met had a university education. As a result, the teachers in Kauda seemed to have a more sophisticated understanding of teaching methodologies. The schools visited by the Team typically used the New Sudan curriculum in the lower primary grades and the Kenyan Primary curriculum in the upper grades. Due to exploding enrollment, there has been a huge increase in recent years in the number of students taking the Kenyan Primary leaving exam; however, the pass rate for the exam is only 1.5 percent.5 The Secretariat of Health (SOH) is also unsupported. It has 4,000 staff, only one of whom is a doctor and all of which are volunteers. They have no drugs or supplies of their own and must rely on non-governmental organizations (NGOs) for these items. Both a health and education regional official sit on the HEAR Project small grants review committee, together with HEAR Project staff, to review and approve applications submitted by PTAs in HEAR Project schools. Both regions present difficult conditions under which to operate a project such as HEAR. There are no good roads, no banks, and only one flight per week from southern Sudan, which makes the transfer of equipment, money, and personnel difficult and time-consuming. Transportation is also extremely restricted during the four months of rainy season. Schools For the purpose of this evaluation, the Evaluation Team visited 15 schools: nine in Kurmuk and six in Kauda. Average enrollment of these schools was 514, with 211 enrolled in the smallest school and 1,200 in the largest. Within the HEAR Project area, fewer than half the schools reach grade eight. However, those that did 5 This figure is based on a study completed by the Regional Education Director in Kauda. Health, Education And Reconciliation (HEAR) Project 12 Mid-Term Evaluation not were adding grades each year. Three of the schools in the sample, all in Kurmuk, were Arabic pattern and supported by the northern government, meaning that teachers earned salaries. Two schools in Kurmuk were privately supported by an NGO, and the other four were supported by the community through volunteer teachers (primarily in Kurmuk) or the collection of school fees to compensate the staff (most commonly in the Kauda area); all of these are English pattern schools. School fees tended to be 35 SDG6 per year per child. Teachers in these schools were paid 150 SDG per month, but because of the variability in family income and issues with the collection and management of school fees (which was most commonly performed by the PTA), teacher compensation was inconsistent in both its frequency and amount. An informal quantitative survey completed by the Evaluation Team at each school they visited showed an average pupil-to-teacher ratio of 80:1 in the lower grades and 15:1 in the upper grades. Most classes did not have a teacher for the whole day, and many classes—especially in the lowest grades—sat without a teacher for nearly half the school day. The education and training levels of teachers were not formally documented at most schools visited, but in Kauda roughly 80% of the teachers had completed primary school, half of which had also completed some secondary school. Fewer than half of the teachers in Kurmuk completed primary eight. English proficiency among teachers in English pattern schools was generally poor, and in at least three￾quarters of these schools, Arabic or mother-tongue translation was necessary for training and for the focus group discussions with the Evaluation Team. Of the 15 schools visited by the Evaluation Team, 13 had reliable enrollment records, although these records were not up-to-date in all cases. Of the 6,682 students enrolled in these 13 schools, 45% are girls. All the schools visited are coeducational. More than 80 percent of these students are enrolled in grade four or lower. Although historical enrollment data was not available, interviews with teachers and parents suggest that families are enrolling their children in larger numbers due to improved security, health, and economic conditions since 2005 and the signing of the CPA. Seven of the schools visited had at least one permanent classroom, yet most relied on semi-permanent structures made of mud, stone, or grass. Most schools had at least two classes, most often in the lower grades, which met under trees. Even when semi-permanent structures were being used, many had no walls with available furniture that was very heavy and difficult to move. With few exceptions, all schools were in need of classroom construction due to either an insufficient amount of classrooms and/or the constant need to repair temporary structures. All of the schools visited have a feeding program facilitated by the United Nations World Food Programme (WFP), and improvements in the health of children were reported by parents and teachers. Only three of the schools had a nearby source of water, however, and access to water was most often cited as the biggest need of the schools. Fewer than half the schools had a kitchen, store, or their own cooking and eating utensils; this was also commonly cited by teachers and parents as a priority need of the school. It should be noted that, even in those schools that did not have a kitchen and store, school feeding was provided by WFP on an emergency basis. If conditions improve and the emergency status is lifted by WFP, school feeding will be provided only to schools with a kitchen and store. The construction of school kitchens is a common activity of the PTAs supported by HEAR Project small grants. Four of the fifteen schools visited by the Evaluation Team had a completed school fence. Both parents and teachers cited this as a high priority for the school in order to keep animals off the school grounds and deter damage to and theft of school property. Fence construction was also a common activity of PTAs, supported by HEAR Project small grants. None of the schools visited have electricity, phones, or first aid kits. HEAR has procured first aids kits and began to distribute them shortly after the completion of the evaluation. 6 The exchange rate at the time of the study was about 1 USD = 2.6 SDG. Health, Education And Reconciliation (HEAR) Project 13 Mid-Term Evaluation Seven of the fifteen schools visited have play areas and six have play equipment such as footballs, volleyballs, goal posts, volleyball nets, and netball posts. The Kauda site has facilitated the development of recreation areas in one of its more remote schools, and the surrounding communities have organized multi-school and cross-community games. PACT, an NGO in the Kauda area, is also engaged in school-based recreation development, and discussions regarding partnership between PACT and HEAR have recently begun. B. Project Design: -1- How well does the HEAR Project correspond to the government education and health priorities in the Three Areas? The government priorities for health and education in relation to the HEAR Project need to be considered distinctly. In general, it is safe to assume that health priorities where the HEAR Project is working are consistent with the priorities of the GNU; that is, there is nothing unique about the locations where the HEAR Project is working that would differentiate it from other marginal areas in Sudan with respect to health. However, the education system faces a different situation—at least in the English pattern schools, which include all the HEAR schools in Kauda and half the HEAR schools in Kurmuk. While the CPA theoretically allows communities to determine the language of instruction, English pattern schools would not be considered a priority of the GNU. In short, one can look at the HEAR Project health activities in the light of the GNU policy on health, as well of the priorities of the Secretariat of Health (SOH) in Kauda and Kurmuk. For education, however, one would look for priorities only at the Regional Education Offices (REO). The “National Health Policy – 2007,” Federal Ministry of Health, Government of National Unity, states the following: “The mission of the National Health Policy is to ensure the provision of health care to all citizens of Sudan, with emphasis on the health needs of the poor and the underserved, disadvantaged and vulnerable in order that they are able to lead socially and economically productive lives ...The content of the primary health care package includes as a minimum: ...the promotion of school health; … the protection and promotion of environmental health and sanitation; and treatment of simple diseases and injuries and mental health. “Child health will be addressed through … the provision of vitamin A supplements, … and the prevention and treatment of potentially fatal childhood diseases, such as diarrhea, malaria and pneumonia.” In addition, in Kauda, the SOH stated that the HEAR Project supports priorities in the areas of disease prevention (especially water-borne illnesses), drug distribution, health training, computer training, and hygiene. In Kurmuk, the County Health Director said that one priority for his office is health education for schools, which the HEAR Project supports. Regarding education, the Regional Education Director in Kauda stated that the HEAR Project supports the priorities of his office through hygiene/sanitation in the schools, PTA training, computer training and access to the Internet, staff training at the ERC, and the printing of “register” booklet for the schools. He suggested that the HEAR Project should cover more schools and conduct training of teachers in English and mathematics. Health, Education And Reconciliation (HEAR) Project 14 Mid-Term Evaluation The Regional Director in Kurmuk said that the HEAR Project supports the following priorities: expansion of education opportunities to rural areas, teacher training, and hygiene. The HEAR Project does not support the priorities identified by the Regional Director of transportation, uniforms, sports, and Arabic textbooks. C. Project Activities7 -5- Provide a brief description of the program outcomes, deliverables, and products. Assess the quality of the deliverables to date. Identify particular strategies, activities, or programs that are effective and describe why they have worked -7- Is the activity effectively addressing gender issues such as inclusiveness, training and promotion, and the gender gap in Education Departments in the Three Areas? If not, why not? What are the areas for improvement? Teacher Training Training for teachers is organized into three sessions. The first is a five-day initial training course that covers an introduction to the HEAR Project and pupil-centered instruction (PCI) on the first day, the health￾promoting school and the Focusing Resources of Effective School Health (FRESH) model of skills-based health education on the second, third, and fourth days, and the Language Experience Approach (LEA) on the fifth. This last day involves the use of posters to teach reading and writing. The five-day initial training manual was revised in November 2007 to reflect the removal of the Interactive Radio Instruction (IRI) component.8 Refresher course #1 is a two-day course which on the first day covers teaching about vitamin A and hygiene. The second day of training covers peace-building and service learning9 and addresses how to apply them in the context of a school open day (see further description of school open days below). Refresher course #2 is also two-days in length and covers planning for skills-based instruction on trachoma and HIV/AIDS on the first day, and developing teaching and learning aids using local resources on the second. This training section includes instructions for making paper-mâché, glue, and colored ink from local materials. A Teacher’s Health Manual is also distributed as a reference book to trainees that treats in greater detail topics such as hand washing, nutrition, vitamin A, immunization, malaria, worms, schistosomiasis, trachoma, lice, landmines, first aid, and HIV/AIDS. All the teacher training materials, including the Health Manual, are in English, and training is conducted in English and Arabic at both sites. The translation needs of the teachers result in nearly doubling the time required for training. Education Ministry officials in both regions are invited to participate in teacher training, and, according to the officials and the HEAR Project staff, most of the time they do participate. In Kurmuk, the evaluation team observed a third refresher course compiled by the HEAR Project field staff for teachers in cluster one schools who had either previously completed the initial five-day training and two 7 This section addresses Questions 5-14 in the SOW. The questions have been organized into categories to enhance the understanding of what was learned about the project. The categories include teacher training, CHP training, PTA/head teacher training, Model Schools proposal, ERCs, distribution of vitamin A and bednets, M&E, partners, management, and sustainability. Each section begins with a restatement of the evaluation questions, as appropriate. These sections are followed by sections for Conclusions and Recommendations (also broken out by category). 8 The evaluators note here that the activity on page 60 of the revised five-day initial training manual contains references to the IRI component, and should be removed or edited in subsequent printings. 9 Service Learning is a teaching strategy that combines classroom-based learning with community service activities. Health, Education And Reconciliation (HEAR) Project 15 Mid-Term Evaluation refresher courses two years earlier, or were new staff in these schools without previous HEAR Project training. In some ways, the initial two official refresher courses are but an extension of the training with new content. The third refresher did not present new topics, but reviewed those from the previous training regimen, including an introduction to the HEAR Project, a brief review of the characteristics of a HEAR teacher10, the FRESH model, skills-based health education11, and developing teaching and learning aids using local resources. A close examination of the revised training materials by the Evaluation Team reveals an important finding that has deep implications for the Project: teacher training materials are heavily weighted toward the learning and dissemination of health and hygiene messages at the expense of education and reconciliation subjects. Of the over 50 hours of training in the combined three courses, approximately fifteen hours are spent on pedagogy or classroom management issues. More than half these 15 hours are spent modeling pedagogy using the subject of health. Eight and half hours are set aside for reconciliation topics such as peace building and community involvement leaving approximately 27 hours for health. HEAR has recognized this imbalance and has had plans to engage a consultant to develop new materials to strengthen the education component. Group discussions and classroom observations in 15 HEAR Project schools conducted by the Evaluation Team suggest that teachers have come away from this regimen with a good and detailed knowledge of health￾and hygiene-related topics. When asked what was the most useful parts of the training, nearly all responded first by pointing out the health knowledge, particularly information about hand washing, personal hygiene, and proper use of latrines. Citing evidence such as increased attendance and fewer trips to the health clinic, teachers claimed that students were healthier as a result of their dissemination of these topics in the classroom and at school assemblies, When asked, most teachers expressed some knowledge about non-corporeal discipline and problem-solving techniques, but the observations of actual practice were mixed. Even in schools where teacher codes of conduct were posted, many teachers carried canes and sticks. Outside of school open days, there were no reports of the use of service learning. Of the methodological subjects covered in the training, lesson planning and schemes of work were most often cited as valued, although, ironically, these particular terms were not used in the training manuals. In general, teachers with a good command of English expressed a better understanding of the methodological subjects covered in the training such as Pupil Centered Instruction (PCI), but in no school that the team visited were these skills or tools observed in use including lesson plans.12 Very few teachers referenced LEA and few of the posters distributed for this purpose were seen in the schools visited; when asked, teachers complained that posters were either damaged or stolen not long after their distribution. Furthermore, the vast majority of classrooms had no walls where posters or students work could be displayed. A common complaint among school staff and parents is the lack of secure storage for teaching aids and books. Other key findings included: • Extremely high teacher-student ratios in the lower grades, as high as 1:100 • Low education or language skills among teachers in the upper grades • No school provided workbooks/ textbooks for students 10 The four characteristics of a HEAR teacher are: uses pupil-centered teaching methods, promotes good health at school and the community, facilitates peace-building and conflict resolution, and facilitates community involvement. 11 This includes the 3P approach to lesson planning: Presentation, Practice, and Performance. 12 It should be noted here again that this observation is based on only a two to three hour visit to 15 of the 64 HEAR Project schools in Kurmuk and Kauda. Health, Education And Reconciliation (HEAR) Project 16 Mid-Term Evaluation • The most common of all observed teaching techniques was the teacher copying text from a student textbook onto the blackboard while the students copied the material into their notebooks. • Teacher-student interaction was typically limited to Question and Answer (Q&A), choral response, and skill and drill—almost exclusively teacher-centered practices. • A lack of basic materials and teaching aids such as chalkboards, chalk, paper, writing utensils, markers, crayons, glue, tape, scissors, measuring sticks, rulers, posters, charts, musical instruments, paint, paintbrushes, clocks, and maps. • The semi-permanence and small size of most classroom structures and lack of movable chairs and desks inside them (or, in the case of trees, under them) made techniques such as displaying student work, moving furniture around to vary classroom arrangement, and small group work very challenging. • A lack of planning time during the school day for teachers to make teaching aids and prepare for PCI, with evening preparation not possible given that few of the communities served by these schools have electricity. • Teaching methods and procedures are modeled during the training exclusively using health-related topics, save for the one-day training on LEA. It should be repeated here that in the Kauda region, teachers are generally better trained given that there are two teacher training institutes nearby. Also, many attended some secondary school and there is a project that brings trained teachers from Kenya. Despite this, there was evidence that neither site, Kauda or Kurmuk, had the necessary conditions under which PCI, LEA, and the development of teaching aids could be put to effective use. Furthermore, HEAR Project staff in Kauda expressed concerns that a high degree of mobility and turnover among teachers in the region left schools in the first three clusters, where teachers were trained in 2007 and 2008, with a diminishing number of trained staff. This was confirmed by the Evaluation Team in subsequent school visits. The degree of knowledge about health, hygiene, and disease prevention seemed less than in schools where training was more recent, but this was often mitigated by the presence of a head teacher trained by the HEAR Project. However, because teachers in Kauda already had better training in teaching methodology to begin with, the degree of knowledge about pedagogy remained constant. Unlike in Kurmuk, however, no refresher courses for teachers have been offered in Kauda. Community Health Promoters Training CHPs are volunteers from the various communities served by HEAR Project schools. Their role is to disseminate health, hygiene, and disease prevention messages to school children and their families, as well as to the community at large. Training for CHPs lasts five days and covers volunteering, effective communication, and the cause, prevention and treatment of common ailments and diseases. Subjects addressed include: diarrhea, diarrhea prevention, oral rehydration treatment, intestinal worms/schistosomiasis, trachoma, hygiene and sanitation, nutrition, vitamin A, malaria, bednets, immunization, sexually transmitted diseases, HIV/AIDS, and first aid. Although there is currently no printed training manual for either trainers or trainees as there are for teachers and PTAs, one is in process of being developed. The team was not able to observe a formal training, but did sit in on a refresher course for previously trained CHPs in Kurmuk, where they also had a chance to engage in a group discussion. The CHP refresher training reviewed topics from the formal training, including roles and responsibilities and the prevention and treatment of common diseases and ailments. Most of the handouts were in English only, though the roles and responsibilities handout was available in both English and Arabic. A pre- and post-training test for knowledge was also distributed and administered orally in Arabic for non-English speakers. Health, Education And Reconciliation (HEAR) Project 17 Mid-Term Evaluation CHPs in Kurmuk expressed pride in their knowledge about health, hygiene and disease prevention and were proud of their ability to mobilize the community to support projects such as latrine digging at the local school as well as nearby residential areas. They also spoke about their promotion of personal hygiene, including hand washing, bathing, the proper preparation and storage of food, and maintenance of a clean and swept household. Based on discussions with CHPs and school staff, it was evident that CHPs also regularly visited schools between a few times a week to once a month. HEAR Project staff require CHPs to report on their activities, but these records are incomplete (as many do not submit them) and thus are insufficient in providing a useful description of the type and frequency of their activities. Due to these factors the Evaluation Team was unable to independently confirm these assertions. CHPs expressed some frustration with their status as volunteers, many admitting that they reduce the frequency of their activities over time, citing their family’s economic needs. Access to water was also consistently brought up as a challenge. It is very difficult, many asserted, to teach about proper hygiene and sanitation without available water. This was further confirmed by the Evaluation Team's observation of empty hand-washing stations at most school sites where the stations were present. A few CHPs shared that community members sometimes avoid them out of shame because the lack of water does not permit them to follow through on what they were taught. Although there appears to be good knowledge among teachers and parents about the proper use of latrines, recent staff reports also describe that their actual and/or regular use by students is in question. Most CHPs also complained about the lack of enough bednets for each family for the prevention of malaria. On the whole, however, most CHPs demonstrated a good understanding of the concepts and skills disseminated in the training, which was confirmed in group discussions with teachers and PTA members. The Evaluation Team was not able to independently confirm these assertions as CHP activities were not directly observed and, as mentioned above, sufficient records of their activities are not collected. HEAR Project staff shared an improved draft of a CHP activities log with the team to address this latter point, but it is not currently implemented. In Kauda, the Evaluation Team found that, although CHPs were trained well, those in the first three clusters of schools were in need of refresher courses. Perhaps more importantly, mobility and turnover has left many schools without active CHPs. In the Kurmuk area, a new Health Sciences Academy is being developed with the support of the Blue Nile State Governor which will be led by the county Director of Health Services. Certification programs for medical assistants, nurses, laboratory technicians, pharmacists, and midwives are or will be offered. Both of these individuals are highly supportive of English as a language of instruction, and the Academy plans to accept students from southern Sudan. The Director of Health Services, in an interview with the Evaluation Team, expressed support for community- and school-based health education of the kind in which the HEAR Project is engaged. He acknowledged that considerable potential for partnership existed in the areas of both community health worker training provided by the HEAR Project, and continuing education for CHPs provided by the Health Sciences Academy. Good communication between the HEAR Project staff and his staff has not yet been established, although this is in part due to the fact that the Director was posted to Kurmuk only in the last few months; this problem was actually resolved during the team’s interview with him, as a HEAR Project health staff person was present, and the Director stated he would be appointing a person to liaise between his office and the project staff. Distribution of Vitamin A and Bednets The HEAR Project has life-of-project targets to distribute 21,763 vitamin A pills and 10,000 bednets. There are no funds within HEAR Project budget to purchase these items, so CAII must negotiate with other organizations to obtain them. Health, Education And Reconciliation (HEAR) Project 18 Mid-Term Evaluation Within the Three Areas, the HEAR Project has distributed 17,215 vitamin A pills (Kurmuk 9,435; Kauda 5,972; and Abyei 1,808). In Kurmuk, the pills are obtained from GOA and in Kauda from WHO, Save the Children, and the Secretariat of Health. To date, there has only been a single round of distribution. The HEAR Project still has to obtain additional pills to meet its target and, if the project should obtain even more pills, they might be able to issue another round of distribution. Ideally, children should receive the pill every six months. Bednets have only been distributed in Abyei (a total of 2,640 nets). UNICEF provided the first 1,000 mosquito nets distributed and the Ministry of Health provided the subsequent 1,640 through the Abyei Hospital and Health Director, Dr. Akram Gabriel. UNICEF provided an initial 1,500 nets to the HEAR Project, but these were destroyed during the fighting in Abyei in May 2008. The HEAR Project has requested 5,000 bednets from UNICEF in Kadugli, but after a long wait, there has as yet been no response. In addition to the required distribution of vitamin A and bednets, the HEAR Project has undertaken the distribution of 12,866 deworming pills: 6,992 for Kurmuk and 5,874 for Kauda. These pills were given by the same organizations that provided the vitamin A pills. PTA/Head Teacher Training Training for Parent Teacher Association (PTA) members and head teachers have been carried out in two clusters in Kurmuk and four clusters in Kauda. This training lasts three days and is supposed to be centered on the planning, creation, and refinement of a school development plan. While days one and two of the training are designed for PTA members, head teachers, and school administrators together, the third day is for head teachers and school administrators only. The first day covers the planning process, including brainstorming on the characteristics of a good school, conducting a local self-assessment, and identifying the challenges the school faces. The second day covers the identification of local stakeholders and their potential roles and the development of the particular activities the PTA, head teacher, and other stakeholders will undertake to address school challenges. The third day engages head teachers and school administrators to assess the progress of the creation of a school development plan and also covers budgeting and monitoring skills. However, neither HEAR Project field offices in Kurmuk or Kauda or the head office in Rumbek had any School Development Plans on file. The Evaluation Team did not see any evidence that any were completed. It was evident, however, that a number of schools benefitted from this process, as it produced successful applications for small grants, though this is not a part of the PTA/Head Teacher training stated purpose. In addition to this training, school record-keeping books developed by HEAR have been received by the Project and will be distributed to head teachers along with training in their use. Head teachers will be enlisted subsequently to distribute these record books to their teachers and train them in their use. The Evaluation Team found that PTAs existed in all schools visited, although the number and intensity of their activities tended to correspond to how recently they were trained. This was particularly evident in Kauda, where the PTAs in clusters one, two, and three received training in 2008, but now had few trained members remaining. A number of activities were common amongst PTAs including classroom, fence, latrine, kitchen/store, and teacher compound construction, the preparation of school lunches, the procurement of water and firewood, and the collection of school fees. In some locations, these activities were support by HEAR small grants. HEAR Project staff reported that PTA members in most sites were dominated by men both in their make up as well as in their decision-making process; this was confirmed in the Evaluation Team’s observations. The team was able to interview more PTA members in the Kurmuk site because they were more recently trained. In Kauda, the Project was expanding to the fifth cluster, and many PTA members had been trained in 2008, Health, Education And Reconciliation (HEAR) Project 19 Mid-Term Evaluation resulting in more turnover in members. It also appeared that PTA activities had decreased as the date of the training receded in time. In Kurmuk, a HEAR Project staff member who is well known and liked in many of the communities where HEAR Project schools are located mobilized PTA and other community members in advance of the team’s arrival. It was further evident that this staff member’s intimate knowledge of the communities served by the HEAR Project, including his fluency in multiple languages and mother-tongues, proved an indispensable resource for community mobilization and for the project activities in general. The Evaluation Team also found that record keeping varied from school to school depending on the skills and commitment of the head teachers. The team’s observation’s gave little indication, however, that head teachers had the tools and skills to effectively and/or consistently perform basic administrative functions such as tracking enrollment and daily attendance, documenting the training and educational backgrounds of faculty, recording student performance and achievement data, effectively managing the PTA, developing broader community relations, and creating and engaging in ongoing revision of school development plans. Where records existed, they were often outdated or incomplete. In interviews with HEAR Project staff in both locations, the issue of sustainability was acknowledged. There are activities in both sites to provide more training and resources to head teachers, including the use of the teacher record books for enrollment, attendance, and student academic performance. In many cases, the Evaluation Team found that head teachers possessed natural leadership abilities and communication skills. Small Grants Though the original focus of the PTA/Head Teacher training was the drafting of a School Development Plan, the preparation of small grants proposals has become a more important component of the HEAR training. The grants provide tangible benefits to schools, such as kitchens, water tanks, latrines, classrooms, and fences and, for a relatively small amount of money, can make a significant contribution to a school. The results also offer visible evidence to the community—teachers, parents and children—of the contributions of the HEAR Project. According to HEAR Project staff, the School Development Plans are not completed because of the limited capacity of the head teachers and PTA members, their varying reading abilities in Arabic and English, and the limited time available for training. However, some of the training objectives are met by drafting the small grants proposals. HEAR Project staff confirmed that, in many cases, this ended up being a more reasonable expectation of the PTA/Head Teacher training and more directly addresses the immediate and short-term needs of schools. To date, the HEAR Project has initiated 26 small grant projects in 25 schools.13 Of the 26 projects, all but three have been completed. In addition, there are five projects under review and a substantial number that have yet to be considered. See Appendix B for a table outlining small grants activities to date. For the life of the HEAR Project, 40 small grants are budgeted at up to $2000 each; thus, there are fewer than nine or ten grants remaining beyond those that have been carried out or are currently under consideration. This number may actually be lower, as funds for school open days and other projects such as playground construction are drawn from the small grants fund. Thus, this number does not allow for a small grant at each school, as the current total number of schools in all Three Areas is 96. There will be over 90 HEAR schools in Southern Kordofan and Blue Nile alone. 13 Abyei=1, Kurmuk= 14, Kauda=10 Health, Education And Reconciliation (HEAR) Project 20 Mid-Term Evaluation The process of grant administration has evolved as the project has gained experience. In almost all cases, the process begins as part of the head teacher/PTA training; as outlined above, it is not the stated purpose of the training but has evolved to be an integral part. Often, subsequent to the training, the head teachers complete the proposal. This process is relatively sophisticated for the participants. Each proposal is then reviewed by a School Grants Committee (SGC), a body that was created following the experience with the first grants. The Committee members typically include the REO Director, SOH Director, HEAR Project team leader, Small Grants Assistant, Health Coordinator, and Payam Education Supervisors. The expansion or formalization of the SGC resulted from challenges in implementing the first small grants process. Project staff thought that the process could be improved by a pre-school visit, further consultation with stakeholders, and prioritization of school problems—activities included in the PTA training. Regarding implementation, one community in Kauda threatened to sue the project for the remaining balance of funds when the school, did not properly carry out procurement procedures. Eventually the matter was settled with the help for the County Regional Education Office, but the experience demonstrated the need for the project to ensure that the head teachers and PTA members fully understand the terms of the contract, as well to ensure participation among multiple stakeholders and better communication with government officials. The project is now strengthening this process and will even consider translating the contract into Arabic for the benefit of the PTA members. School open days The intended outcome of the teacher refresher course #1 is the planning for a school open day. School open days are days on which parents and community members are invited to visit the school and observe presentations, usually dramas or role-plays, by students and teachers. The topics of these presentations, according to reports by HEAR Project staff, usually involve nutrition and hygiene, peace and reconciliation, and girls education. Often involving two nearby HEAR Project schools, football matches and speeches by community leaders are a common part of the activities. HEAR Project staff report that over 200 community members, as well as representatives from local NGOs, attend these events. Funds for school open days come out of the small grants budget. School open days are reported to be very successful events in terms of the broader dissemination of health, hygiene and disease prevention messages to the surrounding community, knowledge about the HEAR Project in the broader community, and participation of other NGOs and government officials. Despite this, only five school open days have been held. As of the date of this evaluation, the following school open days have been held: one in Abyei (3/2008), three in Kurmuk (11/2008, 11/2009, 2/2010), and one in Kauda (11/2009). Gender Equity Forty-five percent of the pupils are girls across the two states in the HEAR schools, with 21% of the teachers and 7% of the administrators being female. Changing cultural and economic conditions have improved the prospect for gender equity in the region, although some cultural and economic barriers remain and retention of girls is still a problem for many schools. Most PTA members and teachers in discussions with the Evaluation Team cite early marriage and school fees as the largest barriers to enrolling and retaining girls in school. Lack of materials to manage girl’s menstrual cycles at the school site was also mentioned as a barrier. Most schools either have or are constructing latrines which include separate facilities for girls—a factor that can contribute to the retention of female students; however, the evaluators noted that in some schools the latrines were not being used. There is much stronger female representation among the community health promoters, of which 41 percent are women.. Currently, little overt training or direct support in gender equity is provided by the Project, but there are some opportunities for such activities, which will be discussed later. Teachers trained by HEAR Sudan Project through March 2010 Health, Education And Reconciliation (HEAR) Project 21 Mid-Term Evaluation Male Female Total % Female Kurmuk 163 66 229 29% Kauda 177 26 203 13% Abyei 246 64 310 21% Grand Total 586 156 742 21% Head Teachers and School Administrators trained by HEAR Sudan Project Male Female Total % Female Kurmuk 33 4 37 11% Kauda 91 6 97 6% Abyei 86 7 93 8% Grand Total 210 17 227 7% Community health promoters Male Female Total % Female Kurmuk 78 35 113 31% Kauda 118 92 210 44% Abyei 73 62 135 46% Grand Total 269 189 458 41% Model School Proposal The HEAR Project has given considerable thought over the past year to establishing Model Schools—two in each of the Three Areas. The HEAR Year Three Work Plan states the following: The purpose of [model schools] will be to demonstrate how project objectives are achievable and how selected developments could be replicated in other schools. Model schools will be exemplars of some of the theories and practices as promoted through the HEAR Project training programs, including such aspects as water storage points, hand-washing stands, latrines for girls and boys, administration notices and displays of information, school clubs and open days. Other nearby schools will visit the model school and learn from the Head Teachers, PTA and teachers on how to organize their school like a “model school.” Education Resource Centers The ERCs provide useful services to students, teachers, officials, and other members of the community. The ERC in Kurmuk came into operation in October 2009 (the official opening was postponed to March 2010) and the ERC in Kauda was opened in June 2009. The normal hours of operation for the public are roughly 9:00-5:00 with a break period for lunch. Specifically, the ERCs have the following functions: • Office space for HEAR Project staff plus storage space, book shelves, and computers • A training venue for HEAR Project workshops, but for other institutions as well, both government and NGOs • A library Health, Education And Reconciliation (HEAR) Project 22 Mid-Term Evaluation • Computer lab and training The books are mostly on education and teaching topics, especially science, mathematics and general reading books, plus a few books on management and some curriculum books from the New Sudan and Kenyan Primary series. The collection also includes large picture books and posters. CAII was able to select books that they judged to be most appropriate from Brothers and Brothers, the organization that donated the books. The 11,000 books arrived and have been divided among the Three Areas. They have already been distributed to Kurmuk and Kauda, and the share for Abyei is waiting to be shipped from Rumbek. Two thousand of the books in Kauda have been distributed, with each school receiving at least 40 books; the distribution of additional books is planned. In Kurmuk, 484 books have been distributed to three schools with 30-40 titles and three to five copies per title to each school. However, is the lack of a a secure room or proper storage facilities has presented challenges for many schools. At the schools visited by the Evaluation Team, the teachers reported that the books have been very helpful. Although there is no documentation at the schools on the use of the books, the teachers report that they use the books and, sometimes, take the books to their classes for their pupils to use. The Evaluation Team did not observe the books being used at the schools. Based on observations and attendance sheets, it appears that, in Kauda, many pupils and teachers visit the ERC to read books, while in Kurmuk almost all the visits are related to computer training, although some do end up reading the books as well. The Evaluation Team observed a few students using the books in the ERC. In particular, the students were interested in reading story books to improve their English skills. Teachers and HEAR Project staff in Kauda report that the books most often used in the ERC are the Kenyan curriculum books, as this curriculum is used in most upper grades in Kauda. As stated above, the other text and reference books available at the ERC library are used by teachers to improve their own knowledge in core subjects. An important function of the ERCs is computer training. Both ERCs have an ambitious schedule for training teachers, officials, and pupils in basic computer skills and using the Internet. At the Kauda ERC, there are five groups of primary school pupils scheduled for five afternoons a week. Teachers are allocated one afternoon and the KTTC teachers and trainees another afternoon. Officials have two afternoons. Out of a total of 6,022 users at the two ERCs from June 2009 through January 2010, 2,542 (42%) were primary school pupils. In Kurmuk, teachers are given training two mornings a week, education and health officials for two afternoons a week, and primary school children for four afternoons a week. ERCs also have video cameras and have initiated a couple of video projects. Perhaps the key target audience for computer training is teachers. In Kauda, teachers meet on Fridays and Saturdays for a total of three hours, and in Kurmuk teachers have five and a half hours per week on Friday and Saturday mornings. The length of the course is three months. Once the teachers gain a few basic skills, they are introduced to the Internet and given guidance on how to identify useful sites for their own professional development and for improving the instructional program in their classes. The Education Technology Specialist has compiled a list of websites that may be useful for teachers. As of the date of the Evaluation Team’s visits, the teachers were beginning the Internet training. The emphasis of the training for education and health officials is slightly different. In addition to basic skills and the Internet, the officials are being introduced to MS Excel early in their training with the idea that their offices will be establishing databases. For example, it is a high priority of the Regional Education Office in Kauda to conduct a survey of all their schools. This work may be facilitated by the USAID BRIDGE Project, and the HEAR Project is committed to help provide some training in anticipation of a new Education Management Information System (EMIS) in the REO. The education and health offices value the training, but also would like a couple of computers to be placed in their offices and have some training in their offices as well. Health, Education And Reconciliation (HEAR) Project 23 Mid-Term Evaluation The HEAR Project is also making a commitment to computer training for primary school pupils in both locations and the students at the Kauda TTC. The commitment to primary school pupils is substantial. When the evaluation team arrived in Kauda, the pupils were booked five afternoons a week and there was a total enrollment of 288 7th and 8th grade pupils from two schools; only 37 primary school pupils are given computer training in Kurmuk. Finally, the ERCs also cater to individuals in the community who have computer skills and who need to access the Internet. The general format of the training consisted of PowerPoint presentations offering basic information about computers; only later did the learners get an opportunity to actually use the computers. This approach seemed dictated by the teaching style of the Education Technologies Specialist, the limited number of computers in each center (about seven), and the large class sizes (often 50 pupils). Following discussions with the Evaluation Team, the Education Technology Specialist began implementing the following changes in Kauda: • Reducing the time given to more theoretical information about computers • Increasing the time for hands-on practice, and dividing the large classes into smaller groups so that each learner has a chance during every class to practice new skills. This often involves placing three learners at each computer. • The issue of reducing the total number of pupils is still under discussion The IT staff have training materials for their own use. The module on the Internet is excellent, and includes detailed instructions on how to use the Internet. The materials on basic computer information are overly detailed, with too much theory and information to be useful to learners. The materials for MS Word and MS Excel are good for reference (in terms of information on the many commands in each program), but are not appropriate as teaching tools. Both ERCs have camcorders that have been used for initial video productions. In Kurmuk, a video was shot on the “girl child” in school. Video clips included the child at home, including an interview with parent, and the child in school. The video clips were sent to CAII in Washington, DC for editing. In Kauda, a teacher training session was videotaped with the expectation that some segments of the video could be used to strengthen future training programs. D. Partners -11- What relationships has the HEAR Project developed with other implementing partners in the Three Areas (both USAID and others)? Has the HEAR Project effectively leveraged these relationships, and what has been the impact on the project? Most importantly, the HEAR Project has established excellent working relationships with the Regional Education Offices and the Secretariats of Health in both Kauda and Kurmuk. The HEAR Project has also established relations with other partners who support the project in a variety of ways. Of critical importance are relations with WHO, UNICEF and GOAL, who have provided vitamin A, deworming pills, and bednets for distribution, as the HEAR Project does not have its own funds to procure these items. UNICEF based in southern Sudan has provided the bednets for Abyei, but the project is still waiting to hear from UNICEF in Kadugli regarding bednets for Southern Kordofan. GOAL and HEAR both distribute vitamin A and deworming pills and coordinate their distribution plans to ensure that there is no overlap of schools. The project also collaborates with the Internews community radio stations in Kauda and Kurmuk. These stations broadcast announcements about the HEAR Project meetings and report on special Project events Health, Education And Reconciliation (HEAR) Project 24 Mid-Term Evaluation and celebrations in schools. In Kurmuk, the station rebroadcasts the HEAR Project trainings and special events, such as school open days and trachoma training, in different languages. In Kauda, the station broadcasted a radio play supported by the HEAR Project with pupils of the Vincent Primary school regarding the importance of health and hygiene. Reports indicate that the broadcast reached a wide audience and a number of listeners called the radio station to comment on the issues discussed. The radio stations rely on the HEAR Project for Internet support and, on occasion, for assistance with transportation. The Kurmuk station continues to broadcast Interactive Radio Instruction (IRI) programs for children and English programs for youth and adults, as requested by the HEAR Project when IRI was included under the project. The HEAR Project collaborates with NCA at the KTTC in Kauda and with ROOF in Kurmuk at the schools they support where the HEAR Project is working. The HEAR Project also collaborates with PACT in Kauda, which supports dissemination of information on the CPA and has water resource projects. The staff from PACT and HEAR sometimes attend one another’s trainings. The HEAR Project has collaborated extensively with the SSIRI Project and has initiated discussions with other USAID-supported partners, including the GEE and BRIDGE. Project staff have also initiated discussions about possible collaboration with the new the Health Sciences Academy in Kurmuk. The HEAR Project is also a member of the South Sudan Health NGO forum. E. Management -8- Does the program have environmental compliance issues? If so, how well has it performed against USG guidelines and regulations? -9- Assess the quality and performance of CAII in managing the implementation of the HEAR Project. What are the team’s strengths, weaknesses, and areas for improvement with respect to contract management and communications with USAID, government of the Three Areas and stakeholders? The Evaluation Team held a two-hour discussion with the HEAR Project staff in both Kurmuk and Kauda on project management issues. The staff said that, in general, the project is well-managed. The Rumbek and Nairobi offices respond to field requests and support the field offices as needed regarding the implementation of HEAR Project activities. The same applies to the CAII office in Washington, DC, when needed. Perhaps one exception to this overall pattern was a delay in shipping vehicles and computers from Rumbek to the field. However, the field staff understand the unavoidable difficulties and delays that are caused by the limited infrastructure, such as the lack of banking facilities in either location and the single flight operating per week to Kauda and Kurmuk. Transportation is also limited to a single vehicle per site (in addition to a few motorbikes), and both areas are virtually shut down for several months a year during the rainy season. The observations of the Evaluation Team confirmed that the training manuals have been developed, revised, and printed, the vehicles and computers are present, and 11,000 new library books are available in southern Sudan. The staff members also expressed some concerns, primary among them being the high turnover of COPs. In the three years and four months of the project to date, the HEAR Project has had three COPs and two Interim COPs; at times, the Project Director in Nairobi has served as an Acting COP. In part, this turnover reflects the difficult working environment of the HEAR Project. On the positive side, the turnover of the COPs has been partially offset by reasonable stability in the field positions: Team Leaders (working two and three years, respectively, with the project), M&E Assistants (two and three years, respectively, with the project), and Health Coordinator (three years). Due, in part, to this turnover of COPs, visits to the field sites did not occur as much as the staff would have liked. Health, Education And Reconciliation (HEAR) Project 25 Mid-Term Evaluation The field staff reported that the turnover has been troublesome because it entails changes in leadership styles, variations in priorities, occasional changes in policies, and at times, a vacuum in leadership, in addition to inconsistencies or changes regarding some new possible initiatives (e.g., maternal and child health activities, EMIS, and model schools). Fortunately, a new COP will report to southern Sudan in April. Given the difficulties of traveling from one state to another, the location of the project office has been problematic. The project office was located in Abyei until a military incident occurred there in May 2008. The staff report, however, that the subsequent move to Rumbek seems to be working out well, as transport to each of the states is possible, taking into account the infrequent flights to each field location. Regarding contract management and communications with USAID, the CTO had the following comments: • CAII is proactive and meets the expectations of USAID. • Besides the annual and quarterly reports, CAII/HEAR provides monthly, bi-weekly and weekly bullet points that keep USAID aware of the field activities • The PMP is up-to-date • CAII/HEAR has exhibited proficient contract management skills: even without the COP on ground for the last six months, thanks to the efforts of the committed CAII/HEAR staff on ground, the activities have been implemented according to schedule. The senior advisor based in Nairobi was able to divide his time between Nairobi and Rumbek to ensure that activities progress on schedule. • The general administration and financial management/reporting has been good. The staff had several comments and suggestions, largely related to HR issues: • The staff reported several problems with HR, believing in part that this is caused by having the individual responsible for HR in the Rumbek office also be responsible for all administrative, logistical, and financial matters. They recommend that a full-time HR officer be hired. • There is some lack of clarity on role and responsibilities of the Senior Health Advisor and the supervisory relationship with the field Health Coordinators. • The staff feels that the Rumbek and/or Nairobi office could provide greater support for identifying sources of vitamin A, deworming pills, and bednets, as well as in establishing partnerships for drilling new boreholes. In addition, the staff mentioned a few internal HR issues that they can convey to CAII. Both field sites seem to have excellent relations and communication with their respective education and health offices, with the exception of the health office in Kauda where the Regional Health Director is relatively new and has had other priorities during his first few months. Nevertheless, the Director expressed support for the project and will appoint a person to liaise with the HEAR Project. F. Project Achievements and Impacts -2- Is the HEAR Project model having success in improving the skills of teachers, community health promoters and PTAs? -3- Is the monitoring and reporting system providing the HEAR Project with adequate data to evaluate its impact? -6- Is the HEAR Project progress to date in meeting the deliverables of the Contract Agreement (as amended) and implementation targets (as specified in project documents) on track for the project’s current stage of implementation? Describe any areas of concern or of accelerated success in implementation. Health, Education And Reconciliation (HEAR) Project 26 Mid-Term Evaluation -13- Is this activity rendering a clear CPA “peace dividend?” To what extent has the HEAR Project responded to the education and health needs of children in the Three Areas? What has been the impact of program interventions to date? -14- Is the project on track to be effective in improving the capacity of education departments, teachers/ administrators, community health promoters and PTAs to provide high quality education in the Three Areas? Is the HEAR Project progress to date in meeting the deliverables (question 6) The HEAR Project has done a good job in collecting and analyzing data that relate to the eleven indicators required as contractual deliverables. As can be seen in the table on the next page and in Appendix E, the Project is on course to meet the end-of-project targets. The table shows data through December 2009 on (a) the status of deliverables at the end of the first quarter, Year 4 (b) the end of project targets, and (c) the percentage progress for each indicator. The only activity that appears to present a major challenge is the distribution of the bednets. The Project does not have funds to procure the nets and must rely on donations; although this is also the case for acquiring vitamin A, obtaining bednets is more difficult. Health, Education And Reconciliation (HEAR) Project 27 Mid-Term Evaluation The following table shows the current status of the project in terms of meeting the life-of-project targets: SUMMARY INDICATORS - TOTALS TO DATE AND END-OF-PROJECT TARGETS (AS OF DECEMBER 31, 2009) S/N Indicator Achieved to Date End of Project Targets % Achieved M F T M F T 1 Learners enrolled 20,007 14,294 34,301 32,283 17,717 50,000 69% 2 Teachers trained 506 145 651 714 266 980 66% 3 Administrators/Officials trained 210 17 227 355 65 420 54% 4 PTAs support 87 140 62% 5 Community development projects completed 20 40 50% 6 Schools supported 99 140 71% 7 Students receiving vitamin A 17,215 21,763 79% 8 Bednets distributed 2,640 10,000 26% 9 Community health workers trained 216 155 371 302 258 560 66% 10 Schools with boreholes & latrines 27 45 60% 11 Communities strengthening school and health services 322 460 70% Health, Education And Reconciliation (HEAR) Project 28 Mid-Term Evaluation The evaluators have the following comments on data storage. At the project office in Rumbek, the data on attendance at workshops are placed in folders organized by date. However, the records for the early part of the project, which were stored at the HEAR Project office in Abyei, were destroyed during the fighting in May 2008. As a result, to give one example, there are no data for the Cluster One five-day training in Kurmuk. However, the relevant numbers are available from quarterly reports submitted before the original documents were destroyed,. The data processing system presents challenges to accuracy because, as only aggregated data (rather than raw data) are entered into the computer database. That is, enrollment totals or the number of participants in a workshop are added up manually and entered into the system. This practice was also noted as an area for improvement in the Data Systems Analysis performed by MSI in March 2009. Another potential problem is that there are not complete copies of all the reports in Rumbek and the respective field sites. For example, upon asking to see some of the School Visit Monitoring Reports in Kurmuk, the Evaluation Team was told the documents were in Rumbek. Conversely, upon asking to see some other forms in Rumbek, the team was told that the reports were in the field. This finding relates to another area for improvement from Data Systems Analysis Worksheet that states, "[p]articularly in light of the unstable situation and poor infrastructure in project areas, developing processes for replication of all source data will help to ensure that original data is not lost and can be easily verified. CAII has recognized this need, but is currently unable to copy school registration forms due to lack of infrastructure. The use of portable scanners could be explored as an alternative to copying machines in each field office." There are also no copies in the Rumbek office for some field reports, such as the monthly CHP reports and the original school assessment documents. The M&E Coordinator had updated information readily at hand as to the number of teachers, CHPs, and PTA members trained by the HEAR Project at each school. The Evaluation Team did not try to verify all of the figures against the original documents, but it did note one discrepancy on the number of bednets distributed in Abyei: the project report states 1,640 were recently distributed, but the available backup data indicate 143 fewer nets were distributed than is indicated in the project report. The figures such as workshop attendance are counted manually and entered into an Excel spreadsheet. The summary sheet in Excel disaggregates the data by clusters and within each cluster, and provides data on each training and the number of vitamin A pills distributed. The data are not disaggregated by quarter, which is the period for reporting to USAID. However, since the events are relatively few and the dates are indicated, it is not difficult for the M&E Coordinator to pull out the relevant data for each quarter. Monitoring and reporting system for evaluating impact (question 3) Although CAII has done a good job tracking data required for the Performance Monitoring Plan (PMP), it has not collected and organized much other data that would be useful to assess the progress, impact, and challenges of the Project. One exception is reporting on the small grants, which is required for a PMP indictor. The HEAR Project recognizes the importance of this issue in its FY2010 Work Plan, where it states "[t]he HEAR Project, in order to go beyond the numbers, will begin to monitor the effect of its various trainings. Tools to do this include the workshop evaluation form, the pre/post assessment form, the school attendance and retention racking form, and school registers." There is also a School Visit Monitoring Report form that is used by the HEAR Project. This form provides mostly known data such as "Does your school have a latrine?" and "Do you have access to water sources?" There are only two questions that begin to get at the specific HEAR Project related activities at the school: "Have you taught pupils any of the health messages you learnt from the HEAR teacher training course?" and "Do you allow pupils to participate in classroom teacher and learning? In what ways?" It is not clear from this form to whom the questions are addressed, and the form also does not encourage collection of data through observation at the school. Health, Education And Reconciliation (HEAR) Project 29 Mid-Term Evaluation Training on the use of these forms will be increased this year to inform the EMIS database which is in development, as well as for general monitoring of the impact of various trainings. Also to be piloted this year are the health surveys that will set baselines on the prevalence of intestinal worms amongst school children, helping to suggest school- or community-based factors that could be responsible for variations in prevalence. Lessons learned from schools with low prevalence will be used to inform other schools. Some of the data not currently collected or organized are the following: • School monitoring forms for each school visit that would provide information on HEAR Project activities such as classroom observations, observed health practices, and information collected from interviews or group discussions • Detailed reports on activities of individual community health promoters (a new form is being implemented that should capture this information) • Assessment of any teacher efforts to implement specific activities taught during the HEAR Project training, such as skills-based hand washing, the Language Experience Approach to teaching a language, the use of low-cost teaching aids, and the suggestions on peace-building and reconciliation • Reports on HEAR Project related PTA activities • Detailed baseline data on schools in previous and new clusters One example of summary data collected is CHP monthly meeting reports. The November 2008 report, for example, states that the CHPs "reported that they have held activities like making sure the schools have a compost pit for rubbish disposal and they have held talks with their pupils in schools about hand washing." The reports also include the number of community meetings. Beginning in January 2010, a new form was created by the Senior Health Advisor that disaggregates data at the school level with achievements, challenges, and a way forward. This form is a significant improvement. For example, while the 29 January 2010 report (compiled from data earlier than the introduction of the new form) states that for Chali Rasha School, the CHPs "talk on trachoma, diarrhea in schools and communities," It provides no information on how many CHPs participated, the number of visits, or how these messages were imparted (e.g., in a class, at an assembly, or at a PTA meeting). The new form will provide more detailed information. Occasionally, some detailed information is collected at the school level. As one example, the Senior Health Adviser included in his trip report an assessment he conducted in Kauda (Nov. 09), Abyei (Oct. 09), and in Kurmuk (Jan. 10), during which he visited a few schools at each location. In Abyei, he reported common problems including lack of fencing, kitchens, and latrines. In Kauda, he reported on individual schools, finding good progress on the development of a school health policy but no progress on the establishing health clubs or using the skills-based approach, and perhaps most importantly, no evidence of actual use of latrines or hand washing kits. Based on the supervision reports, the Senior Health Advisor has made recommendations, including: • The HEAR Project health team needs to develop a common understanding on implementation of the FRESH model. • There should be more time devoted to skills-based health education and Behavioral Change and Communication (BCC) to address the low utilization of hygiene and sanitation facilities. • There is need for a rapid assessment to identify the factors affecting the utilization of latrines. This information is useful for the Senior Health Advisor and to anyone reading his trip report. However, it is not compiled in a way that can be entered into a centralized database on the status of HEAR Project activities at each school. As another example, the HEAR Project has provided many schools with books from their library, but has not tried to collect information on how these books are used in the schools. The gap in this type of data at the Health, Education And Reconciliation (HEAR) Project 30 Mid-Term Evaluation school level may be a consequence of the fact that such data are not required and the Project has limited staff and transportation. There is only one M&E Coordinator and one M&E Assistant in each of the two locations. Improving the skills of teachers, community health promoters, and PTAs (question 2) The training programs have improved the knowledge and skills of teachers and community health promoters with respect to health, but have been less successful in improving knowledge of good teaching practices. At the end of a refresher course for community health promoters, the Project assessed learning by administering a test to the participants. Otherwise, there are no measures to assess the new knowledge and skills of the teachers and community health promoters. However, some of learning gains could be assessed through school visits aimed at determining to what extent the teachers and community health promoters are implementing skills and activities that they were taught during the workshops. In addition, skills learned on the creating of small grants proposals can be determined by the examining the quality of the proposals themselves. One of the challenges in assessing the impact in this area is that, while the skills of trainees may have improved, the application of these skills may be impeded by external factors such as lack of water, poor school conditions, and local power relations. So, for example, the absence of observable pupil-centered instructional approaches may be a consequence of large class sizes rather than a lack of teacher skills. Is this activity rendering a clear CPA “peace dividend?” (question 13) HEAR is providing an important peace dividend. Following two decades of war, schools are springing up and enrollment is mushrooming. HEAR is supporting this process by training teachers, head teachers, PTA members, and community health promoters. Senior officials are also benefiting from participation in the training and computer-related activities at the ERCs. The Project is making a major contribution to health education in schools and the communities, is increasing awareness of pupil-centered learning in the classroom, and has distributed some education materials. Every school that the evaluators visited expressed strong support for the HEAR Project, as did the health and education offices in both states. Furthermore, the ERCs in Kurmuk and Kauda are proving to be important centers of learning with their computers, access to the Internet, and books. The details of the contribution of HEAR are covered elsewhere, but there is strong evidence of a peace dividend. Is the project on track to be effective in improving the capacity of education departments, teachers/ administrators, community health promoters and PTAs (question 14) The Project is on track to provide training to teachers, administrators, community health promoters and PTA members based on the targets in the PMP. As noted elsewhere in this report, however, there is a high turnover of these personnel at schools and a need for more follow-up, especially among schools in earlier clusters. The Project has worked with officials in the departments of education and health and has improved capacity in some areas such as computer/Internet skills. But it is less clear to what extent these officials are managing their offices more efficiently than before given the constraints under which they are working. At present, there remains some dependency on the part of ministries on the resources of the HEAR project, such as for transportation and Internet access. G. External Variables and Sustainability -4- Describe and assess the measures being undertaken to ensure the sustainability of the capacities being developed under the HEAR Project. What are the constraints to sustainability, and what can the HEAR Project do to mitigate them? Health, Education And Reconciliation (HEAR) Project 31 Mid-Term Evaluation -12- What are the main external (exogenous) variables or events that have impacted project implementation? How has the HEAR Project responded? The HEAR Project recognizes the importance of sustainability and is already taking appropriate steps towards this end. CAII has described activities in their 2010 Work Plan that will help develop a basis for sustainability, but there are several factors in the Three Areas that pose risks and constraints to the achievement of this goal. The HEAR Project is working in areas that are politically tenuous. Abyei went through a major military incident in May 2008 from which it is still recovering. Abyei is scheduled to hold a referendum in January 2011 on whether to align itself with the South or North. (This is at the same time that southern Sudan will be voting on independence.) Technically, Abyei is part of northern Sudan, although the settlement agreed to after the violence in Abyei town in May 2008 placed a southerner as head of administration and a northerner as the deputy head. Practically, at least in education, it is managed by people sympathetic to southern Sudan. In contrast to the situation in Abyei, in which citizens will have the right to vote for joining the north or south, the states of Blue Nile and Southern Kordofan are part of the north. However, there are to be “consultations” to help assess the preferences of citizens regarding the future of these areas. In both states, there are a number of people who are sympathetic to the Sudan People’s Liberation Movement (SPLM) and want an education system in English based on the New Sudan curriculum of southern Sudan or on the Kenya curriculum. The situation in Southern Kordofan is less stable than Blue Nile, as exemplified by the following news report in early 2009. KADUGLI, 13 January 2009 (IRIN) - Squabbles between parties to Sudan's North-South peace agreement, rival community interests and the slow pace of development could destabilise Southern Kordofan State, analysts warned. "Southern Kordofan is in a state of political turmoil," Sara Pantuliano, research fellow with the Humanitarian Policy Group, said. "Signs of insecurity are widespread in the western area where grievances about lack of access to services and employment and the blockage of pastoralist movement towards the South have led a number of Misseriya youth to resort to armed violence." http://www.unhcr.org/refworld/country,,IRIN,,SDN,,497050fd2c,0.html In Southern Kordofan, in contrast to Blue Nile, there is no GNU support for education in areas where HEAR is working. The REO and all staff of the 228 schools are volunteers. However, the situation may change if the planned “integration” takes place in which the GNU Ministry of Education will begin to pay the salaries for just one-third of the 1,500 teachers. Should this happen, the consequences are unknown and possibly damaging, depending on the reaction of the remaining 1,000 teachers who would not receive a salary. Eventually, the County Regional Education Office would be incorporated in the Ministry offices in Kadugli. 7. CONCLUSIONS & RECOMMENDATIONS Overall, the project seems to be progressing well over the past two years, and much has been accomplished. The project is on target with the formal deliverables. It has established new ERCs with a library and computers (with training) in Kurmuk and Kauda; developed and revised training materials; trained teachers, head teachers, PTA members, and CHPs; administered 26 small grants; established good relations with the Departments of Education and Secretariats of Health, as well as other partners; and made significant progress in disseminating health messages and beginning to change health practices. Health, Education And Reconciliation (HEAR) Project 32 Mid-Term Evaluation General Conclusions • The HEAR Project activities support the Government of National Unity (GNU) priorities in health and—to some extent—education, as well as the US Government priorities in supporting the CPA by providing peace dividends in the Three Areas. • The project is under considerable pressure from the local education and health authorities to do more and extend its services.14 • The collaborative approach (involving members of the HEAR staff) to this evaluation was extremely productive, allowing for both a richer and more illuminating evaluation, as well as a beneficial process for the project itself. In no way did the team feel that subjects altered or withheld information due to the presence of project staff. Education Conclusions: • Teacher training and the associated materials are heavily weighted toward the learning and dissemination of health and hygiene messages at the expense of education and reconciliation topics. The applicability to other subject areas of the good teaching techniques used in the workshops, especially on health topics, may not be evident to the trainees. • Much of the pedagogical training struck the Evaluation Team as unrealistic. The necessary conditions that would enable teachers to implement many of the teaching methodologies advocated by HEA— including PCI, LEA, and the development of teaching aids—do not exist in most schools. • Some revision to the teacher training regimen is necessary due to the conclusions above. • The Evaluation Team noted that students, who are accustomed to teacher-centered instruction, may require time to adapt to these methods and to practice what is expected of them under this paradigm. These issues are not addressed in training. • Mobility and turnover of teachers may result in a decrease in the impact of the Project over time in many schools unless there is ongoing training and support. • The description of the Model Schools could be strengthened by including a clearer vision of what a good Model School would look like, how the grant money might be spent, and the particular roles of teachers, head teachers, CHPs and PTAs. A key objective of the Model School is to show good teaching practices; however, to date, the HEAR Project has not made this a priority. o The Evaluation Team believes that the HEAR Project staff is already stretched thin in trying to carry out their current responsibilities, leaving them no time to embark on a major new initiative. In light of this, the feasibility of the Model School proposal, as currently conceived, is left in doubt. o The inclusion of $6,000 for small grants for each school may limit the ability of other schools to replicate the Model Schools. Furthermore, the proposed total of $36,000 for small grants (for the six schools) would take money from possible expansion of the small grants program to incorporate schools that are not yet receiving grants. o The Model School proposal may also create the perception of inequity in the regions served by the HEAR Project, running the risk of opinions being formed that one area or school is favored over others. 14 The Evaluation Team cautions that some activities to support the CPA in other sectors (e.g., Economic Growth or Democracy and Governance) might be too big a shift to include within the context of HEAR. Health, Education And Reconciliation (HEAR) Project 33 Mid-Term Evaluation Recommendations: 1. Training materials should be revised or strengthened to reflect the realities of schools and the methodological/pedagogical needs of the teachers. Topics that should be added in place of PCI and LEA should include strategies for: • Teaching in multiage and multilingual classrooms, and the potential for teaching in multi-grade classrooms • Teaching and managing very large class sizes • Teaching under trees and semi-permanent shelters (no walls for hanging posters or student work) • Teaching without student textbooks and teacher’s guides • Day-to-day checking for understanding (alternatives to choral response) and student assessment • Using found and naturally occurring items as teaching aids, rather than relying on even simple items such as crayons or poster paper which don’t exist in most schools • Keeping classroom records • Continuing to support the use of lessons plans and basic guidelines on how to structure lessons 2. Opportunities for ongoing follow-up and continuing training for HEAR Project teachers should be offered. In-service courses should be held at ERCs over school breaks to offer teachers ongoing support and training. According to ERC managers, the centers have the capacity to house and board, and/or provide transportation costs, to groups of teachers for this purpose. Courses should include: • Computer and Internet training • Intensive English, especially reading and writing • Curriculum development (especially the use of the Internet to find appropriate teaching materials and the revision of current curricula, such as the Kenyan primary school books, to reflect more local knowledge) • Mathematics and science • Classroom management 3. Training for head teachers should be decoupled from the PTA training and restructured as “training-of￾trainers” (TOT) focusing on school leadership, records management, community relations, the completion of a school development plan, and the facilitation of HEAR Project teacher training for new or untrained teachers, as well as refresher training at his/her school. This training should be provided for REO staff as well. 4. The Model Schools proposal as presently articulated should be reconsidered. Without extensions of funding and time, the HEAR Project should abandon this idea and reallocate its funding to the small grants program. There are, however, promising possibilities if additional money and time become available. An initial model school project involving three schools could be developed, two in Kauda and one in Kurmuk, along the lines described below: a) Kurmuk: Health Sciences Academy A small number of primary schools in the Kurmuk area, perhaps one from each cluster, could orient their P5-P8 curriculum to reflect a health sciences vocational track leading to enrollment in the Health Sciences Academy in Kurmuk. Students who enroll could become CHPs for their home communities and schools. This would require some curriculum development and teacher training, especially in math and science. The GEE or BRIDGE Project could be enlisted to cover tuition for girls, and perhaps the HEAR Project could establish a scholarship fund for boys. Tuition is currently 75 SDG for the first year and 50 SDG for the second. b) Kauda: Teacher Training Lab Schools Health, Education And Reconciliation (HEAR) Project 34 Mid-Term Evaluation There are two teacher training institutes in the Kauda area – Kauda TTC and the Yousif Kuwa TTI. One or two schools could be designated as laboratory schools for these institutions. Teachers in these designated schools would be provided with resources and in-depth training to subsequently model effective teaching and management skills for teacher trainees from the institutes. They may also be places where new ideas and approaches to addressing the particular needs and challenges of schools in the region may be tested. Furthermore, graduates would potentially become HEAR Project ambassadors, spreading the methods and messages of the project throughout the region and, in so doing, broadening the Project’s reach. The benefits of the approach outlined above would be a clear purpose and vision; close partnership with existing institutions; mitigation of local political problems; encouragement of multiple ownership; and addressing sustainability. Health Conclusions: • The dissemination of health, hygiene, and disease prevention messages by teachers and CHPs through schools is a success and provides evidence that the HEAR Project approach to intersectoral program design should be considered a model, but it is nevertheless confronted by challenges and has room for improvement. • Without ongoing training and support to increase retention, the mobility and turnover of CHPs may decrease the impact of the project over time. The lack of incentives for the community health promoters has an impact on the level of their activities and their retention. • The full realization of health and education impacts on the HEAR Project in Kauda and Kurmuk are severely hampered by the lack of access to clean water at nearly every school site. The lack of access to water brings into question whether students are able to practice the hygiene habits advocated by their teachers and CHPs. • There remains a lack of independent confirmation of the degree to which students and community members are practicing the good health and hygiene habits disseminated by teachers and CHPs. The issue of measuring actual practice may be addressed by increased school observations by HEAR Project staff to ascertain the particular challenges to latrine use, such as cleanliness, privacy, the alteration of long-conditioned sanitation habits. • The HEAR Project must solicit donations of vitamin A and bednets, as it has no money of its own to procure these items. • The project has been particularly successful in distributing vitamin A pills, reaching 17,215 children out of a life-of-project total of 21,763. • An added benefit is that the HEAR Project has been able to distribute deworming pills at the same time as the vitamin A. • Progress has been slow with respect to the distribution of bednets: just 2,640 were distributed out of a target of 10,000. However, 1,500 nets that were to have been distributed in Abyei were destroyed in the 2008 fighting. It has also been difficult to obtain the bednets from UNICEF out of their Kadugli office. Health, Education And Reconciliation (HEAR) Project 35 Mid-Term Evaluation Recommendations: 1. Robust efforts on the part of HEAR, USAID and implementing partners should be undertaken to find organizations that will drill boreholes in or near the school sites. Potential partners include GOAL in Kurmuk and MC Scotland and PACT in Kauda. GOAL is currently reviewing proposals for boreholes. A proposal from the HEAR Project will include certain advantages that the project can provide to GOAL, such as support to the communities and schools on using the water for good health practices. As BRIDGE is just getting reestablished in Kauda, it would be appropriate to leverage support from MC Scotland with boreholes. 2. The HEAR Project health and M&E staff should design tools to conduct observations at the school and community sites to confirm if students and their families are modeling the behaviors disseminated in training, such as using latrines properly, washing their hands with soap at appropriate times, and bathing and preparing food in hygienic ways. Where these practices are not taking place routinely, the Project can then begin to assess the barriers to these behaviors. 3. The implementation of a new activity tracking tool for Community Health Promoters, already designed by HEAR Project staff, should be implemented as soon as possible and integrated into the regular reporting mechanisms. Reconciliation & Governance Conclusions: • PTAs existed in all schools visited, and those that were trained and active contributed to important improvements to their school, although the number and intensity of their activities tended to be in direct relation to how recently they were trained. • Without ongoing training and support, mobility and turnover of PTA members decreases the impact of the project over time. • Writing School Development Plans—the stated purpose of PTA/Head teacher training—has not been achieved, most likely as a result of language and educational barriers and a lack of training time. Perhaps equally likely as an explanation is the fact that the completion of school development plans is not a contractual deliverable. • Training for head teachers is insufficient in so far as skills for leadership, community relations and PTA management, school record keeping, and teacher support and training are concerned. A number of factors may influence this finding, including education levels of the head teacher and/or teachers, access to resources like record books, limited training time, and overwhelming class sizes. The latter factor is a major hurdle to keeping accurate daily attendance records. But even where this was the case, there remained much potential for these skills to be directed in ways that can better sustain the knowledge, attitudes, and skills disseminated in the HEAR Project trainings for teachers, PTAs and CHPs. Sustainability is thus a primary weakness of HEAR Project training as a whole. • Though the small grants program is underfunded, it has proven to be successful and an unintended positive outcome of the PTA/Head Teacher training. HEAR Project staff have greater incentive to facilitate small grants than they do school development plans. The process of applying for and then implementing a small grant is a good capacity-building experience for the head teachers and PTA members. • The Open School Days, usually funded by the small grants, seem to be a successful activity for the schools and communities. • Management of the small grants program has proven to be more complex and time-consuming than anticipated, even sparking some legal conflict, and has required the creation of a grants committee comprised of local stakeholders in education and health. This is ultimately a positive development, as it Health, Education And Reconciliation (HEAR) Project 36 Mid-Term Evaluation increases transparency. It also suggests that the management of the small grants program may require more time and personnel than originally planned. Recommendations: 1. The HEAR Project should create multi-year service plans for each cluster to ensure that years subsequent to the initial training and service regimen include ongoing support activities. These plans should also direct HEAR Project staff to engage in specific activities to meet this objective. Ongoing support activities could include: • Provision of full training programs on every odd numbered year • Provision of ERC-based follow-up refresher training on every even numbered year • TOT training for head teachers and REO staff (see above) • Follow-up with head teachers on the completion of school development plans to augment and customize multi-year cluster service plans. • Efforts to involve the REOs in taking the lead with follow-up activities 2. The HEAR Project should realign the PTA training away from the goal of writing a school development plan and towards the goal of developing a small grant proposal. This does not mean removing the school development plan altogether, but rather setting aside additional time to focus on a development plan. In the short term, the goal would be a proposal for a small grant. It is also recommended that head teachers, via a separate training, direct the completion of school development plans. This also means that HEAR must have funds available to support a grant at each school. 3. Additional resources should be committed to the small grants program. One way to do this would be to redirect funds away from the Model Schools proposal (see below). This would free up approximately $36,000, as well as other related costs, staff time, and labor. Furthermore, by reducing the maximum amount of a small grant to $1,500 from $2,000—an idea endorsed by the HEAR project staff—these two combined steps would permit another 24 small grants to be issued. Ideally, each school should be eligible for a grant; the project would need to find additional savings or infusions of funds to meet this goal. 4. The HEAR Project should designate a small grants manager to be stationed at each site, preferably hired locally. 5. PTA training should be decoupled from head teacher training and refocused toward the goal of developing a small grant proposal. This does not mean doing away with the school development plan altogether, but entails beginning with the small grants and then providing time and more training to create a school development plan with full participation by the head teachers. Gender Equity Conclusions: • While schools in both regions have made important gains in girls’ enrollment in school, issues such as early marriage, school fees, lack of latrines or latrine privacy, and the lack of the provision of materials to manage girl’s menstrual cycles persist as barriers to girl’s enrollment and retention. • The contribution of the HEAR Project to gender equity is likely most significant in the development and support of PTAs and in their subsequent school construction activities (especially separate and private latrines for boys and girls) and in their community advocacy for girls education. • As the HEAR Project training materials are currently structured, little overt training or direct support in gender equity is provided. Health, Education And Reconciliation (HEAR) Project 37 Mid-Term Evaluation Recommendations: 1. Gender equity issues should be added to all training of teachers, PTAs, and CHPs. Also, the PTA selection process should be managed in such a way so as to increase the representation and subsequent participation of women. The following are suggested additions to the training programs: a) To the teacher training: • Sensitivity to gendered favoritism in the classroom b) To the CHP training • Girls mentoring • Menstruation • Harassment c) To the PTA training • Equitable representation • Democratic decision making • Girls mentoring • Harassment • Community education in gender equity 2. The PTA training, in particular, should add a section on gender equity that orients the PTA as community advocates for girls’ education and the promotion and maintenance of conditions at school that increase the retention of girls, such as scholarships, public posting of student and teacher codes of conduct, the existence of separate and private latrines, and the provision of materials to manage girls menstrual cycles at the school site. Education Resource Centers Conclusions: • The vision and strategy for the role of the ERC seems to be unclear, especially with respect to how best to use its books and computers for strengthening education in the states. • The ERCs have an important role in supporting the objectives of the HEAR Project. In addition to organizing training programs, the centers provide a decent library of textbooks and reference materials on education. Many of these books are being delivered to schools, especially in Kauda. • The ERCs have initiated an ambitious computer training schedule for teachers, officials, and students. The instructional program is improving based on the suggestions of the Evaluation Team, such as including less theory and more practice and getting the learners on the Internet sooner. The PowerPoint presentations are of good quality. • Although it appears likely that the computers and the Internet will build teachers’ knowledge of their subject matter, help them to add new teaching materials, and strengthen their understanding of teaching methodologies, there is no way to be certain of this at this time. • The rationale for teaching computer skills to children in Kurmuk and Kauda seems less robust than that for teaching teachers and officials. The fact that there are so many students enrolled in the course in Kauda and so few computers for them to use leaves the potential impact of the training unclear. • The ERCs, especially in Kauda, devote too much of their time and resources to teaching computer skills to primary school pupils. • It is likely that most books, except for the curriculum books, are used to further the teacher’s own knowledge of core subjects. The degree to which this has an impact on teaching quality is unclear. • The camcorders at the ERCs in Kurmuk and Kauda are beginning to be used. • The security of the computer rooms at each site are a concern. The lock on the door of the Kauda ERC, for example, is too small and could be broken easily. • The ERCs serve nearby clientele but are not able to serve schools at a distance. Health, Education And Reconciliation (HEAR) Project 38 Mid-Term Evaluation • The combined space for computers and books—plus the needed space for training in Kurmuk—is too small. Recommendations: 1. The security at both ERCs should be reviewed and appropriate action should be taken as soon as possible. 2. To make the books more useful, the HEAR Project should: • Provide simple orientation or training to schools that receive the books on topics such as: (a) safe storage of the books, (b) their potential usefulness for the teachers and head teachers, and (c) ways in which they can be used with pupils. The actual usage of books should be tracked. • Some of the books or chapters should be aligned with the New Sudan curriculum and the Kenya curriculum so that teachers who are looking for support in certain areas are directed to the most appropriate books. For example, a fifth grade teacher looking for information on teaching science should be directed to a certain group of books. The HEAR Project should consider holding a workshop for teachers whose task would be to align the books or chapters with their syllabi. • Obtain more textbooks and teacher's guides from the Kenyan and New Sudan syllabi. • Obtain more English reading and picture books, especially for lower primary. 3. Given that each center only has about seven computers, to provide the most relevant support to the states priority should be given to classroom teachers and officials. In addition, ministry officials can begin to use the computers for administrative purposes such as an EMIS, which the HEAR Project is beginning to support. 4. Try to reallocate funds or seek other sources to procure additional computers for the ERCs. 5. Likewise, seek funds to construct a new training room on the ERC compound in Kurmuk. 6. The computer training should be revised in the following way (the Educational Technologies Specialist is already making some of these adjustments): • Rearrange the schedule in Kauda in order to reduce the time for formal presentations and provide more hands-on practical work for the learners. Ideally, there should be a practice session on each day that a group comes to the center. • Arrange to get the learners on the Internet as soon as possible. • The ERC should begin tagging sites that are identified as useful by local teachers and place the links to these sites in a separate folder on the server for future reference. Useful documents should also be downloaded and stored or printed for easy access by other teachers. The ERC should allow teachers or officials to print out a limited number of pages to take with them (this will require some reallocation of funds for additional paper). This should positively impact the sustainability of the project. • The ERC should continue to create videos to document project activities and to strengthen their training programs, such as clips of good teaching practices. 7. Finally, the project should consider some actions to make the ERC resources available to schools in more remote locations. The center should offer additional courses during the school breaks and consider establishing a mobile ERC with the most useful books, health education materials, and several laptop computers. Health, Education And Reconciliation (HEAR) Project 39 Mid-Term Evaluation Partners Conclusions: • The HEAR Project had established excellent relationships with its key partners, the Regional Education Offices and the Secretariats of Health in Kurmuk and Kauda. These relationships are essential for the smooth functioning of the project and for increasing the likelihood of its sustainability. • The HEAR Project has been successful in establishing partnerships with organizations that can supply vitamin A and deworming pills and bednets—UNICEF, WHO, and GOAL. • The project has established relationships with other partners and projects, including SSIRI, the Internews community radio stations, PACT, BRIDGE, and ROOF. • Efforts at partner collaboration and networking have further been enhanced by membership in the South Sudan Health NGO forum. Recommendations: 1. The HEAR project should strengthen some existing partnerships and create new ones. These are outlined as follows: GOAL, BRIDGE and PACT Perhaps the biggest need at most schools is access to water. The HEAR Project should increase its efforts to leverage partners with capacity for drilling boreholes (GOAL in Kurmuk and BRIDGE and PACT in Kauda) in order to increase access to water at schools supported by the HEAR Project. SSIRI Project IRI was originally part of the HEAR Project but was subsequently turned over to the SSIRI Project for sound management reasons. Currently, the SSIRI Project is only being supported in Kauda but is considering reengagement in Kurmuk. The HEAR Project should build on this relationship to encourage greater collaboration such as the following: • In the schools where the HEAR and SSIRI Projects overlap, the two projects should do joint training (as was originally done when IRI was part of the HEAR Project) and collaborative monitoring. • The English language skills of many HEAR Project teachers are weak, especially in Kurmuk. The projects should collaborate in making the SSIRI Rabea English programs available to these teachers for their own professional development. In addition, the Ministry of Education in Blue Nile has recently reassigned about 70 English teachers to Arabic patterns schools where they are to teach English. These teachers might benefit from access to the Rabea programs for use in their classes. The SSIRI Project should provide the devices and the programs, and the HEAR Project should help organize the training and implementation. GEE Project The GEE Project supports secondary school and teacher training institutions, providing tuition support, funds for personal items, sanitary kits, and money for school improvement projects. Given the gender equity challenges in both sites, the GEE or BRIDGE Projects could pay school fees for girls in HEAR Project schools in the upper primary grades and provide sanitary kits. These actions would significantly impact the retention of girls in school. In addition, the combination of the HEAR Project small grants and the GEE or BRIDGE Project school improvement funds would result in facilities or activities for the general benefit of the school and in some cases, such as the building of separate latrines, there would be a special benefit for girls. Health, Education And Reconciliation (HEAR) Project 40 Mid-Term Evaluation Internews Community Radio Stations The HEAR Project should build on its good working relationships with the two radio stations in Kauda and Kurmuk to do the following activities in addition to the current coverage: • Create more short messages to be broadcast on topics such as good health practices, the importance of sending girls to schools, community support for schools, etc. • Conduct interviews with officials, teachers and others on topics related to the HEAR Project. • Sponsor a writing contest or other similar ideas in order to get the voices of children on the air. Health Sciences Academy As mentioned earlier, there is potential for the HEAR Project to collaborate with the new Academy. Kauda TTC and Yousif Kuwa TTI A closer affiliation between the HEAR Project and these two teacher training institutions would be mutually beneficial. The GEE Project is already supporting these institutions, and the Evaluation Team has suggested elsewhere that if the Model Schools idea goes forward, it should be in an affiliation with these institutions (and Health Sciences Academy in Kurmuk) using a Lab School model. Irrespective of the above, the HEAR Project should introduce its program and provide training to the students in their last year of study, so that when the students go out to schools to begin teaching they will have knowledge and skills about the HEAR Project. These TTI students could be another source of "ambassadors" for the project. Part of the training should provide skills on how to promote and disseminate the activities related to the HEAR Project to fellow teachers, head teachers, and PTA members. Management Conclusions: • Overall, the project has benefitted from effective management and a good, dedicated staff. The ERCs seem to be operating well, and the staff believe that the project is well-managed. All staff interviewed expressed acceptance of the problems caused by a difficult environment, e.g., only weekly flights and the lack of banks in Kauda and Kurmuk, which have had an impact on project implementation. • The project has excellent relations with CAII and EDC offices in Nairobi and Washington, DC. The new location of the country office in Rumbek is working well. • The staff has been negatively affected by the high turnover of COPs, though the stability of several key field staff, along with ongoing support from the Project Director in Nairobi, has been important for the continuity of the project. • The staff have several concerns with respect to HR and other related issues, some of which are outlined in the “Project Activities” section above. Other issues can be taken up by the staff directly with CAII. • The HEAR Project has reported on the environmental issues related to the construction of latrines at schools. CAII has submitted to USAID the Environmental Review Form for Construction of Pit latrines which stated the latrines are “very low risk.” There should be no environmental compliance issues. Recommendations: 1. CAII should continue to address the instability of the COP position, even though a new COP will begin work in April and may potentially remain throughout the remaining period of the contract. The new COP should begin making visits to Kauda, Kurmuk and Abyei, planning to visit each site at least once a quarter. 2. The COP, along with CAII management staff in Nairobi and Washington DC, should address the HR concerns raised by field staff in the course of this evaluation. Some of these are outlined in the “Project Activities” section above. CAII should also consider the following staffing changes: Health, Education And Reconciliation (HEAR) Project 41 Mid-Term Evaluation • If the recommendations of this report relating to the development of more materials and training focused on effective teaching are accepted, the ERC in one or both locations will need an additional staff member for teacher training and, most likely, some short-term consulting on curriculum development. (There would also need to be resources allocated for printing new teacher training guides.) • If more resources are allocated to small grants, the program would benefit from the addition of a staff member, ideally a local person, in each location to manage the small grants programs. • In order to take advantage of the broadcasting opportunities offered by the two Internews stations, the project would benefit from having a part-time person to develop messages on special project events, good health practices, the importance of sending girls to school, the role of the PTA, the importance of community support for schools, etc. In addition to short messages on these subjects, the HEAR Project should arrange for an occasional interview with appropriate government officials or others. • The field staff in both locations recommended that CAII appoint a separate HR officer in the Rumbek office. Based on the important and diverse responsibilities of the current Finance and Administration Manager in Rumbek, the Evaluation Team supports this recommendation. • One of the staff, such as the small grants manager or the communications specialist, should also serve as a point person for communication with partners. Project Achievements and Impacts Conclusions: • The project has done a good job compiling and reporting data that relate to the deliverables—the PMP indicators—and the project is on schedule in terms of reaching the end-of-project targets (with the possible exception of the distribution of bednets). • The procedures for processing data and ensuring copies of data sheets in Rumbek and the field sites could be strengthened. • Although the data collected is useful, there is a lack of information on the specific activities at any school as well as the activities of individual community health promoters. Without more complete data on school- and community-level activities, it will be difficult for senior project staff and USAID to understand fully where the HEAR Project activities are being successfully implemented, where challenges exist, and what impact the project has had. (For example: Are teachers trying to implement any of the skills or activities that they learned in HEAR training? To what extent are health messages being disseminated and health practices changing?) • At this point in time, there are neither appropriate forms nor procedures for collecting and compiling data at the school level other than data required for the contractual deliverables, although there has been some recent progress in the area of health. • The evaluation team found that there is room for considerable improvement in collecting and reporting information at the school and community level, such as data on the actual implementation of good health practices or of teaching skills/activities taught during training. The lack of this type of data at the school level may be the result of the following: • The data are not required as part of the PMP deliverables. • The contract with USAID does not require impact measures on improved health or learning gains of children. • The project has few staff with limited transportation. There is one M&E Coordinator and only one M&E Assistant in each of the two locations. • The training programs have improved the knowledge and skills of teachers and community health promoters in the field of health, but have been less successful in improving teachers’ teaching practices. • HEAR is providing an important peace dividend. Health, Education And Reconciliation (HEAR) Project 42 Mid-Term Evaluation • The Project is on track to provide training to teachers, administrators, community health promoters, and PTA members based on the targets in the PMP. Recommendations: 1. A more comprehensive baseline data collection tool should be used when selecting schools for new clusters. A number of tools exist, such as that used by the Education Baseline of Northern States of Sudan, that could be modified to reflect HEAR Project activities and impact targets. Such a tool should, for example, include information on whether a health policy exists and is posted, whether latrines are used, and whether children wash their hands after using the latrine and before eating. 2. The following measures should be implemented in the area of data collection and management: • Enter data in its raw state rather than using aggregated data. • Ensure that there are copies of all data and relevant reports in both Rumbek and the field sites. • In preparation for a possible "education audit" by USAID, identify all data that were destroyed in Abyei in May 2008, and verify that there are at least three days of attendance sheets for each five-day training for teachers, for the training of community health promoters, and for the training of head teachers. The data should be disaggregated when head teachers and PTA members are trained together. 3. HEAR Project staff should collect additional information to inform the Project and USAID as to the successes, impact, and challenges of the Project. In the FY 2010 Work Plan, the HEAR Project plans to conduct school monitoring and support visits on a bimonthly basis. This will be an important step. To strengthen the collection, processing, and reporting of data at the school level the following steps should be taken: • Revise the "School Visit Monitoring Report" to collect more complete data on school activities. It should include basic data such as school visited, date, by whom, and other project related information such as the observed use of latrine, hand washing, etc.; notes on classes observed; special events such as Open School Day or attending a PTA meeting; observing of health messages being delivered to an assembly or in a class; distribution of vitamin A; status report of small grants project; etc. Some of the questions in the current form, such as "Does your school have a latrine?", should be retained as part of an occasional school assessment. • Conduct occasional formal supervision visits, perhaps involving all staff and education and health officials whenever possible. Develop a special form with some training for this purpose. • Implement the new data collection form to be given to community health promoters at their monthly meetings to collect information on each individual. • M&E staff should develop a system for storing and reporting the school-based data, organized by clusters. 4. The HEAR Project staff should begin revising the M&E systems to address insufficiencies in the ability to measure project impact by focusing on the measurement of behaviors and practices. For example, if it is verified that latrines are being used and children are washing their hands, then it would be a reasonable to assume that the health of the children should improve. A new baseline survey (although belated) and follow-up M&E should focus on three sets of indicators: • Health indicators o For example, the extent to which latrines are being used by pupils, whether children wash their hands upon leaving the latrine, whether children wash their hands prior to eating lunch at the schools, whether the children are wearing clean clothes, and whether their bodies are clean. o These indicators should be assessed through a combination of direct observation and interviews. • Education indicators Health, Education And Reconciliation (HEAR) Project 43 Mid-Term Evaluation o For example, the extent to which teachers exhibit behaviors that reflect the activities, knowledge and skills taught by the HEAR Project. Based on the current HEAR Project training, this would be limited—perhaps an indication of skills-based training, the use of the 3P (Presentation, Practice, Performance) model, low-cost teaching aids, and the LEA. o Depending on the acceptance of USAID and CAII of recommendations to strengthen the education component of the HEAR Project, there could be a new set of indicators focusing on basic teaching skills. o The survey should be carried out through a combination of classroom observations and interviews. • Local governance indicators. • Indicators related to the role of the head teacher as a school leader. • Indicators regarding the role and function of the PTAs in governing the school and involving parental support from the community. 5. A comparative study with schools outside of the HEAR Project area should also be undertaken. Without such a comparison, it is difficult to determine whether a change in school indicators is unique to the HEAR project schools or attributable to other factors. External Variables and Sustainability Conclusions: • Sustainability is a complex issue, with each part of the Project needing to be considered separately. Some components or activities might be sustained with relative ease, while others might take many years. • Higher school enrollment suggests strong evidence of “peace dividends.” (As mentioned above, more than half the schools do not have grade eight classes, but most of these schools are adding a new grade each year until they reach grade eight.) In order to sustain such a rapid growth in enrollment, however, complex and often deeply political issues related to teacher compensation, education, training, recruitment, and retention will need to be addressed by multiple actors, including federal and state governments in both the north and south, NGOs, and community-based organizations (CBOs). These issues complicate any speculation about the future sustainability of the HEAR Project, which will likely require direct support from USAID for many years to come. • In light of the political uncertainty and the lack of textbooks, means of transportation, and money to pay staff or teachers, it is hard to imagine that there could be much sustainability of the HEAR Project in the short run. • The political factors described in the previous section pose potential risks for HEAR, especially in Abyei and Southern Kordofan. These areas absorb the consequences of the elections recently held in April 2010 (with a postponement of some of the elections in Southern Kordofan to November 2010), the run￾up and actual referendum on independence in southern Sudan, and the voting in Abyei on whether to join the north or south (scheduled for January 2011). • The HEAR Project recognizes the importance of sustainability and has built into its plans for 2010 many activities that will help to provide the basis to sustain important aspects of the Project. Strategies for sustainability are not integrated into all HEAR Project activities. Recommendations: 1. The sustainability-related issues and risks outlined in the above sections should be addressed through measures including: • The provision of multi-year cluster service plans and follow-up and ongoing training for teachers, PTAs, and CHPs for all school clusters. Health, Education And Reconciliation (HEAR) Project 44 Mid-Term Evaluation • The reorientation of head teacher training to a training-of-trainers approach. • The drilling of boreholes at participating school sites where none are within a reasonable distance. • The establishment and/or strengthening of local, regional, national, and international partners. • The inclusion of a session to assess the implications of the recent elections and the run-up to the referendum for southern Sudan (and the vote in Abyei whether to join the north or south) during the upcoming HEAR planning meeting in May. This assessment could include possible scenarios relating to potential instability and violence and the implications for HEAR—both for activities undertaken prior to the end of the contract in September 2011, as well as possible follow-on activities. • Of special note would be possible outcomes of the referendum in Abyei – a vote to join the north or the south, and whether subsequent to the vote there is a peaceful or not peaceful transition. 2. Finally, given the unstable situation in the Three Areas, it may be unrealistic at this point to expect that many of the HEAR activities could be sustained without further support when the Project ends in 2011. Rather, it may be more appropriate to consider continuity as the most realistic template for forward planning. Such planning could take into account the evolving political situation and give special emphasis to future sustainability. This assessment should not detract from the many good ideas that HEAR has included in its plans to support and encourage sustainability. Health, Education And Reconciliation (HEAR) Project 45 Mid-Term Evaluation APPENDICES Appendix A Work Plan HEAR Evaluation Activity MSI Thomas Tilson and Andrew Epstein February 10 (Updated March 5), 2010 BACKGROUND The mid-term evaluation of the Health, Education and Reconciliation (HEAR) Project will be conducted from February 6 through March 6, 2010. The evaluation process will actively engage the stakeholders, including USAID-Sudan, the education and health sectors Blue Nile and Southern Kordofan, the contractor and one subcontractor Creative Associates International Incorporated (CAII) and Education Development Center (EDC), and other partners and stakeholders. Abyei will not be visited. The evaluation will assess the progress to date of the project from its inception. The two external evaluators (contracted by MSI) are Thomas Tilson and Andrew Epstein. While traveling to Kauda, Kurmuk and Rumbek, there will also be a representative from USAID (Anyieth Ayuen to Kurmuk) and two representatives from CAII (Ayella Cosmas, M&E Advisor, and Obini Leonard, EDC Education Technology Specialist). The COP is not able to travel with the Team. The Team met a representative from EDC (Kent Noel, Regional Director) in Nairobi prior to arriving in Juba and will meet with him again in Juba after the Team returns from the field. The Team will take an open and collaborative approach; however, the ultimate responsibility for gathering and interpreting information rests with the MSI evaluators. The Evaluation Team will consider the appropriateness of having the CAII representatives accompany them for each interview and decide whether or not one or more CAII/EDC staff will participate in the interviews. PURPOSE The Evaluation Team will report on the performance of CAII with regard to the outcomes and deliverables stated in the original and modified Cooperative Agreement. This will entail measurement of performance in meeting project goals, objectives, reporting requirements, deliverables, and targets. The broader purpose is to answer the following questions: • What is the project’s strategic relevance to the USAID/USG Sudan program and foreign policy priorities – is this activity rendering clear CPS "peace dividends?” • Is the project effective in enhancing the demand for and supply of social services, institutionalizing a decentralized approach to service delivery and promoting accountability and good governance at all levels? • What is its performance in terms of results achieved? • What recommendations are there for moving forward if the activity were to continue and/or expand – is it replicable and desirable? Health, Education And Reconciliation (HEAR) Project 46 Mid-Term Evaluation • What are the best and worst practices in terms of project implementation? Are there key lessons to be learned that would be useful for other USAID projects in Sudan? There are 14 additional questions to be addressed which are listed in the attached document, ‘Getting to Answers.’ METHODOLOGY The Evaluation Team began to prepare for the evaluation by reviewing project documents provided by MSI, Creative Associates International, Inc., and EDC, including all contract-related files, myriads of reports (Annual, Quarterly, Weekly, Trip Reports), etc. In Juba, the Evaluation Team held briefings with MSI and USAID representatives plus the HEAR COP from CAII. In addition, they held a meeting with the EDC Regional Director in Nairobi. The evaluation will be more qualitative than qualitative, as no research has been done on the project that could provide sound data regarding effectiveness and impact. The one area in which the Evaluation Team will provide quantitative data is in measuring the contractually-based targets against the actual numbers of obtained, e.g., number of teachers trained, number of pupils in the associated schools, the number of vitamin A pills distributed, etc. There will be two major methodological approaches used in the evaluation: document review and interviews. Content analyses will be carried on both types of information. The Evaluation Team will review many M&E documents in Rumbek, as well as in the two field sites. The documents will include: 1. Contract agreement and modifications 2. Project reports: Annual Reports, Quarterly Reports, Bi-weekly Reports, and other reports as deemed appropriate 3. Training manuals and educational resources 4. M&E data including individuals trained, school monitoring reports, etc. The Evaluation Team has developed interview protocols (attached) designed to provide answers to the evaluation questions in the SOW. Some of the people to be interviewed include the following: 1. Relevant project staff, including the COP, M&E Coordinator, trainers, health staff, Education Technology Specialist, and ERC Manager. 2. Senior education and health officials 3. Teachers and head teachers 4. PTA members 5. Community health promoters 6. Representatives from partner organizations, e.g., EDC, Mercy Corps, UNICEF, Voice of Kauda 7. USAID COTR and other officials In addition, the Evaluation Team also plans to observe some classes of teachers trained by HEAR. The Evaluation Team will be looking for evidence of HEAR resource materials and teaching behaviors that reflect the skills taught in the HEAR training programs. The Evaluation Team will conduct extensive data analyses, including the following: 1. There will be extensive content analysis of project documents 2. Notes from interviews will be used for analysis 3. Review of target numbers submitted to USAID by CAII 4. Extensive use of comparison such as progress on project indicators relative to established targets Health, Education And Reconciliation (HEAR) Project 47 Mid-Term Evaluation 5. Occasional use of trend analyses and cross-tabulation The Evaluation Team does not expect to find hard data on project impact on an increase in pupil learning or improved health. Therefore, in responding to questions relating to impact, the Evaluation Team will seek to pull together perceptions and opinions of a range of stakeholders including head teachers, teachers, parents, PTA members, etc. The Evaluation Team developed a Work Plan (this document) including: 1. Basic plan for responding to each of the evaluation questions – Getting to Questions (attached) 2. Methodology (this section) 3. Interview protocols (attached) 4. Schedule for the Evaluation Team including details on places to visit, people to interview, classes to be observed, etc. (attached) OUTCOMES AND DELIVERABLES The following documents will be submitted: 1. Initial Work Plan (current document) following briefing sessions with MSI and USAID 2. A debriefing presentation and outline draft complete with PowerPoint to stakeholders in Juba prior to departure of the evaluation team 3. Draft Final Report with the following sections prior the departure of the Team Leader a. Executive Summary b. Introduction c. Background d. Methodology e. Findings f. Conclusions and Lessons Learned g. Recommendations h. Annexes 4. Revised Final Report within 5 work days after receipt of comments submitted by USAID Health, Education And Reconciliation (HEAR) Project 48 Mid-Term Evaluation GETTING TO ANSWERS – HEAR PROJECT - February 7, 2010 METHODS DATA COLLECTION QUESTIONS TYPE ANSWER/ EVIDENCE Yes/No Descriptive Comparison Explanation Cause/Effect METHOD e.g., interviews protocols SOURCE Who Where what SELECTION APPROACH All Sample Sample based on what’s new DATA ANALYSIS METHODS Content analysis Frequency distribution Cross tab GENERAL 1. What is the project’s strategic relevance to the USAID/USG Sudan program and foreign policy priorities – is this activity rendering clear CPA “peace dividends?” • Descriptive • Document review • HEAR SOW and objectives as amended • USAID/USG documents • Interviews – USAID, education and health officials, relevant staff • Relevant USAID and staff • Key government officials • Content analysis of documents and interviews 2. Is the project effective in enhancing the demand for and supply of social services, institutionalizing a decentralized approach to service delivery and promoting accountability and good governance at all levels? • Descriptive • Document review • Interviews • Education and health officials, head teachers, parents • Key relevant document • Relevant staff and officials • Content analysis of documents and interviews 3. What is the performance in terms of results achieved? • Descriptive • See question 6 in next section 4. What recommendations are there for moving forward if the activity were to continue and/or expand – is it replicable and desirable? • Descriptive • Documents • Interviews • Analyses • All sources leading up to findings and recommendations • All sources leading up to findings and recommendations • Content analysis of documents and interviews 5. What are the best and worst practices in terms of project implementation? Are there key lessons to be learned that would be useful for other USAID program in Sudan? • Descriptive • See question 10 in next section EVALUATION QUESTIONS PROJECT DESIGN 1. How well does HEAR correspond to the government education and health priorities in the Three Areas? • Comparison Document review and interviews • Ministry education and health, and SSRRC officials. • HEAR documents • Teachers and community • Key relevant staff and officials • Sample teachers and community health promoters • Document comparison Content analysis including coding of interview notes or Health, Education And Reconciliation (HEAR) Project 49 Mid-Term Evaluation METHODS DATA COLLECTION QUESTIONS TYPE ANSWER/ EVIDENCE Yes/No Descriptive Comparison Explanation Cause/Effect METHOD e.g., interviews protocols SOURCE Who Where what SELECTION APPROACH All Sample Sample based on what’s new DATA ANALYSIS METHODS Content analysis Frequency distribution Cross tab health promoters at visited schools (possible random sample on site) transcripts 2. Is the Health Education and Reconciliation model having success in improving the skills of teachers, community health promoters and PTAs? For example: How effective has HEAR been in improving the capacity of PTAs to improve school governance? And what are the results, if any, of PTAs activities on improved education through increased attendance? Recommendations should be provided on what elements should be modified/cut/added to improve the approach. • Descriptive. • Yes/No • Documents review • Interviews • School/classroom observations • Focus group • School and PTA records including pupil attendance • Interviews teachers, HT, PTA and community health promoters, and non PTA parents • Class observations • Sample schools in consultation with staff represented a range of services delivered • PTA and non-PTA focus groups chosen by HT • Class observation of a sample of trained and non￾trained teachers distributed among grade levels (randomly selected as possible) • Content analysis including interview transcripts • Analysis of classroom observation forms based on best practices from training manuals 3. Is the monitoring and reporting system providing the program with adequate data to evaluate its impact? • Yes/no • Document review • Interview • M&E reports • Interview M&E Specialist and COP and, possibly, COTR • Review all M&E documents • Relevant staff • Content analysis including interview transcripts/notes 4. Describe and assess the measures being undertaken to ensure the sustainability of the capacities being developed under HEAR. What are the constraints to sustainability, and what can HEAR do to mitigate them? • Descriptive • Document review • Interview • HEAR documents. Interview COP, education & health officials • Relevant staff and officials • Content analysis of documents and interviews Health, Education And Reconciliation (HEAR) Project 50 Mid-Term Evaluation METHODS DATA COLLECTION QUESTIONS TYPE ANSWER/ EVIDENCE Yes/No Descriptive Comparison Explanation Cause/Effect METHOD e.g., interviews protocols SOURCE Who Where what SELECTION APPROACH All Sample Sample based on what’s new DATA ANALYSIS METHODS Content analysis Frequency distribution Cross tab PROJECT IMPLEMENATION 5. Provide a brief description of the program outcomes, deliverables, and products. Assess the quality of the deliverables to date. Identify particular strategies, activities, or programs that are effective and describe why they have worked. For example, • Have there been improvement to children’s health as a result of HEAR’s distribution of vitamin A, mosquito nets and the construction of latrines and/or boreholes? • Descriptive • Yes/no • Descriptive • Documents • Records • Interviews • Project documents including contract and mods, PMP, revised SOWs, qtr and annual reports • Health studies • School records (e.g., attendance) • Parents, project staff, community health promoters, teachers HT, PTA • Relevant documents and staff • Sample schools, parents • Content analysis of documents and interviews • Content analysis of documents and interviews • How effective are Education Resource Centers (ERCs) in providing in-service training and follow up support to teachers? • M&E data • Descriptive • Data review • Interviews • M&E documents (e.g., training attendance sheets and schools monitoring forms) • Teachers, HT • Relevant documents and data and teachers from sample schools • Content analysis of documents and interviews • Compare results to targets • Trend analysis • How effective are they in recruiting and training community members to become health workers? • M&E data • Descriptive • Data review • Interviews • M&E documents (e.g., training monitoring forms) • Community health promoters • Relevant documents and data and community health promoters • Content analysis of documents and interviews • Compare results to targets • Trend analysis • How effective has HEAR been in encouraging community health workers to deliver effective health and hygiene messages? • Descriptive • Data and documents • Interviews • Community health promoters • M&E data • Parents • Relevant documents • Parents from sample schools • Content analysis of documents and interviews • Health, Education And Reconciliation (HEAR) Project 51 Mid-Term Evaluation METHODS DATA COLLECTION QUESTIONS TYPE ANSWER/ EVIDENCE Yes/No Descriptive Comparison Explanation Cause/Effect METHOD e.g., interviews protocols SOURCE Who Where what SELECTION APPROACH All Sample Sample based on what’s new DATA ANALYSIS METHODS Content analysis Frequency distribution Cross tab • How effective are HEAR-distributed resource materials in reinforcing student learning? • Descriptive • Interviews • Observations • Teachers, HT, parents • Sample schools • Content analysis • Interviews • Observations • Are education levels of targeted children improving as a result of HEAR activities? • Descriptive • Comparative • Data • Interviews • Past and current school data (e.g., attendance, enrollment) • Local school exam data • ERC training data • Interviews teachers, parents • Sample schools • Content analysis • Interviews • Trend analysis and cross tabs 6. Is HEAR progress to date in meeting the deliverables of the Contract Agreement (as amended) and HEAR’s implementation targets (as specified in the project documents) on track for the project’s current stage of implementation? Describe any areas of concern or of accelerated success in implementation. • Yes/No • Descriptive • Documents • Interviews • Contract documents • M&E data • M&E Specialist, COP • Relevant staff and documents • Comparison targets and deliverables • Content analysis documents and interviews 7. Is the activity effectively addressing gender issues such as inclusiveness, training and promotion, and the gender gap in Education Departments in the Three Areas? If not, why not? What are the areas for improvement? • Yes/No • Descriptive • Records/ documents • Interviews • ERC training records • Training materials • HEAR staff & education officials • School records and staff • Relevant staff, officials and documents • Sample schools • Content analysis of documents, records, and interviews • Trent analysis and cross tabs 8. Does the program have environmental compliance issues? Is so, how well has it performed against USF guidelines and regulations? • Yes/No • Comparison • Documents • Observations • Documents • ERC and schools • Sample schools • ERC sites • Content analysis 9. Assess the quality and performance of Creative Associates International Incorporated in managing the implementation of HEAR. What are the team’s strengths, weaknesses, and areas • Descriptive • Interviews • Documents • Contracts documents and reports • Staffing trends • COP, COTR, M&E, • Relevant documents, staff and officials • Sample HTs and • Content analysis of documents and interviews Health, Education And Reconciliation (HEAR) Project 52 Mid-Term Evaluation METHODS DATA COLLECTION QUESTIONS TYPE ANSWER/ EVIDENCE Yes/No Descriptive Comparison Explanation Cause/Effect METHOD e.g., interviews protocols SOURCE Who Where what SELECTION APPROACH All Sample Sample based on what’s new DATA ANALYSIS METHODS Content analysis Frequency distribution Cross tab for improvement with respect to contract management and communications with USAID, government of the Three Areas and stakeholders? Has the contractor provided adequate value for money? relevant staff and officials, and parents, HTs parents 10. Identify the best and worst practices, success stories, and testimonials. Are there important lessons to be lessons to be learned, and is there a story which can best demonstrate to the U.S. Congress lasting impacts on the project? • Descriptive • Documents • Interviews • Findings and interviews • Documents provided by CAII and EDC • Relevant documents and interviews • Content analysis of documents and interviews 11. What relationships has HEAR developed with other implementing partners in the Three Areas (both USAID and others)? Has HEAR effectively leveraged these relationships, and what has been the impact on the project? • Descriptive • Yes/No • Documents • Interviews • Relevant documents and staff from implementing and supporting partners • Relevant documents and staff • Content analysis of documents and interviews 12. What are the main external (exogenous) variables or events that have impacted project implementation? How has HEAR responded? • Descriptive • Documents • Interviews • Documents re. political environment, security. • COP, relevant officials, teachers, parents, and USG officials • Relevant documents and officials • Sample schools • Content analysis of documents and interviews PROJECT IMPACT TO DATE 13. Is this activity rendering a clear CPA "peace dividend"? To what extend has the HEAR program responded to the education and health needs of children in the Three Areas? What has been the impact of program interventions to date? • Descriptive • Documents • Interviews • Previous needs assessment • Teachers, parents and community health promoters • Relevant documents • Sample schools • Community health promoters • Content analysis of documents and interviews Health, Education And Reconciliation (HEAR) Project 53 Mid-Term Evaluation METHODS DATA COLLECTION QUESTIONS TYPE ANSWER/ EVIDENCE Yes/No Descriptive Comparison Explanation Cause/Effect METHOD e.g., interviews protocols SOURCE Who Where what SELECTION APPROACH All Sample Sample based on what’s new DATA ANALYSIS METHODS Content analysis Frequency distribution Cross tab 14. Is the program on track to be effective in improving the capacity of education departments, teachers/administrators, community health promoters and PTAs to provide high quality education in the Three Areas? • Descriptive • Comparative • Documents • Interviews • Contract documents • Reports • Staff, officials, teachers, parents • Relevant documents, staff and officials • Sample schools • Content analysis of documents and interviews Health, Education And Reconciliation (HEAR) Project 54 Mid-Term Evaluation Appendix B Activities Funded by Small Grants Activities Completed Under Review Total Kitchens 7 2 9 Water tanks 5 1 6 Latrines 4 4 Fencing 3 3 Classrooms 2 2 4 Open School Days 3 3 Borehole repair (combined with below in one school) 1 1 Sports Equipment 1 1 TOTAL 26 5 31 Health, Education And Reconciliation (HEAR) Project 55 Mid-Term Evaluation Appendix C Schedule USAID/Sudan Schedule for Mid-Term Review of the Health and Education Reconciliation Program (HEAR) Phase II Field work February 6 to March 6, 2010 Revised March 5, 2009 Saturday, February 6, 2009: Juba • Tom Tilson and Andrew Epstein arrive in Juba o Lodging: Jebel Lodge o Transport: MSI • Team Planning Meeting Begins with Ami, Andrew and Tom (see separate TPM schedule) • USAID BBQ Monday, February 8, 2009: Juba • Team Planning Meeting continues with Ami, Pia, and Anyieth • 14:00 – 15:45 Leesa and Tassew to join • 15:00 – 15:30 Meeting with Peter Malnak, USAID/Sudan Head of Mission in Juba Tuesday, February 9, 2009: Juba to Kurmuk • Participants in Kurmuk trip: Tom Tilson (Evaluator), Andrew Epstein (Evaluator), Anyieth Ayuen (USAID) • Met at airport by CAII/HEAR staff o Lodging: Lutheran World Federation o Transport: Car arranged by HEAR (paid by MSI) • Staff meeting for introduction and program overview Wednesday, February 10, 2009: Kurmuk • Continue with staff briefing • Visit SRRC • Visit new Teacher Training Institute • Visit Zerbia Program School, Cluster 1 • Meeting with Director of County Regional Education Office • Meeting with ERC Education Technologies Specialist • Meeting with Dr. Atar Evans, Former County Health Director Health, Education And Reconciliation (HEAR) Project 56 Mid-Term Evaluation Thursday, February 11, 2009: Chali (3 hour drive) • Visit to Cluster 2 schools • Penamayu • Chali Rasha • Chali Talib • Borfa Friday, February 12, 2009: Kurmuk • ERC computer training • Refresher course for Cluster 1 at Kurmuk Model School Saturday, February 13, 2009: Kurmuk • Visit Cluster 1 schools • Kurmuk Model School • Comboni School • CHP refresher course for Cluster 1 • ERC computer class Sunday, February 14, 2009: Kurmuk • Document review and rest Monday, February 15, 2009: Kurmuk • Meet with Dr. Musa Bakr, National Health Insurance Fund clinic • Visit Cluster 1 schools o Jorot East o Jorot West • Meet with Dr. Ali Khawui, Director of Health Services • Meeting with partners o GOAL o Meeting within Kurmuk Community Radio, Internews o Meeting ROOF • HEAR staff debriefing Tuesday, February 16, 2009: Kurmuk to Malakal • Team Departs Kurmuk • Arrive in Malakal o Lodging: (South Sudan Hotel) Health, Education And Reconciliation (HEAR) Project 57 Mid-Term Evaluation Wednesday, February 17, 2009: Malakal • Work on trip report Thursday, February 18, 2009: Malakal to Kauda • Participants travel to Kauda: Tom Tilson (Evaluator), Andrew Epstein (Evaluator), CAII M&E Coordinator • Arrive in Kauda o Lodging: NCA • Meet HEAR Staff • Confirm the program for the week for changes and or modification Friday, February 19, 2009: Kauda • Orientation with staff • ERC computer training • Meeting with partners o PACT o Voice of Community Radio, Internews • Meet with Secretariat of Health Saturday, February 20, 2009: Kauda • Meet with Regional Education Office • Visit Cluster 4 schools • Alhilu • Ardikanan Sunday, February 21, 2009: Kauda • Document review and rest Monday, February 22, 2009: Kauda • Visit Cluster 1 schools o Badura o Karindi Tuesday, February 23, 2009: Kauda • Visit Cluster 1 schools o Kauda Model o Kudi A o Lower Kumo • Visit SRRC • Visit MC-Scotland Health, Education And Reconciliation (HEAR) Project 58 Mid-Term Evaluation Wednesday, February 24, 2009: • Visit with KTTC, both HEAR and GEE • Visit Yousif Kuwa re. GEE • Meet with GEE mentors • Staff debriefing Thursday, February 25, 2009: Kauda to Rumbek • Team departs Kauda • Arrive in Rumbek. Palm Tree Friday, February 26, 2009: Rumbek • Review M&E documents Saturday, February 27, 2009: Rumbek to Juba • Compilation of trip report, preparation for presentation Sunday, February 28, 2009: Juba • Day off Monday, March 1, 2009: Juba • Compilation of trip report, preparation for presentation Tuesday, March 2, 2009: Juba • Compilation of trip report, preparation for presentation • Show first draft of presentation to MSI Wednesday, March 3, 2009: Juba • 10:00 Presentation of draft findings and recommendations to USAID • 2:00 Presentation to USAID and stakeholders • Further discussions of final recommendations and way Health, Education And Reconciliation (HEAR) Project 59 Mid-Term Evaluation Thursday, March 4, 2009: Juba • 10:00 Presentation to Acting Deputy Mission Director, Jeremy Gustafson • Writing Draft Friday, March 5, 2009: Juba • Writing Draft • Submit draft Saturday, March 6, 2009: Juba • Depart Juba Health, Education And Reconciliation (HEAR) Project 60 Mid-Term Evaluation Appendix D Evaluation Questions Project Design 1. How well does HEAR correspond to the government education and health priorities in the Three Areas? 2. Is the Health Education and Reconciliation model having success in improving the skills of teachers, community health promoters and PTAs? For example: How effective has HEAR been in improving the capacity of PTAs to improve school governance? And what are the results, if any, of PTAs activities on improved education through increased attendance? Recommendations should be provided on what elements should be modified/cut/added to improve the approach. 3. Is the monitoring and reporting system providing the program with adequate data to evaluate its impact? 4. Describe and assess the measures being undertaken to ensure the sustainability of the capacities being developed under HEAR. What are the constraints to sustainability, and what can HEAR do to mitigate them? Project implementation 5. Provide a brief description of the program outcomes, deliverables, and products. Assess the quality of the deliverables to date. Identify particular strategies, activities, or programs that are effective and describe why they have worked. For example, • Have there been improvement to children’s health as a result of HEAR’s distribution of vitamin A, mosquito nets and the construction of latrines and/or boreholes? • How effective are Education Resource Centers (ERCs) in providing in-service training and follow up support to teachers? How effective are they in recruiting and training community members to become health workers? • How effective has HEAR been in encouraging community health workers to deliver effective health and hygiene messages? • How effective are HEAR-distributed resource material in reinforcing student learning? • Are education levels of targeted children improving as a result of HEAR activities? 6. Is HEAR progress to date in meeting the deliverables of the Contract Agreement (as amended) and HEAR’s implementation targets (as specified in project documents) on track for the project’s current stage of implementation? Describe any areas of concern or of accelerated success in implementation. 7. Is the activity effectively addressing gender issues such as inclusiveness, training and promotion, and the gender gap in Education Departments in the Three Areas? If not, why not? What are the areas for improvement? 8. Does the program have environmental compliance issues? If so, how well has it performed against USG guidelines and regulations? 9. Assess the quality and performance of Creative Associates International Incorporated in managing the implementation of HEAR. What are the team’s strengths, weaknesses, and areas for improvement with respect to contract management and communications with USAID, government of the Three Areas and stakeholders? Has the contractor provided adequate value for money? Health, Education And Reconciliation (HEAR) Project 61 Mid-Term Evaluation 10. Identify the best and worst practices, success stories, and testimonials. Are there important lessons to be learned, and is there a story which can best demonstrate to the U.S. Congress lasting impacts from the project? 11. What relationships has HEAR developed with other implementing partners in the Three Areas (both USAID and others)? Has HEAR effectively leveraged these relationships, and what has been the impact on the project? 12. What are the main external (exogenous) variables or events that have impacted project implementation? How has HEAR responded? Project impact to date 13. Is this activity rendering a clear CPA “peace dividend? To what extent has the HEAR program responded to the education and health needs of children in the Three Areas? What has been the impact of program interventions to date? 14. Is the program on track to be effective in improving the capacity of education departments, teachers/ administrators, community health promoters and PTAs to provide high quality education in the Three Areas? Health, Education And Reconciliation (HEAR) Project 62 Mid-Term Evaluation Appendix E Tables on Deliverables Summary Indicator Table - (Quarter 1, Year 4, December 31, 2009) Quarterly Achievements Deliverable Indicators Annual Target Q1 Q2 Q3 Q4 Total Year 4 Achieved Percentage Achieved Number of schools supported by the project 24 3 - - - 3 12% Number of teachers trained, improving teaching quality 170 0 - - - 0 0% Number of learners enrolled in USG-supported primary schools or equivalent non-school￾based settings 9,000 1,128 - - - 1,128 12% Number of Parent Teacher Associations strengthened and training provided 33 0 - - - 0 0% Number of school administrators and head teachers trained 133 0 - - - 0 0% Number of students covered by Vitamin A program 2,700 152 - - - 152 5% Number of long-lasting insecticide treated bed nets per student in each household 5,000 1,640 - - - 1,164 32% Number of schools and Primary Health care facilities with Latrines and Boreholes 10 0 - - - 0 0% Number of communities/ locations involved in strengthening school and health services 92 13 - - - 13 14% Number of Community Health Workers recruited/trained for school linkages 120 45 - - - 0 0% Number of community development projects completed 14 4 - - - 4 28% Health, Education And Reconciliation (HEAR) Project 63 Mid-Term Evaluation Summary Indicators - Totals to date and End of Project Targets (As of December 31, 2009) Achieved to Date End of Project S/N Indicator Targets M F T M F T % Achieved 1 Learners enrolled 20,007 14,294 34,301 32,283 17,717 50,000 69% 2 Teachers trained 506 145 651 714 266 980 66% 3 Administrators/Officials trained 210 17 227 355 65 420 54% 4 PTAs support 87 140 62% 5 Community development projects completed 20 40 50% 6 Schools supported 99 140 71% 7 Students receiving vitamin A 17,215 21,763 79% 8 Bednets distributed 2,640 10,000 26% 9 Community health workers trained 216 155 371 302 258 560 66% 10 Schools with boreholes & latrines 27 45 60% 11 Communities strengthening school and health services 322 460 70% Health, Education And Reconciliation (HEAR) Project 64 Mid-Term Evaluation Appendix F Possible Success Stories -10- Identify the best and worst practices, success stories, and testimonials. Are there important lessons to be learned, and is there a story which can best demonstrate to the U.S. Congress lasting impacts from the project? A possible success story is Penamayu Primary School, located about a two- hour drive away from Kurmuk. This is an English pattern school consisting mostly of pupils and teachers who have returned from refugee camps in Ethiopia. Prior to the intervention of HEAR, all classes were held under trees. However, just three days after the HEAR training for PTA members and head teachers, the school rallied the community to quickly construct many grass classrooms that finally provided some shelter from the sun and rain. In addition, the community has started an agricultural plot to grow groundnuts and sorghum as a source of income for the school. Finally, the community has gathered a substantial pile of rocks and is trying to get help from a NGO to build permanent classrooms. Health, Education And Reconciliation (HEAR) Project 65 Mid-Term Evaluation Appendix G Interview and group discussion protocols Health Ministry Education Ministry Teachers CHP PTA Parents (FGD) Head Teachers 1. What is your role with the Health Ministry? 1. What is your role with the Education Ministry? 1. How long have been a teaching? 1. How long have you been a CHP? 1. Are parents more involved in schools since the HEAR project started? If so, in what ways? 1. How long have been a teaching? 2. Tell me about your interaction with the HEAR project staff and activities; who have you met/communicated with and what have you seen? 2. Tell me about your interaction with the HEAR project staff and activities; who have you met/communicated with and what have you seen? 2. What is your training and education? 2. What is your training and education? 2. What are the biggest issues that keep parents from being more involved in schools? Is the HEAR project adressing these issues? 2. How long have you beend a head teacher? 3. Have you takled to any of the project beneficiaries like teachers, students, or paretns? If so, what have they told you about the HEAR project? 3. Have you takled to any of the project beneficiaries like taechers, students, or paretns? If so, what have they told you about the HEAR project? 3. Which HEAR project training have you completed? (Initial, refreshers…) 3. Did you complete the CHP training? If so, was the training helpul? What was especially useful? 3. Are children healthier since the start of the HEAR project? Why or why not? 3. What is your training and education? 4. What are the primary health priorities in this part of Sudan? 4. What are the primary Education priorities in this part of Sudan? 4. Have HEAR project staff observed you teaching? Have these visits been helpful? Why? 4. Do you continue to get support from the ERC? If so, in what ways? 4. Are children doing better in school? Why or why not? 4. Which HEAR project training have you completed? (Initial, refreshers…) 5. How do the HEAR project activites align with or support these priorities? 5.Do the HEAR project activites align with or support these priorities? What parts of the project are especially helpful? What are your concerns about the project? 5. For those who have attended HEAR project training: Are you able to use the ideas and skills from the training in your classroom? If so, which ones? If not, why not? 5. Describe your health promotion actvities in the community. 5. Have any of you interacted with HEAR project staff? If so, who and what were they doing? 5. Have HEAR project staff observed you teaching or your activities to manage the school? Is so, can you describe your interaction with the HEAR staff? Health, Education And Reconciliation (HEAR) Project 66 Mid-Term Evaluation Health Ministry Education Ministry Teachers CHP PTA Parents (FGD) Head Teachers 6. What needs to be done for thes project activities to continue after the USAID funding cycle is complete in 2011? 6. What needs to be done for this project to continue after the USAID funding cycle is complete in XXXX? 6. Do you believe the HEAR project is meeting the needs of teachers and students? Why? 6. Do you belive the HEAR project is adressing the health needs of the community? If "yes," in what ways? 6. What are the biggest challenges here in the community to the health and education of your children? Are HEAR activities helping to address the challenges? 6. About how many of your teachers have completed any of the HEAR training? 7. What internal and external challenges must be addressed in order for this program to meet its goals and continue after the funding cycle is complete? 7. Do you believe the HEAR project will improve the capacity of the education ministryto provide high quality education in the Three Areas? 7. Has the HEAR project increased the involvement of parents in your school? Can you explain? 7. Has the health of young people improved since the HEAR project started? If so, how do you know? 7. Are there any challenegs that come from outside the community, such as political, ethnic, or econimic challenges? 7. Of those who have attended training: Are they able to use the ideas and skills from the training in their classrooms? If so, which ideas/skills? If not, why not? 8. What recommendations do you have to improve the project? 8. What internal and external challenges must be addressed in order for this program to meet its goals and continue after the funding cycle is complete? 8. Has the HEAR project addressed the health needs of students and their families? If yes, in what ways? 8. How were you recruited to become a CHP? What do you like about this job? Dislike? 8. What recommendations do you have to improve the project? 8. Do you believe the HEAR project is meeting the needs of teachers and students? Why. 9. What recommendations do you have to improve the project? 9. Has the HEAR project addressed the health needs of students and their families? (if "yes," in what ways? 9. Is the ERC successful in recruiting enough CHP's? What are the best ways to do this? 9. Has your schools received a school grant? If so, to do what? How has this proceded? Do you have any concerns? 9. Has the HEAR project increased the involvement of parents in your school? If so, please explain. 10. Of the schools served by each ERC, how many are English schools? 10. Has student attendence improved? If so, why. 10. Has the community responded positively to your activities? Why or why not? 10. Has the HEAR project addressed the health needs of students and their families? If so, can you describe the activities and changes? Health, Education And Reconciliation (HEAR) Project 67 Mid-Term Evaluation Health Ministry Education Ministry Teachers CHP PTA Parents (FGD) Head Teachers 11. Has student attendence improved? In "yes," how much a difference has there been and is this improvement as a result of HEAR activities? 11. What recommendations do you have to improve the project? 11. Has the health of your students improved since the HEAR project started? 12. Have the enrollment, retention, and performance of girls increased? If "yes," can you please explain? 12. Has student attendence improved? If so, why? 13. Do you or other teachers and students, visit the ERC? If so, how aften and for what purposes? If not, why not? 13. Has the academic performance of your students improved? 14. What recommendations do you have to improve the project? 14. Have the enrollment, retention, and performance of girls increased? If so, can you explain the changes and why they have occurred? 15. Do teachers and students visit the ERC? If so, how aften and for what purposes? If not, why not? 16. Do you believe the HEAR project is meeting the education priorities of this area of Sudan? If "yes," can you explain? 17. What recommendations do you have to improve the project? Health, Education And Reconciliation (HEAR) Project 68 Mid-Term Evaluation COP COTR SSRRC M&E Specialist Supporting Partners USG Officials 1. Do you believe the HEAR project is having a positive effect in the communities where it has been implimented? How do you know? 1. Is the HEAR project monitoring and reporting system providing you with adequate data to evaluate its impact? 1. Is the HEAR project addressing the health and education priorities of the Government? If so, in what ways? If not, why not? 1. Describe the M&E system for the HEAR project: what data are you collecting, how often, and in what ways? 1. In what capacity is your organization partnering with the HEAR project? 1. What external variables may impact the HEAR project implimentation in the Three Areas? 2. What measures being undertaken to ensure the sustainability of the capacities being developed under HEAR. What are the biggest challenges to sustainability? 2. Describe the strenths and weaknesses of CAII in managing the implimentation of the project. 2. Is the HEAR project addressing the health and education priorities of this particular community? If so, in what ways? If not, why not? 2. Where is the data stored? 2. In what ways does this partnership impact the beneficiaries of your organization's projects? 2. Are there steps project staff can take to minimize this impact? 3. Is HEAR on track to meet the deliverables of the Contract Agreement (as amended)? What factors impede this progress? 3. In what ways does the HEAR project align with the USAID/USG Sudan program and foreign policy priorities – is this activity rendering clear CPA “peace dividends?” 3. Is the HEAR project partnering effectively with other organizations in the area? If so, which ones and in what ways? 3. What is done with the data once is has been collected and stored? 3. Does this partnerhip benefit your organization? If so, how? If not, why not? 4. Describe the strenths and weaknesses of CAII in managing the implimentation of the project. 4. What recommendations do you have to improve the project? 4. Is there any information important to the project that is not or cannot be collected? If so, what is it? 4. What has been your, or to your knowledge, the organization's interations with HEAR project staff, and how frequently? 5. What are the primary external variables that hinder the project. How has the project responded? 5. How is the data analyzed and by whom? 5. What recommendations would you make to improve the partnership? Health, Education And Reconciliation (HEAR) Project 69 Mid-Term Evaluation COP COTR SSRRC M&E Specialist Supporting Partners USG Officials 6. Of the schools served by each ERC, how many are English schools? 6. Have the data been used to make program adjustments or improvements. If so, describe them. 7. How are CHP's recruited? What are required qualifications? 7. Is HEAR on track to meet the deliverables of the Contract Agreement (as amended)? What factors impede this progress? 8. Describe the strenths and weaknesses of CAII in managing the implimentation and monitoring of the project. 9. Of the schools served by each ERC, how many are English schools? Health, Education And Reconciliation (HEAR) Project 70 Mid-Term Evaluation ERC Manager ERC IT Specialist 1. How long has this ERC been open for use by the community? 1. Describe the iT resources currently available at eqach ERC. 2. What is your role? 2. What IT resources are planned for the future? 3. Describe the people who come to the center? Who are they and why do they come? 3. What are the most common uses of the IT resources by those who visit the ERC? 4. Besides training, what are the most common activities here at the ERC. 4. What are the biggest challenges to the implimentation of the IT component? 5. What resources get used the most? 5. How do people find out about the IT recources? 6. What resources get used the least? 6. What are the primary IT needs of the community served by the ERC? ERC Manager 7. Are there resources that people request or need but which are not available at the center? 8. How do people find out about the center? 9. Describe the teacher training: how are trainees recruited, who leads the training, and of those who begin it, how amny complete? 10. Describe the CHP training: how are trainees recruited, who leads the training, and of those who begin it, how amny complete? 11. What have been the biggest challenges to fully implimenting the ERC activites and resources? 12. Is the ERC meeting the educational and health needs of the community? 13. What other organizations (government, partners) are involved in the management of the ERC? 14. What recommendations do you have for improving the project? Health, Education And Reconciliation (HEAR) Project 71 Mid-Term Evaluation Appendix H Documents Reviewed USAID Contracts and Amendments with CAII CAII HEAR Proposals HEAR Annual Work Plans All Annual Reports Quarterly Reports Bi-weekly Reports Trip Reports Performance Monitoring Plan (PMP Data Quality Assessments (by USAID) Success Stories ERC Weekly and Monthly Reports ECR Computer Reference Guides Workshop Report HEAR ERC Document Collection Policy ERC Challenges and Opportunities Sudan National Health Policy, Federal Ministry of Health, 2007 Kauda Education Report, Regional Education Office Sudan National Policy Volunteer Sector, Federal Ministry of Health, 2004 Proposals for Model Schools, HEAR Fragile States Strategy, USAID, 2005 Health, Education And Reconciliation (HEAR) Project 72 Mid-Term Evaluation Appendix I Classroom Observation Tool Health, Education And Reconciliation (HEAR) Project 73 Mid-Term Evaluation Health, Education And Reconciliation (HEAR) Project 74 Mid-Term Evaluation Health, Education And Reconciliation (HEAR) Project 75 Mid-Term Evaluation Health, Education And Reconciliation (HEAR) Project 76 Mid-Term Evaluation Health, Education And Reconciliation (HEAR) Project 77 Mid-Term Evaluation Health, Education And Reconciliation (HEAR) Project 78 Mid-Term Evaluation Health, Education And Reconciliation (HEAR) Project 79 Mid-Term Evaluation Health, Education And Reconciliation (HEAR) Project 80 Mid-Term Evaluation Health, Education And Reconciliation (HEAR) Project 81 Mid-Term Evaluation Health, Education And Reconciliation (HEAR) Project 82 Mid-Term Evaluation