THE Mitchell MID-TERM EVALUATION REPORT Community Learning and Resource Center Vanga, Bandundu Province and Complementary Instructional Strategies Luozi, Bas-Congo Province June 14,2004 TABLE OF CONTENTS TABLE OF CONTENTS ...................................................................................................................................... I ACRONYMS ......................................................................................................................................................... 1 EXECUTIVE SUMMARY ................................................................................................................................... 1 1 . INTRODUCTION AND BACKGROUND ........................................................................ 3 A . Objectives of the Evaluation ................................................................................................................ 3 B . Evaluation Methodology and Report Outline ...................................................................................... 4 1 . Evaluation Methodology .................................................................................................................................. 4 Classrooni observations .......................................................................................................................................... 4 Interviews ................................................................................................................................................................... 4 Documents Review .................................................................................................................................................. 5 2 . Report Outline .............................. ... .................................. ...... ................................................................... 6 C . Project Context and Background ......................................................................................................... 6 D . Project Results Framework ............. ..... ....................................................................................... 7 11 . PROJECTS' IMPLEMENTATION PROGRESS: FINDINGS AND ANALYSIS ................................................................................................................................................. 9 Result I: Establishment of a Community Learning and Resource Center in C'anga. Bandundu Province .... 9 Result 1 . 1 - Resource Center established .................................................................................................................... 9 Result 1.2 . Resource Center operational and activities started ................................................................................ 10 Result 1.3 . Resource Center educational services ~rovided .................................................................................. 10 Result 1.4. Resource Center sustainability plan defined ...................................................................................... 17 Result 2: Complementary lns~ructional Strategies in Luozi. Bas-Congo Province - "Projet d'Appui ri la Pkdagogie Active" (P . A . P . A. ) ..................................................................................................................... 17 Result 2 .I: Learning needs assessed for CIS ........................................................................................................ 17 Result 2.2. Pilot project activities implemented ................................................................................................ 17 Result 2.3. Project sustainability ........................................................................................................................... 19 111 . IMPACT OF THE PROJECTS ............................................................................................... 20 A . Vanga ....................................................................................................................................................... 20 I . Observable changes .............................................................................................................................................. 20 I . I Teachers'directors' beliefs and teaching principles ...................................................................................... 20 1.2 Teachers' competencies in innovative instructional strategies ...................................................................... 20 1.3 Services to the Community ........................................................................................................................... 20 2 . Problem areas ....................... ................................................................................................................... 21 2.1 Teachers' competencies in innovative instructional strategies .................................................................... 21 2.2 Services to the Commrmity .................... . ............................... ....... ............................................ 21 2.3 Institutional capacity building and sustainability .......................................................................................... 22 B . Luozi .................................................................................................................................................... 22 1.1Teachers/directors' beliefs and teaching principles ................. ......... ................................................... 22 1.2 Teachers' competencies ................................................................................................................................ 22 1.3 Institutional capacity building and sustainability ......................................................................................... 23 IV . LESSONS LEARNED ...................................................................................................... 25 A . For EDC in colfaboration with AED .................................................................................................. 26 I . For Teacher Competencies in innovative instructional Strategies ........................................................................ 26 2 . At the institutional capacity building level ............................................. ..................................................... 26 2.1 Vanga ..................................................... ............................................................................................ 26 2.2 Luozi .......................................................................................................................................................... 27 B . For USAID .......................................................................................................................................... 27 ANNEXES ............................................................................................................................................................ 28 ACRONYMS AED: ANAPECO: CIS: CLRC: Diagrammes (H, F, T) : DRC: EDC: F: IFESH: M: M&E: MAF: PAPA: SANRU: SOW: T: TAC: TMG: USAID: Academy of Educational Development Association Nationale des Parents d'Eleves du Congo Complementary Instructional Strategies Community Learning and Resource Center Diagrammes (Hommes, Femmes, Total) Democratic Republic of Congo Education Development Center Female International Foundation for Education and Self￾Help Male Monitoring and Evaluation Mission Aviation Fellowship Projet d'Appui a la Pedagogie Active Rural Health (Sante Rurale) Project Scope of Work Total Telecentre d' Apprentissage Communautaire The Mitchell Group United States Agency for International Development EXECUTIVE SUMMARY USAIDIDRC (Democratic Republic of Congo) has decided to establish critical pilot projects in the basic education sector, which has suffered severely from decades of corruption, economic decline and civil war. The projects will provide information for USAID itself, the DRC government, and other partners about what is realistically feasible and potentially successful in the DRC, and will lay the foundation for further USAID i~wolvement in the education sector. The following report presents the findings of the mid-term evaluation of two initial small￾scale education pilot projects, the Vanga Community Learning and Resource Center (hereafter "the Center") and the Luozi Complementary Instructional Strategy Project. Funded by USAIDIDRC, both projects are being implemented by Education Development Center (EDCeworking alone, in the case of the Luozi project, and in collaboration with the Academy for Educational Development (AED), in the case of Vanga. The projects are using different approaches and strategies to arrive at the same objective, i.e., to improve the quality of basic education and address a number of challenges facing the education sector. The field trip for this mid-term evaluation was carried out from April 13 to May 1, 2004. After a brief description of the context and the nature of the results of the project, the present report focuses on the accomplishments of the projects in light of the stated expected results and lessons learned in terms of strategies, impact, level of the communities' sense of ownership, and sustainability of the projects. The findings show that: After less than a year of actual implementation, the two projects have the potential for achieving their objectives, and most activities are being implemented successfully. There are changes taking place in the field as far as classroom methodology is concerned. Teachers and the community at large are fully motivated and see the benefits of the projects. Even if there are no tangible results as far as students' achievements are concerned, the fact that there is a change in the teaching methodology may lead to a positive impact on school results in the future. There is a need, however, to capitalize on the gains and to make improvements in some weak areas, especially by focusing on in-depth training in the new methodology and on institutional capacity building at every level of the projects. The following recommendations concentrate on the problem areas and offer suggestions for the expansion of activities under both projects. For Teacher Competencies in innovative instructional Strategies There is a need to concentrate more on helping the teacher trainees acquire the new competencies. More in-depth training workshops are needed in ways to integrate didactic materials into the instructional process and to create more materials for subjects such as mathematics and French. Most important, however, is additional in-depth training in the use of the Problem-Based Learning (PBL) approach. In the same vein, teachers will need to give careful, creative attention to the National Curriculum so that they can easily integrate the macro-level objectives into their more specific lesson objectives. In the case of Vanga, closer supervision is needed. It may be necessary to have EDCIKinshasa education specialists monitor the training activities in Vanga more regularly. For the training to achieve sustained impact, the project should capitalize on the very important follow-up mechanism represented by school directors, pedagogical advisors and inspectors by offering these three groups more in-depth training in pedagogy and in supervisory skills. The effects of the training that the teachers have received in active pedagogy cannot be sustained if there are no supportive instructional materials, such as reference books or textbooks, for teachers to use as a basis for their day-to-day teaching. At the institutional capacity-building level The Evaluation Team strongly recommends that actions be taken to strengthen the leadership and the team spirit at the Center. Furthermore, more work is needed with the Steering and the Management Committees for their empowerment. Luozi There is an urgent need to start working with the Steering Committee, as was done in August 2003 with the Vanga Steering and Management Committees, to define roles and responsibilities and to explain the nature of involvement expected from committee members. In addition, further support, including transportation and data recording and processing machines, should be provided at the local level of the educational system (Sous-Proved and inspectorates) to enable them to fully participate in the implementation of the project. The sustainability plan should be implemented as soon as possible, using a participatory approach. The scope of work of the present evaluation stated that some recommendations should be made for the expansion of the projects' activities. However, the Evaluation Team believes that more time should be given to the two pilot projects so that their impact can be seen and evaluated. To be able to draw replicable lessons and because the overall strategy targets the educational system in the selected areas, it is necessary to conduct a full impact evaluation after another school year. In the meantime, the Team recommends that future projects combine the Vanga model (a community resource center with a training staff) and the Luozi model. The Team strongly recommends that the trainers be chosen mostly among inspectors and pedagogical advisors. The Evaluation Team emphasizes the need to take a more community development-based approach at the start of any future project. Even though there are successes in the two projects, more could have been achieved if the implementers had used a more participatory approach from the beginning. I. Introduction and Background Under Contract No. HNE-1-00-00-00078-00 and followi~~g USAID'S Task Order No. 806, The Mitchell Group (TMG) requested the services of two consultants, Ayele Adubra and Bienvenu Marcos, to carry out a midterm evaluation of two pilot projects funded by the U.S. Agency for International Development (US AID) in the Democratic Republic of Congo (DRC). The projects are located in Vanga (Bandundu Province) and Luozi (Bas-Congo Province). The evaluation was conducted during the period April 14 to May 1, 2004. The term of reference of this mission are attached to the present report (Appendix A). It should be noted that before this evaluation mission, TMG had conducted several trips to the DRC in connection with the projects since November 2002: November 2002: Launching of the Vanga pilot project and drafting of the first Monitoring and Evaluation Plan for Vanga. April 2003: Luozi pilot project start-up and drafting of the first Monitoring and Evaluation Plan for Luozi. July 2003: Interim assessment of the Vanga pilot project, participation in the Center's inauguration, and Phase One training in M&E methodology in Kinshasa for EDCIKinshasa staff and the Luozi Project Coordinator and in Vanga for the Center's staff. November 2003: Finalization of M&E plans for the Vanga and Luozi pilot projects and Phase Two training in M&E methodology. March 2004: Monitoring trip to Vanga to monitor progress made in the project's implementation and conduct work sessions with the Center's staff on the efficient use of the M&E indicators and tools. A. Purpose and Objectives of the Evaluation The overall purpose of the evaluation, as agreed upon with USAIDIDRC, was to assess progress made in the implementation of the two projects in the context of the expected results set out in the contract with EDC (in collaboration with AED). The objectives of the evaluation are threefold (see Appendix A): Evaluate implementation progress to date for the two pilot projects; Assess the impact of the education activities on student participation, teacher participation, quality of teaching, learning and access to educational materials; and Make recommendations on the adjustments needed for future phaseslprojects, on how to expand the current project to reach many more recipients, and on how to ensure sustainability. The mid-term evaluation focuses on changes taking place in classroom methodology after the training of teachers in Luozi (October 2003 and February 2004) and the training of the Vanga Center training staff (July 2003 and February 2004). This particular aspect is all the more important as the findings will enable EDC to (a) tailor the remaining actions to fill the gaps in teachers' skills acquisition, and (b) build a complete replicable model to be implemented in the extension/expansion phase of the project. Prior to the mid-tenn evaluation exercise, the TMG Evaluation Team (hereafter "the Team") had developed a conceptual framework to guide the analysis of the overall impact of the project. Although the framework is intended mainly as a guide in conducting the end-of￾project evaluation, the Team believes that it is worthwhile to describe what is meant by "impact" and "effectiveness" in general. It is also important to mention that the framework was shared with EDC (Washington) prior to TMG's November 2003 field visit and with EDCIKinshasa during the M&E workshop in November 2003. The M&E plans of both projects incorporate the conceptual framework. The effectiveness and impact evaluation will: Assess the relevance of the projects' activities in achieving the goals and objectives of the projects as defined by USAID and EDC. Assess the relevance of the projects' activities to the needs and culture of the beneficiaries and stakeholders. Evaluate the scope of the projects' activities both in relation to the targets set by project implementers and also in relation to the size of the population being targeted. Assess the degree of change brought by the projects' activities vis-a-vis not only behaviors and attitudes but also achievements and results. Assess the degree of sustainability of the project. B. Evaluation Methodology and Report Outline The evaluation was conducted from April 14 to May 1, 2004. The collection of data used a combination of techniques: observations, interviews and document review. 1. Evaluation Methodology Classroom observations Classroom visits were conducted by four observers (Elodie Musafiri and Noelle Wishi from EDCIKinshasa and Bienvenu Marcos and Ayele Adubra from TMG). Each teacher was observed in two lessons by two different observers (one from EDC and one from TMG). In total, 18 teachers in Luozi and 15 in Vanga were observed. In order to assess the impact of the training, the Team selected a control group made up of teachers from three untargeted schools. Due to accessibility and time constraints, the Team used two criteria to identify the control group: first, the schools had to be within the territoire of Luozi but far enough from project schools to reduce the risk of contamination by the training; and second, all classes targeted by the project had to be represented. A separate report will detail the findings of the classroom observations. This evaluation report will only present the major findings on changes taking place in the classroom. Interviews The Team conducted interviews with most of the stakeholders and partners of the project (USAID, EDCIKinshasa, Education sector partners, Steering Committees, Management Committee, project staff, etc.) and with the direct beneficiaries of the projects (e.g., teachers, students, healthcare personnel). These interviews were sometimes conducted one-on-one or in focus groups using questionnaires for guidance (Appendix B). A complete list of key informants can be found in Appendix F. Interview schedules and guides were developed for each category or type of participants, grouped as follows: Category One: Organizations or individuals with direct or indirect responsibilities for implementing the project (USAID, EDC, Vanga Center staff, Luozi Project coordinator); Category Two: Organizations or individuals involved and/or representing the project's beneficiaries (education authorities, Steering Committees, Vanga Management Committee); Category Three: Direct beneficiaries (teachers, school directors, pedagogical advisors, inspectors); Category Four: Center users (students, youth, parents' committees, women's associations). The questions asked of the participants were designed to collect information on both the process and the results each project. Each individual or group of individuals interviewed was asked (see questionnaires, Appendix B): To specify their role, responsibility and/or contribution to the implementation of the project. To make a brief statement or evaluation, as of the interview date, concerning the implementation of the project, in terms of both process and achievements. To indicate areas of satisfaction, problems encountered, lessons learned, and best practices. To suggest measures to be taken in order to ensure the further development of the projects' activities and the achievements of their objectives. To make suggestions or recommendations for the future, including expansion and replicability. Furthermore, each participant was offered the opportunity to make additional comments or recommendations. Documents Review The Team reviewed various project documents, lesson plans, school diaries (journaux de classe), and other relevant material. The list of the documents studied appears in Appendix G. The list includes the following reference documents: "Te'le'centre d'Apprentissage Comrnunautaire (TAC) et Strate'gies e'ducatives complkmentaires - Program Description," June 2002. "Work Plan Revised for No-Cost Extension," drafted and distributed by EDC on June 6,2003. "Basic Education Strengthening for Teachers with Complementary Instructional Strategies and Appropriate Technologies - Cost Extension Amendment # 1: Program Description." "Community Learning Resource Center in Vanga, Bandundu Province of the DRC: Monitoring and Evaluation Plan Updated by TMG in Collaboration with EDCIWashington, AEDIWashington, EDCNouthLearn, EDCIKinshasa and Vanga Center Staff," January 2004. "Luozi Complementary Instructional Strategies: Monitoring and Evaluation Plan Updated by TMG in Collaboration with EDCIWashington, EDCIYouthLearn, EDCIKinshasa and Luozi Project Coordinator," January 2004. "Scope of Work for the Mitchell Group: Evaluation of EDCIAED Education Program in DRC," April 2004 (see Appendix A). 2. Report Outline After a brief description of the context and the nature of the projects' results, the present report focuses on: The accomplishments of the projects in the light of the stated expected results, strategic objectives and overall goal, as specified in the Cooperative Agreement in the session entitled "Program Description" and the subsequent amendments. Lessons learned following the analysis of the results in terms of strategies, impact, level of the communities' sense of ownership and sustainability of the projects. Finally, conclusions and recommendations are made for (i) the future of the project activities with the view of achieving the fundamental objectives, and (ii) for future actions in order to expand the reach of USAIDIDRC basic education project. C. Project Context and Background Under its strategic objective "Promote Health, Food Security and Transition to Peace," USAID is addressing a number of the population's needs in the DRC through improving food security and livelihood, enhancing child and maternal health in targeted zones, and strengthening civil society and promoting justice. USAIDIDRC sees education as the underpinning of all the various program components and recognizes that without a literate and educated society, many of the-efforts made to improve the Congolese people's chances of survival and minimal quality of life will be wasted. Since the early 1990s, two major factors have characterized the Congolese education system. First, its drastic and ongoing pauperization due to the consequences of economic collapse and internal civil war; second, the population's strong commitment to and demand for education. The supply and demand for education comes from parents. Parents' committees, part of a national network of parents known as I'Association Nationale des Parents d'Eleves du Congo (ANAPECO), manage schools in collaboration with school principals and are overseen, at a distance, by religious authorities. Because support is no longer available for basic education needs, including school supplies, written materials and teachers' salaries, the quality of teaching and learning has plunged. Stories of children and adults returning to ilIiteracy abound. Children who are rarely exposed to printed materials cannot be expected to attain functional literacy. The exceptions are found in the private schools that are supported by high-income parents in the main cities. In response, USAID/DRC has decided to intervene in the basic education sector, with critical pilot projects that will provide information for USAlD itself, the DRC government and other partners about what is realistically feasible and potentially successful in the DRC, to lay the foundation for further USAID involvement in education. USAID has made a strategic decision that the geographic areas for education interventions will coincide for the most part with the current health zones where the USAID-funded Rural Health (Sante Rurale, or SANRU) project is located. To date, two pilot projects are being implemented in the Democratic Republic of Congo. The first one is located in Vanga, Bandundu Province, and the second one is in Luozi, Bas-Congo Province. These initial small-scale education pilot projects are using different approaches an I( strategies to reach the same objective, i.e., to improve the quality of education by addressing a number of issues facing the education sector, including: The capacity of commui~ities to manage schools effectively and efficiently. The ability of parents' committees to advocate for and manage change. The ability of principals and teachers to lead and teach effectively. The need for material inputs to support quality teaching and learning. In November 2002, a joint technical team from Education Development Center (EDC) and Academy for Educational Development (AED), with the support of The Mitchell Group Inc. (TMG), designed and drafted the first activities of a "Community Learning and Resource Center," the pilot project being implemented in Vanga, Bandundu Province. This project is based on the use of information and communication technologies to support education. In January 2003, EDC conducted an assessment of learning needs in the DRC to determine the type of complementary instructional strategies that will best support selected basic education efforts in the country and how different technologies can play a role in implementing those strategies. This assessment led to the choice of Luozi for the implementation of a second pilot project. It was launched in April 2003 with the visit of EDG project director . TMG is specifically responsible for the monitoring and evaluation systems in the two pilot projects. The M&E plans have been discussed and validated using a participatory approach with inputs from all the stakeholders (EDC, AED, Steering and Management Committees, Center staff and USAID). D. Project Results Framework As stated in the Program Description and the EDC's current work plan,1 the expected pilot results from the two pilot projects are the following: Vanga pilot project: "Establishment of a community learning and resource center" 1. Resource Center established. 2. Resource Center operational and activities started. 3. Resource Center educational services provided. 4. Resource Center sustainability plan defined. 5. Resource Center Monitoring and Evaluation Plan defined and implemented (TMG). Luozi pilot project: "Assess and pilot complementary instructional interventions (CIS) to support classroom instruction in selected thematic areas and grade levels of the national curricuIum" 1. Learning needs assessed for CIS. 2. Pilot project activities implemented. 3. Project sustainability plan defined. 4. Pilot M&E defined and implemented (TMG). See the document entitled "dot-EDUIDRC - Work Plan Revised for No-Cost Extension,'' page 13 (draft distributed on June 6, 2003) These results aim at contributing to USAID's education strategic objective for the DRC of "improving basic education for children, and especially for girls, in targeted areas in the DRC." The results should also support USAID's existing programs, particularly in rural health (SANRU), and complement other partners7 education programs (UNICEF, Cooperation Belge, the World Bank) by focusing on the learning process while other partners are focusing more on the deployment of significant inputs, such as school infrastructure, school furniture, curriculum development, textbooks, and other printed learning and teaching materials. 11. Progress of Project Implementation: Findings and Analysis Progress in implementing the two projects was evaluated using the monitoring and evaluation plans updated by TMG in November 2003 and in collaboration with the main stakeholders and partners of the projects. Result 1: Establishment of a Community Learning and Resource Center in Vanga, Bandundu Province Using the performance indicators identified for each activity, the review covered a six-month period (August 2003-January 2004). From the beginning, it was noted that the majority of the activities were being implemented. It should also be stated that some activities appear difficult to evaluate due to lack of clarity in the data. in some cases, the collection of data has not been systematically integrated into the overall M&E plan at either the quantitative or the qualitative level. in particular, the monthly reports from the Center and from EDCIKinshasa are still too general and do not help in clearly assessing the status of the performance indicators. In addition, some statistical data seem to be erroneous, and at times the statistics or the reports lack coherence. But 011 the whole, analysis of the documents, triangulated with interviews and observations, have helped in rendering a true picture of the projects. The table entitled "Mid-term Evaluation of the Vanga and Luozi Pilot Projects - Progress of Project implementation: Results Framework" (Appendix C) presents the status of each indicator for the above-mentioned period. In addition, the text incorporates some graphs for an easy-to-follow representation of progress made. The evaluation of the expected results has led to the following observations: Result 1.1: Resource Center established The interim evaluation conducted by TMG in July 2003 noted that the Community and Resource Learning Center had been inaugurated by U.S. Ambassador Aubrey Hooks in the presence of USAID Country Director Anthony Gambino and other dignitaries. At present, the Center comprises: A refurbished facility housing 17 desktop computers, 2 notebook computers, 1 laser printer, 1 color printer, 2 digital cameras, 8 inverters, 1 projector, 1 video recorder, 1 TV set, 1 DVD player and consumables ready for use. A satellite ground station installed and configured to connect to the Internet. it is important to highlight the fact that the issue of an independent source of electricity for the Center was a major concern at the time of the interim evaluation, is still unresolved, and weighs heavily on the conduct of activities at the Center. At the time of this evaluation, the Center could only get electricity for an average of three hours a day (6 pm to 9 pm) with an additional two hours on odd weekdays (9 am to 1 1 am). However, EDC's Chief of Party has informed the Evaluation Team that solar panels were being bought and their installation would allow the Center to have an estimated eight hours of electricity per day. Result 1.2 - Resource Center operational and activities started In general, the status of the indicators under this result shows good performance. 1. After the hiring and initial training of 10 members of the Center's staff (Interim Evaluation, July 2003), AED and EDC provided further training in the following subjects: active pedagogy, institutional organizations, project management and conflict resolution. Active pedagogy: YouthLearn developed 10 training modules and designed 10 additional online modules. In addition to the July 2003 training session which it organized, YouthLearn organized another session in February 2004 with the help of the EDCIKinshasa education and didactic material specialists. IFESH delivered training sessions on institutional organizations, project management and conflict resolution. Concurrently with the IFESH training, the EDCIKinshasa accountant and staff assistant conducted a training session on financial management (financial process and accounting). 2. The Steering and Management Committees members received the same IFESH training (October and November 2003). Despite these accomplishments, it should be noted that: The training modules developed by YouthLearn are still not available at the Center to serve as reference materials for the Center staff, teachers and directors who received the training. Nine months after the inauguration of the Center and two months before the scheduled end of the project (June 30th, 2004), the Center's operations manual is still not available. According to EDC and AED, it is still in the making. As a consequence, operations at the Center are carried out without clearly defined rules or regulations. This lack of institutional structure accounts in part for the weaknesses observed in the Center's leadership, team spirit, distribution of roles, and work atmosphere. The Team received a number of spontaneous testimonies from the Center staff and the Steering and Management Committees, as well as the Center's users, confirming that the atmosphere at the Center has not been conducive to productive work. This situation, if not addressed, constitutes a serious threat to the operations and sustainability of the Center. Result 1.3: Resource Center educational services provided This result is divided into five intermediate results: Intermediate Result 1.3.1 : Training of teachers and school administrators in active pedagogy. Intermediate Result 1.3.2: Educational services provided to the community. Intermediate Result 1.3.3: Training in and use of information and communication technologies. Intermediate Result 1.3.4: Educational and cultural activities held in partnership with other organizations. Intmnediate Result 1.3.5: Community outreach. Intermedinte Result 1.3.1: Trnining of tenclzers nnd sclzool ndministrntors in nctive pedfll?ogy. Activities that have been implemented under this intermediate result are: 20 trainees (15 teachers and five directors) selected from five schools have received training in active pedagogy and in the use of didactic materials. Classroom visits conducted to monitor how the trainees apply active pedagogy or create and use didactic materials to support the National Curriculum. On average, each teacher has received four visits since October 2003. Each observation is recorded and a report written incorporating comments made after the observation. Then follows a feedback and discussion session at the Center with all the teachers involved in the project. Center training staff organizes regular microteaching lessons based on active pedagogy. These lessons are followed by discussions during which teachers share their experiences. Other very important activities have yet to start: The training in active pedagogy of inspectors, pedagogical advisors and all the teachers from the targeted schools. The monthly discussions on the implementation of the new methodology to be held with inspectors, advisors and school directors. According to EDCIKinshasa, the delay in these activities is due to financial and logistical problems (remoteness of the inspectors from Vanga, difficulty in reaching them, and lack of accommodations in Vanga for conducting training). As far as training in didactic materials is concerned, it should be noted that teachers in Vanga have not received the kits given to their colleagues in Luozi. The lack of that support constitutes a serious limitation on the quality and the quantity of didactic materials created during and after training. In addition, unlike Luozi, the training in the creation and use of didactic materials was given by Center training staff who had not received any specific training in that domain. On the whole, the classroom observations conducted during the evaluation could not effectively evaluate the teachers' competence in the creation and use of didactic materials, as most of the lessons presented (in subjects such as grammar, history and vocabulary or spelling) did not incorporate the use of didactic materials. The selected teachers and directors have also been trained in the use of the computer, especially Encarta educational software. They stated that they are able to research and print didactic materials for their own lessons. On average, 17 teachers (from targeted and non￾targeted schools) request the assistance of the Center training staff each week in finding support materials for their teaching. However, the teachers and directors who are part of the project state that they have not been trained yet in using the Internet for pedagogical research. Finally, it should be noted that the selected teachers' attendance at the Center's library is rather low. This is due to the lack of textbooks and other didactic publications and resource materials there. Intermediate Result 1.3.2: Educational services provided to the community Educational services provided to the community have been varied. They range from classes for women in literacy and numeracy to the display of information and the organization of conferences, debates and computer training. The evaluation of these activities in the light of the performance indicators led to the following conclusions: Overall, there has been a slowdown and at times a suspension of several activities since November 2003. The reason behind this low performance can be attributed to (a) TMG and IFESH trainings delivered to the Center staff during the month of November; (b) seasonal agricultural activities; and above all, (c) Internet network failure. The number of women visiting the Center fluctuates from one month to the other. Here again, there was a decrease of 70% between September and November. However, there was a slight increase in December. The following chart shows the users' attendance at the Center by month and by sex. The slowdown is especially noticeable for the women, who were mostly busy with seasonal agricultural activities. Center's Attendance At~g. Sept. Od. Nov. Dec. Jan. 2003 2003 2003 2003 2003 2004 Month It should be noted that among the women attending weekly literacy and numeracy classes at the Center, there are three groups whose participation has increased steadily since October 2003. They include the "Groupe des Mamans de la Mission," "Groupe des Mamans de Vanga Village" and "Groupe des Mamans de Kimbulu." Literacy and Nutneracy P,ug Ser:h. Qb. No:!. kc. Jan. 2003 2003 2[1[13 ZrJO- 2003 21504 Month The adult classes for women covered such varied topics as preventing accidents in the home, protecting children, personal hygiene, immunization, cooking recipes, resolving marital conflicts, menopause, and more. Aug. Sept. Oct. Nov. Dec Jan. 2003 2003 2003 2003 2il03 2004 Month Likewise, the information displayed on the board at the Center covers a wide variety of subjects: health, community development, politics, and national and international news. The conferences and debates organized regularly by the Center have also covered diverse themes, such as growing cassava, the need to eat well, diseases linked to good manners, and the history of the Democratic Republic of Congo. These themes are pertinent to the objectives assigned to the Center. The attendance among teachers and students is substantial. However, women's presence, which averages about 30% at such events, is irregular and at times very minimal. Conferences and Debates Aug. Sept. Od. Nov. Dec. Jan. 2003 2003 2003 2003 2003 2004 Month Stories, proverbs and parlottes are featured regularly by the Center. Among those attending are a good number of students, as the following chart shows. Note that attendance among young women was superior overall to that of young men and that overall attendance grew sharply in January. I Tales, Prmrbs and "Parlottes" Aug Sepl Od. Nov Dcc. Jan. 20U3 2003 2003 2003 2003 2004 Month The Center delivers training in basic software (MS Word, MS Excel) for the different categories of potential users in the community. But this activity confronts a number of difficulties: (a) users are, in general, unable to pay the training fees (this is particularly true of teachers and students); (b) there is 110 structured training plan and accompanying syllabus to be given to the users to facilitate training and assimilation; and (c) the training methodology doesn't give enough stimulation/encouragement. Here again, one can observe a decrease in the number of users between October and December 2003, followed by a rebound in January Training in Computer's use Aug. Sept. 03. Nov. kc. Jan. 2003 2003 2003 2003 2003 2004 Month = The library resources are evidently scarce, comprising mainly cartoons and political and sports magazines. Students appear to be the most regular users of the library. By contrast, teachers' use is minimal, accounting for barely 10% of the total. The explanation for this is the lack of reference and didactic books in the section on documentation. Libraws general attendance Aug. Sept. Oct. Nov. kc. Jan 2003 2003 2003 2003 2003 20114 Month I Library atlenclance by trained teachers I - Aug. Sept. OcZ. NW. Usc. Jan. 200'3 2003 2003 2003 2003 2004 I Month Intermediate Resdt 1.3.3: Training in and use of information and communication teclrnoIogies The objectives targeted by activities under the above intermediate result appear to be far from being achieved. In fact, it can be noted that: The demand for training on Internet use and search exceeds the current supply at the Center. This is due to the fact that the Center can only accommodate a very limited number of users at a time and also because of the training fees, which many members of the community (especially teachers, students and health workers) have judged too costly. The performance under this result is irregular, rising in September, falling and hitting a low point in November 2003, then peaking in December 2003 before dropping again in January 2004. I Training on Internet use C &pt Od. Now Dec. Jan. 2003 2005 2003 2003 2004 Month Regarding Internet subscriptions, at the time of the evaluation only the missionaries and the expatriates working at MAF benefited from a connection to the network installed at the Center. The evaluation shows that there is a demand for the service, but the server capacity is limited. The following table shows a drastic decline in electronic consumer services provided by the Center for the period under evaluation, in particular between October and December 2003. The reason is the network failure, caused by a breakdown in the access point over a long period of time. In addition, the total income of the Center is insufficient to cover its operating expenses, which are estimated at $4,500 per month. In analyzing the sources of income, it appears that e-mail services are the main sources, ranging from 50 to 78% of the total income, followed by the printing and copying services (about I0 to 20 %). Income generated by the Center [in Congolese Francs) Aug. Sew. Od. Nov. Pec. Jan. 2003 21333 2003 2003 2003 2004 Month Only one video-forum session has been held since the Center opened. It was organized for the "mamans" or the women and students. The theme was the relationship between parents and students. Although the event was reported to be a success, there are no data to substantiate this report or permit evaluation of this activity. The figures for video-reporting have also been irregular since the Center's opening. The average was three events monthly between September and December. The events being reported were mostly groups or family life events, such as Bible campaigns, ministers' consecrations, weddings, and so on. They do not embrace the community development goals of the Center. Intermediate Result 1.3.4: Educational and culturnl activities held in partnership with other organizations After six months of operations, no activity has been conducted to accomplish this intermediate result. Intermediate Result 1.3.5: Community outreach Interesting outreach activities have been held that target the different categories of community members in Vanga and its surrounding areas (students, teachers, health and administration personnel, and others). Students have particularly benefited from the outreach activities that focused on them. Different techniques have been used for advertising the Center and its services. They range from mass campaigns, exhibitions and posters to door-to-door calls and talking to individuals. Discussions, interviews, video-forums, and brochures have also supported the advertising campaigns. The following chart shows a drastic reduction in the number of people reached during October and November 2003. However, one should also note a slight increase for December 2003- January 2004. I I Community outreach activities Aug. Sept. Oct Nuv. Dec. Jan. 2003 2003 2003 2003 2003 2004 Month I Result 1.4: Resource Center sustainability plan defined The activity to be conducted under this intermediate result is in progress. The first draft of the sustainability plan prepared by AED was presented to EDC and TMG in March 2003. However, the participatory approach recommended for this activity has not been used fully. The Steering and Management Committees, the Center's staff and other stakeholders have not yet been associated with the process. During the evaluation discussions held with the two committees' members, they confirmed their interest in and support for the quick formulation and implementation of such a plan. Result 2: Complementary Instructional Strategies in Luozi, Bas-Congo Province - "Projet d'Appui a la PCdagogie Active" (P.A.P.A.) Result 2.1: Learning needs assessed for CIS The activities under this intermediate result have been completed. The needs assessment was conducted idanuary 2003 and the findings presented to USAlD in April 2003. Result 2.2: Pilot project activities implemented The evaluation of the activities under this intermediate result in the light of identified performance indicators shows a satisfactory level of completion. Regarding the assessment of changes in classroom methodology, a separate and more exhaustive report will detail the findings. In the meantime, the following activities have been completed: The drafting and submission to USAlD of the project's work plan and implementation plan. The hiring of the project's coordinator in April 2003 and of his assistant in March 2004. The organization of information sessions in April 2003011 track: J To inform local authorities of the education sector in Luozi (Sous-Proved, inspectors, pedagogical advisors, school directors) about the outcomes of the January needs assessment and the objectives and strategies of the project, and J To discuss the vision, mission and objectives of the project with the representatives and other partners in the community. A Steering Committee and three technical sub-committees (training, didactic materials and radio) have been established for the supervision and follow-up of the project activities. YouthLearn has developed a training plan and 10 training modules on active pedagogy to train teachers and administrative personnel targeted by the project. Six inspectors, four pedagogical advisors and 10 school directors were trained during two two-day sessions, in September 2003 and February 2004. Twenty teachers, including nine women, were trained in September 2003 and February 2004 during two six-day sessions based on the 10 modules developed by YouthLearn. A third session is scheduled for May 2004 to prepare them for the next phase of the project, in which they will train their fellow teachers. An inventory of local opportunities and resources to be used in active pedagogy and based on local development goals was conducted with the help of the trainee teachers during the September 2003 session. Every inspector, pedagogical advisor, school director and teacher targeted by the project received a copy of the National Curriculum to support planning and classroom activities. EDC and Radio Ntemo have signed a letter of intent and three-month contract. This has led to the kick-off of community radio activities, as outlined in the project description, starting in April 2004. The feasibility study for the extension of radio coverage for the Luozi sector schools was conducted in August 2003. With the technical support of a consultant hired by EDC, five technicians for Radio Ntemo and two selected teachers were trained in the production of educational programs and shows in March 2004. As of April 22, 2004, the following radio programs have been produced: J Two 15-minute programs in French and in Kikongo on themes such as the need for girls to go to school and the importance of using didactic materials in classroom instruction. J Four educational microprograms of three minutes each on topics such as active pedagogy, girls' schooling, and the importance of using didactic materials in instruction. Radio Ntemo began broadcasting educational programs twice a week starting in April 2004, with one program being a "re-broadcast." A radio set was given out to each targeted school. An agreement was also signed in April 2004 with Radio Vuvu Kieto, a Catholic radio station based in Mbanza Ngungu, to relay the same programs. There are problem areas in the implementation of some of these activities: The training provided to inspectors and pedagogical advisors under EDCNouthLearn's training plan does not fully take their mission into account. These professionals are expected to oversee the implementation of the new methodology. As such, they require not only in-depth training in the methodology but also monitoring and supervision skills. The training modules are not available for the trainees to refer to for refreshers. The conclusions of the feasibility study have not been taken into consideration as yet. As of the time of the evaluation, only 70% of the Luozi schools were receiving Radio Nterno broadcast programs (three schools out of ten cannot receive them for technical reasons. The project implementers have, however, designed an alternative: the recording of the programs on tapes to be given to the remote schools for group listening and discussions. Teachers complain about broadcasting hours because they are not suitable for professional reasons. In addition, the single radio set given to the school is kept by the school director, rendering it difficult to access. Result 2.3: Project sustainability So far, no specific strategy for sustainability has been identified by EDC and the other organizations in charge of implementing the project. Roles and responsibilities are not yet clearly defined for the members of the Steering Committee and the sub-committees. The Steering Committee convenes at least once a month. However, the attendance rate is only 54%. The technical sub￾committee meetings are not regular. The only sub-committee which has met so far is the training sub-committee, and it met only in preparation for and at the end of the two training sessions. Most Steering Committee members indicate that they do not know what their responsibilities within the project are. The findings on implementation progress, as they have been outlined above for both the Vanga and the Luozi Pilot Projects, lead to the following comments on the impact of the two projects. 111. Impact of the projects The Evaluation Team believes that it is too soon to assess the real impact of the projects, as it has been less than a year since implementation activities began. The Luozi project effectively started with the first training workshop in October, and the Vanga project, after its inauguration in July. However, at this time in the life cycle of the two projects, patterns are emerging that indicate both changes and problem areas. These changes, if they are sustained, might turn into observable impacts. The problem areas, on the other hand, might hinder impact if nothing is done to address them. A. Vanga Since its inauguration, the Vanga Community Resource and Learning Center has come a long way. A good many of the issues raised by the July 2003 interim evaluation report have been partially resolved. These issues involved the kind of activities to be implemented and ways to gather support in order to ensure the project's ownership and sustainability. As has been detailed above, there are now visible and very positive outcomes to be seen in the Vanga Project. The Center staff, after receiving basic computer training and two training sessions with YouthLearn in active pedagogy, implemented a series of actions that are now yielding fruit. 1. Observable changes 1.1 Teachers'/school directors' beliefs and teaching principles The training of teachers and school directors in the targeted schools, together with related follow-up activities, have brought about almost the same changes in teachers' beliefs and teaching principles as the ones in the Luozi Project. There is the same awareness of the need to shift from an all teacher-centered approach to a more active, learner-centered one, based on the use of didactic materials. 1.2 Teachers' competencies in innovative instructional strategies Teachers reported the same newly acquired skills as in the Luozi Project, with a strong focus on creating didactic materials to support their teaching, using the Center's facilities for Internet search or Encarta. The targeted schools display a lot of newly created didactic materials either in the classrooms or in the school director's office, which serves as a "resource room." The teachers of the targeted schools know how to solicit expertise and resources from the Center-and ofien do so-both to build on their lessons by organizing regular students' visits and to find ways to use the National Curriculum for their teaching. 1.3 Services to the Community Records at the Center and observations show that the Center has become a reality in the life of the cominunity and has started to shape new habits in the use of information and communication technology. People use the Internet for various purposes (business, medical research or school work), as well as the photocopying and printing services. Others come to take word or data processing courses. There is a confirmed interest in the community, especially among the young, in using the resources of the library. The Center staff has also successfully organized events--debates, conferences and storytelling-which bring the community together. The Team's interviews with the different categories of users (young people, students and civil servants) show they appreciate all these activities. As far as women are concerned, groups of women from Vanga Mission, Vanga Village and Kimbulu (a neighboring village) come regularly to the Center for literacy and numeracy programs. The interviews conducted during the evaluation show that they, too, appreciate the opportunity given to them. After nine months of existence, the Center has become an integral part of the Vanga Community and of its surrounding area, as reports show that people travel as far as 100 km to use its facilities. However, the evaluation findings highlight some problem areas. 2. Problem areas 2.1 Teachers' competencies in innovative instructional strategies Although all the teachers reported awareness and adoption of active pedagogy, the classroom observations show that fewer teachers used didactic materials compared to Luozi. In general, they are less effective in the use of the didactic materials, and their classes still display more traditional instructional strategies, such as negative reinforcement and rote learning. In addition, the same weaknesses (detailed below) found in Luozi-problems with integrating the use of didactic materials in all the subjects and with fully adopting a Problem-Based Learning approach-are valid in Vanga. Despite the Center training staffs efforts to train teachers in active pedagogy and to provide follow-up training via school visits and meetings with the teachers, a more grounded approach is needed so that the new habits can be internalized. The fact is that training in active pedagogy and innovative instructional strategies in Vanga seems to be less structured than in Luozi. In addition, the training receives only remote supervision, since YouthLearn and EDC/Kinshasa educational specialists are not present. 2.2 Services to the Community As stated earlier in the report, technical and managerial problems affect the smooth running of the Vanga Community and Resource Learning Center. The Center cannot operate at full capacity because it lacks an independent source of electricity. Until now, because it is dependent on the Vanga Hospital's source of electricity, the Center can be open only for three hours per day, with an additional two hours on Mondays, Wednesdays and Fridays. In addition, since January 2004, access to the Internet has become irregular due to breakdowns in the network system, and this has slowed down Internet and email users' motivation. As a result of its growing popularity, the Center already appears too small. At night, with all the activities (Internet and email consultation, classes and people coming for the library) going on, there is a real problem of space. The Vanga as well as the Luozi Projects carry in their design the need to work on linkages with other development partners working in the same areas, mainly USAID's SANRU, SECID and other agencies such as UNICEF. It should be noted that there has been little effort to identify activities with other partners in the locality at this point, except for a few conferences held by medical doctors of the hospital in Vanga. 2.3 Institutional capacity building and sustainability The various initiatives conducted since July (TMG's interim evaluation findings and recommendations, EDCIAED field trips, IFESH training workshops) have assisted in harmonizing expectations at every level of the project and in transferring skills in project management. At present, most stakeholders share a common understanding of the mission and objectives of the project. However, there are still some glitches that may lessen the chances of achieving sustainability. There are rumors and tensions that affect the team spirit of the Center staff. The Center's team leader has become unpopular and lacks managerial competence to run the Center. This situation affects attendance at the Center; it should be considered seriously and addressed immediately. The socio-cultural environment in Vanga, with the presence of three communities-the missionaries, the Vanga church and hospital, and finally the villagers-makes project implementation complex. Local stakeholders have been very slow to accept ownership, which will make sustainability difficult to achieve. The fact that the Management Committee has been reduced to a single person (the director of the hospital), and the poor attendance at the Steering Committee's meetings, are proof of the need to continue working to build ownership. B. Luozi The discussions with the various actors in the project show that there are observable changes at three levels: (i) teachers7/directors' beliefs and teaching principles, (ii) teachers' competencies and (iii) institutional capacity. 1. Observable changes 1.1 Teachers/school directors' beliefs and teaching principles The following statements heard during the interviews summarize the changes in beliefs and teaching principles: The teacher should not be the only source of knowledge The child is at the center of the learning process The teacher should rely on the use of didactic materials The teacher should encourage participation in the classroom The teacher should motivate the students. 1.2 Teachers' competencies innovative instructional strategies A large number of trainee teachers reported that after the training that they are able to: create more diversified didactic materials use didactic materials effectively use local materials to create didactic materials use other innovative instructional strategies (songs, excursions, games, etc.) in order to motivate learners and create an environment more conducive to learning. The Luozi school system stakeholders (Sous-Proved, directors, parents' representatives and the Steering Committee) are unanimous in their view that not only is the project a "wake-up call" for things they knew were not going right as far as teaching is concerned, but also it has given trainees the necessary tools to change old habits. These old habits are documented in the needs assessments for the project (Visser and Arias, 2003). In all the interviews, the statement: "le projet est venu nous reveiller" (the project has come to wake us up) was made regularly. In general, the classroom observations show that the teachers who used didactic materials were effective in presenting and manipulating the materials. They used a lot of positive supporting materials and some of them applied different strategies to motivate the students. 1.3 Institutional capacity building and sustainability A factor which affects positively on institutional capacity building and sustainability in any project is the existence of a "felt" need. If there is an awareness of a need in the community, then a project has a chance of harnessing the energy generated by that awareness towards implementing a solution to the need, and by doing so motivate the community to claim ownership. The observations and discussions with the Luozi Community represented by the Steering Committee and the various implementers at the local level show that they are fully aware of the poor state of the educational system in the Congo in general and in Luozi in particular. They see without any doubt the benefit of having a project focusing on complementary instructional strategies to improve basic education. They not only recognize the importance of managing the project to make it sustainable but also the need to acquire institutional capacity to manage the project in a sustainable manner. The above changes represent potential for success and can be listed among the gains of the Luozi project after less than a year of implementation. However, the mid-term evaluation reveals areas that need more work. 2. Problem areas 2.1 Teachers' competencies in using innovative instructional strategies In general, the classroom observations show that although trainee teachers are aware of the value added in integrating didactic materials into the instructional process, they do not use them systematically for each and every lesson they teach. Some subjects, such as French and mathematics, are still taught using only chalk and the blackboard as supports. During the discussions, a large number of trainee teachers admitted that they needed more training in creating didactic materials for such subjects and in making certain sophisticated didactic materials, such as preserved specimens (animals) or elaborate drawings. The findings highlight three major concerns in classroom methodology: First, in the majority of cases, when a piece of didactic material is used in the classroom, it is not fully integrated into the various steps of knowledge acquisition. In most of the lessons observed during the Team's field trip, it seems that the teacher was reviewing old concepts using didactic materials. There is no clear demarcation in the progression of the lesson between the concepts already known by the students and the new knowledge. This situation denotes a weakness in instructional strategy: the material does not support the introduction or the application of the new notion. The fact that there are scarcely any teachers' questions to check progress or to invite analysis or interpretation in the recorded classroom interactions corroborates this finding. Second, the Problem-Based Learning (PBL) approach is not yet used by the trainee teachers. There seems to be a gap in building the new knowledge on what the students already know. When students are active in class, they are answering teachers' cueing. Only one lesson involved a group activity. The communication flow is still 90% teacher-to-student. An illustration of that is that trainee teachers still rely a lot on rote learning techniques. Third, there seems to be no clear effort to address gender issues in the classroom. The recorded interactions show that in most cases, teachers elicit answers from students who raise their hand. There was only one case where the teacher actually invited girls to answer questions or comment on classroom activities. 2.2 Institutional capacity building and sustainability As far as mechanisms to ensure institutional capacity building and sustainability are concerned, the Luozi Project seems to be where Vanga was in July (See Adubra and Marcos, 2003). The community in Luozi, represented by the Steering Committee and the various implementers at the local level, does not have a full understanding of the vision, mission, and objectives of the project. The mandate given to the Committee is not clear, and roles and responsibilities are not yet defined. The Steering Committee is not fully integrated into the decision-making process of activities performed at the local level. They are called upon just to report on activities or provide information on activities to come. As a consequence, the members of that committee are frustrated and meeting attendance is low. The Steering Committee is not fully on board with project activities and thus not equipped to carry on the project. Regarding the project's sustainability, there seems to be a missing link between EDC and the schools: a sound and effective system for following up on training at the school level. EDC has a coordinator in Luozi, but he does not have a background in education, and the two educational specialists hired by the projects are both in the Kinshasa office. The vacuum thus created hinders follow-up on training and does not ensure a smooth implementation of the new methodology, which needs sustained and close support. IV. Lessons Learned At the end of the mid-term evaluation of the Vanga and Luozi pilot projects, a few lessons can be drawn: After less than a year of implementation, the two projects have the potential to achieve their objectives. There are changes taking place in the field as far as classroom methodology is concerned. Teachers and the community at large are fully motivated and see the benefits of the projects. Even if there are no tangible results as far as students' achievements are concerned, the fact that there has been a change in the methodology will impact positively on school results in the future. There is a need to capitalize on the gains and to focus more on weak areas, especially via in-depth training in the new methodology, as well as on building institutional capacity at every level of the project. V. Recommendations A. For EDC in collaboration with AED Two levels of recommendations will be made: 1. For teachers' competencies in innovative instructional strategies The Luozi Project design calls for the training of the rest of the targeted schools by the present trainee teachers. TMG feels, however, that there is a need to concentrate more on the acquisition of the new competencies by the current trainees. More in-depth training workshops are needed in the integration of didactic materials into the instructional process; in the creation of more materials, with special attention to being creative with subjects such as mathematics and French; and most importantly, in the use of the Problem-Based Learning approach. In the case of Vanga, closer supervision is needed so that the Center training staff is not left alone to impart the knowledge they have received from YouthLearn. It might be necessary to have EDCIKinshasa education specialists monitor the training activities in Vanga more regularly. The fact that schools targeted by the projects have the National Curriculum is a plus. However, the National Curriculum remains at the macro level. As far as lessons and instructional objectives are concerned, there is a need to train teachers in how to go from the stated overarching objectives to micro-level or specific lesson objectives. For training to have a sustained impact, the very important follow-up mechanism represented by school directors, pedagogical advisors and inspectors should be integrated. They should receive a more in-depth training than the teachers, since they will be the ones to provide regular support and do remedial training in the field if needed. The training in active pedagogy cannot be complete if there are no supportive instructional materials such as reference books or textbooks for teachers to base their teaching on. Having seen the weaknesses in the teachers' own education, the Team stresses that there is a need to supply these teachers with written materials. In Vanga, it can be done by buying a few copies of textbooks available elsewhere in the DRC. The same need exists for Luozi. 2. At the institutional capacity building level 2.1 Vanga Because of the problems already stated, the Evaluation Team strongly recommends that action be taken to replace the "Chef d'Equipe." In the same vein, more work is needed to empower the Steering and the Managemeut Committees. To ensure the sustainability of the project, a clearly designed strategy to achieve ownership by as many members of the community as possible is required. 2.2 Luozi There is an urgent need to start working with the Luozi Steering Committee, as was done in August 2003 with the Vanga Steering and Management Committees, to define roles and responsibilities and to explain the nature of involvement expected from its members. Furthermore, some support (in the form of transportation and data recording and processing machines) should be provided at the local level in the educational system (Sous-Proved and inspectorates) so that they can fully collaborate in the implementation of the project. The Sustainability Plan should be developed and implemented as soon as possible using a participatory approach. The participatory approach calls for the integration of all stakeholders into project activities at every stage of the initiative, from conceptualization to the actual implementation. B. For USAID The scope of work for this evaluation stated that some recommendations should be made for the expansion of the projects' activities. The Evaluation Team believes that more time should be given to the two pilot projects for their impact to be seen and evaluated. To be able to draw replicable lessons and because the overall strategy targets the educational system in the selected areas, it will be necessary to conduct a full impact evaluation after another school year. In the meantime, for future projects the Team recommends a structure that will combine the Vanga model (a community resource center with a training staff) and the Luozi model: the Vanga model for its community center, resource components and training staff, and the Luozi model in which training is provided to selected trainers (using outside expertise), followed by the training of teachers. The Team strongly recommends that the selected trainers be chosen mostly among inspectors and advisors. In all cases, the Evaluation Team emphasizes a more community development-based approach from the start of any future project. Once again, even though there are successes in the two projects, more could have been achieved if the implementers had used a more participatory approach from the beginning. The discussions held with other agencies such as IFESH and UNICEF really point to the benefits of working with and empowering the community right from the conception of any project. ANNEXES Scope of Work (Appendix A) Interviews Guides (Appendix B) Progress of Project lmplementatio~~ - Result Framework (Appendix C) Statistics Tables (Appendix D) List of visits and meetings (Appendix E) List of contacts (Appendix F) List of documents consulted (Appendix G) APPENDIX A SCOPE OF WORK Scope of Work for the The Mitchell Group Evaluation of EDCIAED Education Program in DRC Program to be evaluated: Community Learning and Resource Center, Vanga, Bandundu Province and Complementary Instructional Strategies in Luozi, Bas Congo Province Cooperative Agreement No. 623-A-00-02-00 1 14-00 I. Background As a planned activity in the Mitchell Group's task order, an evaluation of the Dot-EDU alliance Education projects in the DRC will take place from April 14, 2004 to April 30, 2004. The projects are being implemented by EDC and AED, with Monitoring and Evaluation support and guidance from the Mitchell Group. It was initiated in September 2002, and was planned for a total amount of: $1,199,887 over an initial period of 12 months. The project is now scheduled to terminate the 30' of June, 2004. TMG has made several trips since November 2002: P November 2002: for Vanga start-up during that trip TMG drafted the first version of the Vanga M&E Plan P April 2003: for Luozi start-up during the trip TMG drafted the first version of the Luozi M&E Plan P July 2003: Conducted an interim evaluation of the Vanga Pilot Project and participated in The Vanga Community Resource and Learning Center inauguration Conducted Phase I of training in M&E methodology in Kinshasa (EDC/Kin and Luozi project Coordinator) and in Vanga for the Center's staff P Nov 2003: for finalizing M&E Plan of Vanga and Luozi using the participatory approach Conducted training in M&E methodology (Phase 11) P March 2004: Monitoring trip o Conducted work sessions on M&E plan with the various staff members for a more efficient use of the M&E indicators and tools to monitor the project at their level; and, o Monitored progress on project implementation Objectives of the trip P To conduct a systematic mid-term evaluation of the 2 projects (Vanga and Luozi) with the following sub-objectives: - Assess the relevance of the project activities in achieving the goals and objectives ofthe projects as defined by USAID and EDC - Assess the relevance of the projects' activities to the needs and culture of the beneficiaries - Evaluate the scope of the projects' activities in relation to the targets set by project implementers but also in relation to the size of the population being targeted - Assess the degree of change brought by the projects' activities as far as behaviordattitudes but also achievements and results are concerned - Assess the degree of sustainability of the project > In Luozi and Vanga assess changes in methodology after the 2 training sessions conducted by EDC/YouthLearn 11. Expected Results Result 1. Establishment of a Community Resource Center in the Vanga Mission, Bandundu Province - AED. Result 1.1 Resource Center Established Result 1.2 Resource Center Operational and activities started Result 1.3 Resource Center educational services provided Result 1.4 Resource Center sustainability plan defined Result 1.5 Resource Center M&E plan defined and implemented (TMG) Result 2. Assess and pilot complementary instructional interventions (CIS) to support classroom instruction in selected thematic areas and grade levels of the national curriculum. Result 2.1 Needs Assessed for CIS Result 2.2 Pilot CIS implemented Result 2.3 Pilot M&E defined and implemented (TMG) 111. Tasks of the Evaluation Team In general the evaluation team will develop a report composed of three sections: 1) Progress in Implementation: The team will evaluate implementation progress for the project to date, measuring achievements against the work plan set down for the project by the TMG. These include the above-mentioned results and the detailed indicators 2) Impact of the Project: Assessment of the impact of the education activities on student participation, teacher participation, retention of students, quality of teaching, learning, and access to educational materials. The team will note any unintended consequences of the project as well as areas where the original design of the project may have been inadequate for the attainment of the project goals 3) Recommendations: The evaluation team will make recommendations for adjustments that may need to be made for future phaseslprojects, how to expand the project to reach many more recipients, and how to create sustainability. IV. The Evaluation Report The evaluation report will comprise three sections as follows: A. Project Implementation Progress: The overall purpose of this section will be to analyze progress in implementation compared with the original plan. This section will include: Evaluation of the project's strengths and/or weaknesses in implementation. This will include an assessment of the methodology originally foreseen for the project and its adequacy for increasing improving basic education, especially for girls, to a large number of children, given limited financial resources. Identification of constraints to project success not originally foreseen, and failures, if any, of the project to produce expected results. Identification and description of successful approaches and interventions. The evaluation team will specifically analyze the relevance of the approach to the USAIDDRC SO for education Impact Evaluation The evaluators will assess the impact of the project on improving basic education in the areas targeted. In this section evaluators will: Review progress on higher-level results indicators to compare original targets with attained results. This will include student participation, teacher participation, community involvement, achievement of students, quality of teaching, learning, and access to educational materials as per the indicators. Assess and compare the impacts between Vanga and Luozi (Using observation guides in the classroom). Linkages made with other USAID sectors (notably health and livelihoods) Linkageslrelevancy with other donors and implementers (i.e. UNICEF) Linkages made with the girls' scholarship projects in the same areas. C. Recommendations The team will provide three types of recommendations: (a) those concerned with any issues on project approaches, (b) those concerned with expansion outward to a larger beneficiary pool for future projects, and (c) those concerned with sustainability of current activities. The recommendations will be linked to the Mission's new Strategic Objective for Education in the 2004-2008 ISP. V. Illustrative Questions to be asked by the evaluation team Program Objectives/Results Have the original program objectives/results been achieved? What are the lessons learned from the activity that can be applied to future programs? What new technologies were used and how did they improve the teachingllearning environment Was the activity as originally designed the most effective way to improve basic education, especially for girls, in the DRC? Impact Can the impact of the pilot project be discerned? What has been the program impact on basic education of the targeted areas through the resource center and the complementary instructional strategies? Has the program been effective in addressing gender concerns, particularly in regard to increasing access to education, performance, and retention of girls? Future Projects What should be done definitely for future projects to meet the SO and the IRs laid out in the USAIDDRC ISP? How can the project be expanded from a pilot project to large-scale replication to reach the maximum number of primary school-age children with limited resources? How can the current activities be phased out in the most sustainable way? VI. Key Informants VI. Tentative Schedule Tuesday 13, April: Arrival in Kinshasa Wednesday 14-Saturday1 7; Work sessions with USAID, EDCIKin (Mid-term evaluation interviews and training using the observation tools) Sunday 18 Depart for Luozi Monday 19- Friday 23: Data collection: observations and interviews Saturday 24: Return to Kin Sunday 26: Kin Monday 26: Kin: Meet with UNICEF Education staff (Beatrice, etc) Tuesday 27: Depart for Vanga Wednesday 28-Friday 30: data collection in Luozi Friday 30: Return to Kin Saturday: Departure Team Composition: TMG (Bienvenu Marcos, Ayele) Other people who will accompany the team to the field: Nicholas Jenks, Darfour Ndakakanu, Mary Louise Eagleton Requirements for reporting and dissemination: A draft evaluation report will be submitted to USAID on May 15,2004. The Mission will give comments no later than May 20, and the report will be finalized by May 30. APPENDIX B INTERVIEW GUIDES Groupe No 1 : USAID, EDCIWashington, AEDIWashington, EDCKinshasa, Staff des projets (Chef dlEquipe, 5 Formateurs, Technicien, Relations Publiques, Caissikre, Gardien) Quels sont vos r6les et responsabilites dans la mise en oeuvre duldes Projet(s) pilote(s) de Vanga et de Luozi ? Pouvez-vous faire le point, a ce jour, de l'execution duldes Projet(s) pilote(s) de Vanga et de Luozi au regard des indicateurs de performance contenus dans les Plans de Suivi et dlEvaluation de chaque projet ? Pouvez-vous nous dire comment assurez-vous, au niveau de responsabilite qui est le vetre, le suivi de la mise en oeuvre duldes Projet(s) pilote(s) de Vanga et de Luozi ? Quels instruments utilisez-vous pour le faire ? Quelles sont les difficult& que vous rencontrez dans ce domaine ? Quelle evaluation faites-vous de la mise en oeuvre duldes Projet(s) pilote(s) de Vanga et de Luozi ? Quels en sont les points positifs ? Quelles en sont les insuffisances et limites ? Quelles sont les difficultes et contraintes majeures qui entravent la realisation des objectifs duldes projet(s) ? Quelles recommandations etes-vous en mesure de faire dans le sens de la realisation des objectifs duldes projet(s) ? Groupe No 2 : ComitCs de Pilotage, Comite de Gestion, Comite de Resolution des Conflits, Commissions techniques, Sous-Proved Quel sont les r8les et responsabilites de votre Organe dans la mise en oeuvre du Projet? Qu'est-ce que le Projet devrait faire ? Qu'est-ce qui a ete fait ? Qu'est-ce qui n'a pas ete fait ? Pourquoi, selon vous ? Comment etes-vous tenus inform& des realisations du projet ? Comment se deroule la mise en oeuvre du projet ? Quels sont les resultats positifs ? Quelles sont les insuffisances ? Comment voyez-vous l'avenir du projet a court terme (d'ici le 30 Juin 2004) et a moyen terme (dans 1 ou 2 ans) ? Quelles sont les dispositions que vous prenez pour assurer la durabilitk du projet ? Quelles sont les recommandations que vous pouvez faire pour la poursuite et la reussite du projet ? Groupe No 3 : Directeurs d'ecoles, Conseillers Pedagogiques, Inspecteurs 1. Que savez-vous du Projet ? 2. Quels changements avez-vous observes chez vos enseignants qui ont participe a la formation ? 3. Quels sont les benefices que votre ecole en a tires ? 4. Comment est-ce que le programme de bourses des filles contribue au projet ? 5. Quelles sont les recommandations que vous pouvez faire pour l'avenir ? Groupe No 4 : Enseignants 1. Que savez-vous du Projet ? 2. Qu'est-ce que la formation vous a apporte sur le plan positif (pidagogie active, matiriel didactique, technologie locale, ambiance de la classe ...) ? Quelles sont les insuffisances ? 3. Quelles sont les recommandations que vous pouvez faire pour l'avenir ? Groupe No 5 : Eleves 1. Vous venez de suivre un cours. Qu'avez-vous aim6 dans ce cours ? 2. Qu'est-ce qui a change par rapport B l'annee derniere (6P"e et jeme annies) ? Groupe No 6 : Parents 1. Que savez-vous du Projet ? 2. Est-ce que les enfants vous parlent du travail en classe ? 3. Quels changements avez-vous note (collecte d'objets 2r la maison, excursions, atmosph2r-e de la classe . .. ) ? Groupe No 7 : Utilisateurs du Centre de Vanga (jeunes, personnel de la santk, commerqants) 1. Que vous apporte le Centre ? 2. Quelles recommandations pouvez-vous faire pour amkliorer les prestations du Centre? Groupe No 8 : Mamans (Pre'sidente des fernmes) 1. Est-ce que vous fiequentez le Centre ? 2. Si oui, qu'est-ce que cela vous apporte ? Qu'est-ce que 1e Centre peut faire de plus pour vous ? 3. Si non, pourquoi ? Qu'est-ce qu'on peut faire afin que vous veniez au Centre ? APPENDIX C PROGRESS of PROJECT IMPLEMENTATION (See separate document) APPENDIX C MID-TERM EVALUATION OF THE LUOZI AND VANGA PILOT PROJECTS -0-0-0-0-0-0-0-0-0-0-0-0-0- PROJECT'S IMPLEMENTATION PROGRESS -0-0-0-0-0- RESULTS FRAMEWORK April 30, 2004 COMMUNITY LEARNING AND RESOURCE CENTER RESULT 1 : Vanga Community Learning and Resource Center Performance Indicator Indicator Description Means of Verification Physical structure completed, equipment and furniture installed consistent with project specifications Power source available Satellite Internet service installed and operational. Resul Building renovated according to the refurbishment plan Equipments and furniture installed as defined in the specifications for equipment, furniture and consumables Installed as defined in the technical specifications Installed as defined in the technical specifications in the Implementation Plan (to be developed statements = Field visits Delivery and installation statements Field visits Delivery and installation statements Field visits introductory courses to I the webiernail, data processing I certificates computers use and Internet search Copy of each module and schedule Completed Status of Indicator as of April 2004 Not yet completed Comments Completed Power source available 3 hours per day (6 PM to 9 PM) and 2 hours (9 AM to 11 AM) three times a week (Monday, Wednesday and Friday). Solar panels are being purchased The Center experienced an Internet network breakdown from November 2003 to March Result 1.2: The Center is operational and activities started Copy of each module and schedule 20 modules developed to support the use of technology in active Center Staff trained in I The components of the courses: I A copy of training Training completed in May 2003 Syllabus, instructional procedures and materials Completed Completed 10 modules developed and 10 other modules available on line I Performance I Indicator Center Staff trained in the 20 modules Training guides for modules developed and available online and in hard copies for use by Center Staff Center Staff trained in basic management and project management Steering and Management Committees trained in project management systems maintenance Center Policies and Procedure Manual developed and validated by Steering and Management Indicator Description Means of Verification Face to face and online training￾to equip trainers at the center with skills needed to train teachers Guides will be used by trainers to conduct subsequent training at the Center Documents which will set rules and regulations to ensure smooth operation of Center -. - . -- .- Training evaluation forms completed ~uakerl~ reports Pre and ~ost tests Courses available on line Document Review Courses, Training reports Interviews Document Review . Courses, Training reports Interviews Document Review Courses, Training reports Interviews Copy of the document and Interviews Status of Indic of April 2(1 Completed ator as I Comments facilitated by YouthLearn in July 2003 and February 2004 Completed Completed IFESH facilitated two (2) training sessions in project management and conflicts resolution in October and November 2003 Completed Completed In progress IFESH facilitated two (2) training sessions in project 1 management and conflicts resolution in October and trained in May 2003 I I A document entitled "Rkglement I d'ordre inttrieur" has beein designed with the participation of the Steering Committee, the ~ana~ement Committee and the Center Staff in August 2003. This document was signed on April 30, 2004. Result 1.3: Resource Center educational services provided Intermediate Result 1.3.1: Training of teachers and school administrators in active pedagogy 20 teachers of the 5 selected schools trained in active pedagogy Center Staff trains 20 teachers. Trainees are selected using gender equity This training session has been entirely ensured by the Center Staff Training evaluation forms completed Copy of training session reports Completed Performance Indicator 7 inspectors and pedagogical advisors trained in active All the teachers of the 5 selected schools trained in active pedagogy All the trained teachers are able to create didactic materials Increased number of didactic materials created by trained teachers All the teachers of selected schools trained in the effective use of didactic materials At least 75% of trained teachers use didactic materials to support teaching based on the National Curriculum Each trained teacher is subjected to at least two classroom visits per term Indicator Description - -- - - - - - -- - - - -- - - Center Staff trains Inspectors, Pedagogical Advisors and Directors of targeted schools. They receive specific training in innovative instructional strategies. Center staff trains all the teachers. All the trained teachers acquire the capacity to create didactic materials that are relevant and adapted to the innovative instruction and national curriculum Description : Material developed to support the innovative instruction and national curriculum - Unit : number of didactic materials Center Staff supervises training in newly developed didactic material to illustrate instructions in the classroom Teachers use newly developed didactic material to illustrate instructions in the classroom Means of Verification - - - - -- --- - - Training evaluation forms completed Copy of training session reports interviews Training evaluation forms completed Copy of training session reports interview Visits Direct Observation Interviews Visits Direct Observation Interviews Visits Direct Observation Interviews Direct Observation Reports Interviews Visits reports Interviews Status of Indicator as of April 2004 Not completed Not completed In progress Being Implemented Being implemented Being Implemented Being implemented Comments This activity has been canceled by EDC for the following reasons (Inspectors and Pedagogical Advisors offices are far from Vanga, lack of transport and accommodation facilities in Vanga etc. . . .) Some teachers from those schools have been informally initiated by their colleagues in active pedagogy, especially in the creation and the use of didactic materials A lot of didactic materials is displayed in the classrooms visited but there is no evidence that all the trained teachers are able to create these materials TBD after calculation and analysis of the data collected during the classrooms observations (see special report) The effectiveness in use of didactic materials by all the teachers of selected schools cannot be proved at this time TBD after calculation and analysis of the data collected during the classrooms observations (see special reuort) Each trained teacher has received 4 classroom visits since October 2003 Performance Indicator Indicator Description Inspectors, pedagogical advisors and the principals of the selected schools meet at least once a month to discuss active pedagogy implementation Trained teachers meet regularly to discuss progress made-in the use of the new methodology Center Staff organizes active pedagogy demonstration lessons at least once a term in each school. Increased use of the Internet by the trained teachers. Increased number of participants to the debates, tales, proverbs, 'parlottes" sessions Increased use of the library by trained teachers Center Staff coordinates the meetings Meetings organized by each school under the school director's supervision Increase in frequency and number Increase in frequency and number of users Means of Verification Meeting Reports Visits Reports Visits Visits Interviews Surveys and interviews Surveys and interviews Surveys and interviews Status of Indicator as of April 2004 Not completed Being implemented Being implemented Being implemented Being implemented Being implemented Comments To date, Inspectors and Pedagogical Advisors have not been trained Trained teachers and their directors meet at school level and at the Center to evaluate their own capacity to teach effectively Demonstration lessons are facilitated by the Center Staff The teachers have not yet been trained in the use of the Internet This activity is particularly appreciated by students, especially by girls (see Graph) The number of trained teachers using the library is poor, due to a lack of didactic manuals at the Center visiting the Center Intermediate Result 1.3.2: Educational Services provided to the Community the Center decreased by 70% from September to December Increased number of women I Surveys and interviews I Being implemented I The number of women visiting Increased number of women trained in literacy Increased attendance among Center Staff organizes literacy women to the literacy and numeracy sessions Surveys and interviews Surveys and interviews Being implemented -. attend the literacy and nurneracy sessions on a regular basis Being implemented January 2004 (see Graph) Three groups of "mamans" 2003 The number of women trained in literacy is acceptable -an increase of 57% from November 2003 to Performance Indicator Literacy and sessions cover a diverse range of themes and issues The information (news, events, ...) displayed by the Center is relevant to its objectives The information on the display board is diversified and updated regularly Increased number of participants to the conference and debate sessions The topics of the conferences are relevant to the development goal of the Center's development objectives Increased number of participants to the training sessions and by kind Training sessions are relevant to the development needs of the community Increased number of users and attendance by community members Indicator Description The objectives assigned to the Center call for diversity and relevance of issues to the participants needs. Topics covered during the conferences organized by Center Staff should reflect objectives described in the documents of the project Center Staff organizes # of training sessions according to community's needs The Learning Center is to be a Community Resource Center Means of Verification Surveys and Document analysis Surveys and document analysis Surveys and document analysis surveys and interviews Surveys and document analysis Surveys and interviews Surveys and document analysis Surveys and interviews Status of Indicator as of April 2004 Being implemented Being implemented Being implemented Being implemented Being implemented Being implemented Being implemented Being implemented Comments Examples of themes or issues: "Immunization", "La menaupose", "Resolving conflicts within the couple", etc .... The information displayed during the evaluation process are related to health, political, and local development issues The information displayed is diversified, but no evidence that it is regularly updated The number of participants to the debates is variable. Women represents about 30% of the participants for the period under evaluation (see Gra~h) The topics covered during the conferences held at the cknter are related to nutrition, history of the country, children diseases, etc.. .. TBD after calculation and analysis of the data collected during the evaluation process The number of participants to the debates is variable from a period to another. The community members, especially teachers and students do not have the capacity to pay the training fees The number of users and attendance is variable. Women participation (about 33% for the period under evaluation) is poor (see graph). Performance Indicator The library resources are diversified and relevant to the Indicator Description community's needs. Means of Verification Library catalogue Field visits Increased number of trained people in the community Status of Indicator as of April 2004 Being implemented Comments Increased number of subscriptions to Internet Services Increased income generated by fee-based servicesbffered by - Center by type of activities At least one monthly session of video-forum organized for the local Women's Association At least 75% of the Women's Association members attend the monthly session of video-forum Increased number of video￾filming activities relevant to the Center's objectives and by type of events By category (students, teachers of other schools, health and administration personnel, women and business medwomen), by type of training (software, browsing...). Local institutions and individuals Surveys and interviews Many students come to the Center for reading, but the capacity of the library is poor (insufficient number of books, lack of didactic manuals, etc.. .) Intermediate Result 1.3.3: Training and Use of Information and Communication technologies Being implemented 1 The number of trained people is Photocopying, word-processing, printing, scanning ... The type of activities covered by the Video services of the Center Surveys and interviews Accounting books Surveys and interviews Center Registry, Surveys and interviews Accounting books, Financial reports Being implemented Only the Missionaries and MAF staff benefit from this service at moment. The server capacity is not sufficient to increase the Being implemented Being implemented number of subscribers The income generated by fee- -. Only one (01) video-forum has been organized to date. The topic was "La Famille et les relations based services is poor and is decreasing continuouslv Being implemented parents/e'l2ves" No data available Being implemented An average of 3 video-filming activity per month is completed by the Center Performance Indicator Activities held in partnership with other organizations are relevant to the Center's Intermediate Result 1.3.4: Educational and cultural activities held Indicator Description Surveys and document analysis objectives Increased number of organizations in partnership with the Center schools than the selected, health and administration personnel, women and business menlwomen. Surveys and partnership contract signed Activities held in partnership with other organizations are diversified. No data Means of Verification Surveys and document analysis Partnership with SANRU Project is not yet established Not yet implemented Status of Indicator as of April 2004 Not yet implemented Comments Being implemented Community outreach activities are varied and diversified Community Outreach Unit's Reports and Surveys Being implemented Intermediate Result 1.3.5: Community Outreach The Center has established an agreement with a local organization named (TV Action) in the field of TV programs broadcasting. This agreement has been canceled by TV Action because of misunderstanding issues with the Head of the Center Increased number of activities to promote awareness of the Center The efforts made by the community outreach specialist are important, but decreased seriously between September and November 2003. The number of people reached decreased of about 70% from September to October 2003 (see Graph). The activities are varied and Description: By category of students, teachers of other diversified Community Outreach Unit's Reports and Surveys Sustainability Plan developed using a participatory approach Performance Indicator The participatory approach should include all the stakeholders. - Unit: Document source Center sustainabilil Meetings reports Indicator Description ~lan defined I I I I Result 1.4: Re: Means of Verification In progress A first draft has been developed by AED and discussed during the joint EDCIAED TDY in March 2004, but the projects' stakeholders have not been of April 2004 Status of Indicator as Comments LUOZI COMPLEMENTARY INSTRUCTIONAL STRATEGIES RESULT 2: Assess and pilot complementary instructional strategies (CIS) to support classroom instruction in selected thematic areas and grade levels of the national curriculum Performance I Indicator Description I Means of Verification I Status of Indicator as I Comments Indicator of April 2004 Assessment report delivered. Work plan and implementation plan drafted and approved by USAID. Project staff selected and hired. implemented in Luozi territory, Bas-Congo Province Luozi education authorities made aware of conclusions derived from the learning needs assessment and the importance of pilot project objectives and strategies Result 2.1: Learning Needs Assessed for CIS learning achievement in DRC Gives a set of activities to be USAID formal approval I Result 2.: Based on the activities proposed in the learning need assessment Needs assessment conducted in January 2003 Instructional Needs Assessment conducted. - - Copy of the report report approved by US AID Key staff: Project Coordinator Meetings with the Education authorities at the district level to agree on the importance of complementary Instructional strategies in the classroom. Information on findings, analysis, conclusions and recommendations for meaningful Completed Pilot Project Activities im Copy of the work plan and the implementation plan Report delivered in April 2003 USAID formal approval Staff Work Contracts Learning needs assessment report Copy of the report SC meetings reports Trip report Completed lemented Completed Completed Completed This documents are periodically updated by EDC A project Coordinator was hired in April 2003 and his Assistant recriited in March 2004 Completed in April 2003 I Performance I Indicator Description Indicator Stakeholders meetings to agree Meetings with different upon pilot project vision, goals community members groups to and activities conducted. present the main conclusions derived from the learning needs assessment Technical Committees committees established and established. committee officers nominated and appointed Teacher training modules and Produced: Syllabus, instructional plans produced and tested procedures and materials according to the new innovative developed I approach (enquirylprobleml 1 project-based learning). 6 Inspectors, 4 Pedagogical Inspectors, Pedagogical I counselors and 10 Directors I ~o&selors and Directors receive trained. a specific training adapted to their supervision role in the school system. They will also be I ( directors trainers during the them to train their colleagues during the following phases of the project Gender addressed in training Include gender issues in training Means of Verification List of participants Trip report List of Steering Committee members Lists of technical Committees members Trip report Copy of each module and plan Training evaluation forms completed Copy of training session report Training evaluation forms completed Copy of training session report Training modules contents Status of Indicator as of April 2004 Completed Completed Completed Completed Completed No evidence that gender issues have been addressed in training Comments The first information session was held in April 2003, and other initiatives followed after, but many stakeholders appear to be unaware of project's vision, goals and activities A steering committee and three technical committees have been established since April 2003 The Teacher training modules are not available at the project's office This category of participants did not have opportunity to attend the whole training session like the teachers Two (2) training sessions have already been conducted by Y outhLearnEDC in October 2003 and January 2004 The training modules are not available at the project's office for verification Performance Indicator Inventory of local opportunities and resources that can support PBL and reflect local development challenges completed as part of the training Resource kits developed and being used in classrooms Teacher trainers and teachers from targeted schools provided with National curriculum and resource kits. Memorandum of Understanding with Radio NTEMO signed. Radio coverage extension study completed. Education radio staff trained. Indicator Description Means of Verification Inventory provides sufficient materials for teacher training to be conducted and serve as models for teaching in the targeted schools. Kits developed to support the implementation of innovative CIS and national curriculum ~nventor~eports Portfolios presented by teachers that include instructional materials developed and demonstration of new strategies using PBL and awareness of gender equit Sample kit available for evaluation Observed used of kits by teachers Number of teachers who have Training sessions reports received a copy of the National List and signatures of curriculum and resource kits beneficiaries during each training session. Establish a partnership A copy of the Memorandum agreement between EDC and Radio NTEMO Aims to establish relay antennas A copy of feasibility study to reach the ten pilot schools to be selected by the steering committee in compliance with people to produce education completed programs to be aired par Radio A copy of training NTEMO certificates Status of Indicator as of A~ril2004 Being implemented Being implemented Completed Completed Completed in August 2003 Completed Comments No data available See special report on classroom observations All the trained teachers, directors, pedagogical advisors and inspectors are provided with a copy of the National Curriculum A letter of intent and a contract were signed in March 2004 for a period of three (3) months (April to June 2004) The conclusions derived from this study are not being implemented Two (2) teachers (members of Project's Radio Commission) One (1) member from Radio NTEMO Fan Club and five (5) Radio NTEMO members Staff were trained in March 2004 Performance 1 Indicator Description I Means of Verification I Status of Indicator as I Comments Indicator Education spots produced for radio broadcasting to support R.l Radio education spots will be produced to disseminate or complement problem-based Education spots broadcasted by Sustainability plan developed. learning Education spots broadcast to Radio NTEMO over community covering targeted schools Sustainability plan validated by Steering Committee. support Activity 1 Operations specified in role descriptions and Sustainability plan performed by Steering Committee. Average meetings attendance rate for the Steering Committee. Result 2, Developed on a long-term basis with the participation of community members and inputs from technical committees members Plan discussed and validated during a Steering Committee special session. Plan also validated by the community Role descriptions discussed with Steering Committee in Luozi on April 3oth 2003 (see Trip Report, appendix D) Average periodic attendance aggregated by Steering Committee member Spots available on tape and ' compact disk draft of April 2004 In progress In progress Two radio programs and four (4) education spots produced Program broadcasting started on April 1,2004, but coverage of the targeted schools is limited ;: Project sustainability plan defined No completed 1 To date, nothing has be done in A copy of the SC report acceptable to EDC and USAID Descriptive observations of on-job performance Copies of SC reports Project staff reports Not completed Not completed this area of activity 53 to 65 % on average 14 to 17 members attend regularly Steering Committee meetings APPENDIX D STATISTIOUES DE VANGA I Formation I Formation I Formation I Formation I Informatique 1 Interne Mois Naviga t HIFIT HIF Sept. 39 10 49 2003 Oct. 47 24 71 2003 Nov. 40 20 60 2003 Dec. 20 11 31 2003 Total Pkriode 1 202 1 lol / 303 FrCquentation I du Centre . Mois I Sept. 2003 Oct. 2003 Nov. Total 1 3260 1 2110 Pkriode et AlphabCtisation Andragogie Ion Actions de FrCquentation FrCquentation Sensibilisation Bibliothiique Bibliothiique I Enseignants ConfCrences SCances de I DCbats "Parlottes", Recettes Mois H Aoiit 43 2003 Sept. 68 2003 Oct. 85 2003 Nov. 29 2003 Dec. 26 2003 Jan. 80 2004 Total 331 Pkriode To be confirmed by the Center Team Leader APPENDIX E LIST OF VISITS AND MEETINGS 4/12 - 5/2,2004 Monday, April 121h: Arrival in Paris of Bienvenu Marcos Tuesday, April 13~~: Arrival in Kinshasa of Ayele Adubra and Bienvenu Marcos Wednesday, April 141h: Meeting with Mary Louise Eagleton, Monitoring and Evaluation Specialist and Darfour Ndakakanu, Contracting Technical Officer at the USAIDIDRC Mission Planning session of the TMG Team: Scope of work, conceptual framework for the mid-term evaluation, evaluation instruments and data collection process) Briefing session and interview with Kathy Ntalaja, Chief of Party, EDCIKinshasa Thursday, April 15~~: Interview with Elodie Musafiri, Education Specialist, EDCIkinshasa Interview with Noelle Wishi, Education Specialist Assistant in charge of didactic material Working session with Elodie Musafiri and Noelle Wishi (presentation and discussion of the conceptual framework and the guides for the mid-term evaluation, technical planning for Luozi) Friday, April 16~~: Interview with Joseph Lombo, Accountant and Administrative Assistant, EDCIKinshasa Interview with Darfour Ndakakanu, Contracting Technical Officer at the USAIDIDRC Mission Instruments photocopying and logistics arrangements for Luozi Saturday, April 17~': Technical planning session of the TMG Team: Luozi working schedule Sunday, April lath: Travel to Luozi by road Monday, April 19": Visit to EP Tusavuvu and EP Maduma Primary Schools: classroom observation and data collection Interview with EP Tusavuvu and EP Maduma schools directors Interview with Pkre Blaise, Luozi Pilot Interview with Grey Mayangi, Project's Coordinator Assistant Tuesday, April 20": Visit to Nkundi Primary Schools (EP Nkundi Centre, EPA Nkundi and EP Ndimba Luala: classroom observation and data collection Interview with Nkundi trained teachers Interview with Nkundi schools directors Interview with Sous-PROVED Wednesday, April 21'': Visit to EP Nzadi and EP Ntoto Ndombe I1 Primary Schools: classroom observation and data collection Interview with EP Nzadi and EP Ntoto Ndombe I1 schools directors Visit to Lufuku Primary School: classroom observation and data collection Interview with Lufuku trained teachers Interview with Lufuku schools directors Visit to a non targeted school (to be precised) Interview with Primary schools Inspectors Visit to the Administrative Authority (Administrateur du Territoire) Thursday, April 22nd : Visit to Luozi Centre Primary School: classroom observation and data collection Interview with Luozi Centre school director Interview with Luozi trained teachers Interview with Radio NTEMO staff Friday, April 23rd: Visit to Kindesi Primary School: classroom observation and data collection Interview with Kindesi school director Interview with Kindesi school trained teachers Visit to a non targeted school (to be precised) Working and debriefing sessions with EDCIKinshasa Education staff (Elodie Musafiri and Noelle Wishi) Interview with Luozi Parents Committee Interview with Luozi project's Steering Committee Saturday, April 24th: Travel to Kinshasa Sunday, April 2sth: Technical Planning meeting for Vanga (TMG Team) Monday, April 26% Debriefing meeting with Kathy Ntalaja, EDCIKinshasa Chief of Party Meeting with Sophie Chapman and Mohamed Fall, UNICEFIDRC Technical planning session of the TMG Team: Vanga working schedule Tuesday, April 27'h: Departure to Vanga via MAF Flight Interview with CRLC Education Team Interview with CRLC Community Outreach Specialist Wednesday, April 2sth: Visit to Kimbulu, Vanga I and Vanga I1 Primary Schools: classroom observation and data collection Interview with Kimbulu, Vanga I and Vanga I1 schools Directors Interview with "Groupe des Mamans de Vanga" Interview with "Groupe des Marnans de Kimbulu" Interview with CCAR Technician Interview with Management Committee Thursday, April 29th: Visit to EP Mova and EP College Dr. Fountain Primary School: classroom observation and data collection Interview with EP Mova and EP College Dr. Fountain directors Interview with CCAR users (focus groups) Interview with all trained teachers from targeted schools Interview with CCAR Team Leader Interview with CCAR Steering Committee Friday, April 3oth: Travel to Kinshasa via MAF Flight Saturday, May 1'': Debriefing session with Mary Louise Eagleton, Monitoring and Evaluation Specialist and Darfour Ndakakanu, Contracting Technical Officer at the USAIDIDRC Mission Debriefing meeting (TMG Team) Departure to Paris of Ayele Adubra and Bienvenu Marcos Sunday, May 2"d: Departure to Cotonou of Bienvenu Marcos APPENDIX F LIST OF CONTACTS USAIDIDRC Mary Louise Eagleton, Monitoring and Evaluation Specialist Darfour Ndakakanu Masegi, Contracting Technical Officer UNICEFIDRC Sophie SCHAPMAN, ChargCe du Programme Enfants VulnCrables Mohamed Fall, Charge du Programme Education EDCIKinshasa Kathy Ntalaja, Chief of Party Elodie Musafiri, Education Specialist Noelle Wishi, Education Specialist Assistant, in charge of Didactic materials Joseph Lombo, Accountant and Administrative Assistant Esther Kanya-Ngambi, Secretary Joseph Nkongolo, Driver EDCILuozi Blaise Matondo, Project Coordinator Grey Mayangi, Project's Coordinator Assistant Radio NTEMO Atou Matubuana, Chef de Station Jose Matondo, Animateur Producteur Edouardo Nzakimuena, Animateur Producteur Jean Mali Dimosi, Animateur Producteur Felly Nsungu, Animateur Producteur Luozi Steering Committee Andre Disuemi Muntiema, Sous-Proved, President du Comite de Pilotage Veronique Kuetukudila Mbasami, Representante des Femmes Marie-Rose Dimonekene, Representante des Ecoles Catholiques, SecrCtaire du ComitC de Pilotage Daniel Bakala Kalundi, ANAPECO Samba Milandu Leya, Representant des ONG Makanda Ndangi, Representant des enseignants Georges Banani, Representant des Jeunes Nsamu Tukenga, Representant des enseignants Matola Lubaki, Representant des Ecoles Catholiques Ketuba Balembama, Representant des Ecoles Kimbaguistes Nsemi Buetabela, Representant des Ecoles non conventionnees Ndandi Matondo, Representant des Ecoles Salutistes Lukelolua Nkombo Sona, Representant des Ecoles Protestantes Luozi Sector Inspectors Zephirin Niambu Ntiakulu, Chef de Pool Primaire de Luozi Georges Mandangi wa Lusemo, Chef de Pool Primaire de Mangembo Antoine Mvuwina Ngoyi, Inspecteur itinerant Pool Primaire de Luozi Antoine Makileke Makala, Inspecteur itinkant Pool Primaire de Mangembo Luozi Sector Schools Directors Laurent Nsenrni Botabela, EP Luozi Centre = Ndandi Matondo, EP Maduma Trained teachers from Luozi Sector Schools Bahombila Mabundu, EP Luozi Centre Masamba Tende, EP Tusavuvu Balewula Babakula, EP Tusavuvu Maniongo Muanda, EP Ntoto Ndombe I1 Luyindula Mayamueni, EP Ntoto Ndornbe I1 Nsenga Basuamanga, EP Nzadi Nsilulu Nsimba, EP Nzadi Lukifimpa Mavumisa, EP Maduma Baluenga Nangu, EP Maduma Mamingi, EPA Nkundi Ntangu, EPA Nkundi Mpombo, EP Nkundi Centre Milandu, EP Nkundi Centre Anatole Basakini, EP Ndimba Luala Bernadette Lutete, EP Ndimba Luala Makani, EP Kindesi Bakelong, EP Kindesi Mikondol, EP Lufuku Kuyena, EP Lufuku Luozi Committee of Parents Nsongolo Ernest, Sous-Division EPSP Nsompi Mahinga, Sous-Division EPSP Mantezila Kumeza, EPA Maduma Balenda Mpasi, EP Maduma Bakukisa Mangietimona, EP Nzadi Dialungana Kiangani, EP Nzadi Mbumba Bueya, EP Tusavuvu Bikeso Florentine, Ntoto Ndombe Geya-Nzambi Mampuya, EP Vidibila Vanga Steering Committee Ngombo Isa Manikisa, Representant du Secteur Education Muyemba Matondo, Representant du Secteur Education Mukala Guy-Gerard, Representant du Secteur Education No6 Kotho Kilumu, Representant de 1'Eglise Protestante CBCO Francine Kinwani, Representante des Elkves Baby Ngombe, Representant des Elkves Philemon Muvumu, Representant du Secteur Sante Astrid Kakawete, Representante des Mamans Lukwanza Lungu, Chef du Village Gerard Muzele, Representant du Village School Directors from Vanga Sector M. Diadia, EPl Vanga J. Muzinga, EP2 Vanga K. Ndundu, EP Collkge Dr. Fountain J. Ngombo, EP Mova U. Mbata, EP Kimbulu Trained teachers from Vanga Sector Schools Damien Sona, EP2 Kimbulu Joseph Ngwanza, EP2 Kimbulu Charlotte Kimbenga, EP Kimbulu Axel Bunu, EP 1 Ngamubu Mangasi, EP 1 Mukolo Kapita, EP1 Manunga Mupie, EP2 Tambu Salayumbu, EP2 Lili Wanga, EP2 Matutila Kilarnbu, College Dr. Fountain Mpasa Muyarna, College Dr. Fountain Tambu Mayamba, College Dr. Fountain Musinga Kalala, EP Mova Masangi Mukala, EP Mova Kandebu Kantiete, EP Mova APPENDIX G List of Documents Consulted Education Project Strategy Paper - Improving Basic Education for Children, Especially Girls in the DRC - Kinshasa, June 2002 - Draft Report Associate Cooperative Agreement No 623-A-00-02-00 1 14-00 dot-COM Alliance - dot-EDU Associate Award : Democratic Republic of Congo - Tklkcentre d'Apprentissage Communautaire (TAC) et Stratkgies kducatives complkmentaires - Program Description Contract NO. HNE-1-00-00-00078-00, Request for Proposal (RFP) for Proposed Task Order to provide technical assistance in support of USAID/DRC's new basic education program, September 18,2002 Community Learning and Resource Center in Vanga, Bandundu Province of the DRC - Monitoring and Evaluation Plan - The Mitchell Group - Ayele Adubra & Bienvenu Marcos, January 2004. Community Learning and Resource Centre in Vanga, Bandundu - Interim Evaluation Report - Ayele Adubra and Bienvenu Marcos, July 2003 Community Learning and Resource Center in Vanga, Bandundu Province of the DRC - Monitoring and Evaluation Plan - Consultants' Report - The Mitchell Group - Ayele Adubra & Bienvenu Marcos, January 2004 Work Plan - Community Learning and Resource Center Vanga, Bandundu - Complementary Instructional Strategies Luozi, Bas-Congo - Leader Award No GDG-A-00-0 1 -000 1 1-00 - Associate Cooperative Agreement No 623-A-00-02-00 1 14-00 Work Plan - Revised for No-Cost Extension Request 6/6/03 - Community Learning and Resource Center Vanga, Bandundu - Complementary Instructional Strategies Luozi, Bas-Congo - Leader Award No GDG-A-00-0 1-000 1 1 -00 - Associate Cooperative Agreement No 623-A-00-02-00 1 14- 00 10. An Assessment of Learning Needs to Inform Complementary instructional Strategies for the Democratic Republic of Congo - Jan Visser, Learning Development Institute, & Sonia Arias, Education Development Center, January 2003 1 1. Democratic Republic of Congo - Community Resource Learning Center and Complementary Instructional Strategies - Technical Trip Report - Monica Biswas, February 3,2003 12. dot-EDU/DRC Work Plan Timeline - 4/23/03 13. Rapports d1Activit6s du Centre Communautaire dlApprentissage et des Ressources de Vanga (Mois d'aoijt, septembre, octobre, novembre et dicembre 2003 et Janvier 2004) 14. Programme National de 1'Enseignement Primaire - Direction des Programmes Scolaires et MatCriel Didactique, Ministkre de 1'Education Nationale, Enseignement Primaire, Secondaire et Professionnel, Kinshasa-Gombe, EDIDEPS, 2000