October 2013 This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Eric Allemano (Team Leader) and Carol Benson, Babacar Diouf, and Alhousseynou Sy under Task Order AFR-12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION EVALUATION REPORT Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania TLMP in Senegal Intentionally left blank EVALUATION OF THE USAID-FUNDED TEXTBOOKS AND LEARNING MATERIALS PROGRAM (TLMP) IN ETHIOPIA, GHANA, MALAWI, SENEGAL, SOUTH AFRICA AND TANZANIA TLMP IN SENEGAL Prepared by: Eric Allemano, Team Leader Carol Benson, Team Member Babacar Diouf, Team Member Alhousseynou Sy, Team Member October 6, 2013 The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Performance Evaluation of TLMP – Senegal i Table of Contents ACKNOWLEDGEMENTS ................................................................................................................iii LIST OF ACRONYMS AND TERMS ...............................................................................................iv THE SENEGALESE SCHOOL SYSTEM .......................................................................................... v MAP OF SENEGAL ............................................................................................................................ v EXECUTIVE SUMMARY ................................................................................................................... 1 Overview .............................................................................................................................................................................. 1 TLMP in Senegal .................................................................................................................................................................. 1 Research Methods .............................................................................................................................................................. 2 Successes and Challenges ................................................................................................................................................. 2 Lessons Learned ................................................................................................................................................................. 3 Key Recommendations ..................................................................................................................................................... 4 1. EVALUATION PURPOSE, QUESTIONS AND OBJECTIVES ................................................. 6 1.1 The Background to TLMP ........................................................................................................................................ 6 1.2 Evaluation Objectives ................................................................................................................................................ 6 2. PROGRAM BACKGROUND ......................................................................................................... 7 2.1 ECSU Responsibilities ................................................................................................................................................ 7 2.2 MOE Context and the Educational System .......................................................................................................... 8 3. EVALUATION METHODS AND LIMITATIONS ...................................................................... 9 3.1 Methods ........................................................................................................................................................................ 9 3.2 Research Conducted at ECSU ................................................................................................................................ 9 3.3 Research Activities in Senegal ................................................................................................................................. 9 3.4 Limitations of the Study .......................................................................................................................................... 10 4. FINDINGS AND CONCLUSIONS .............................................................................................10 4.1 ECSU Achievements and Challenges ................................................................................................................... 10 4.1.1 Specific Achievements........................................................................................................................................ 11 4.1.2 Specific Challenges .............................................................................................................................................. 12 4.2 TLMP Output ............................................................................................................................................................ 14 4.3 Project Management ................................................................................................................................................ 15 4.4 Project Implementation .......................................................................................................................................... 15 4.4.1 Materials Development ....................................................................................................................................... 15 4.4.2 Achievements and Challenges for the Senegalese MOE ........................................................................... 16 4.4.3 A Vignette on How Teachers Actually Used the TLMs ............................................................................ 17 5. LESSONS LEARNED ...................................................................................................................18 5.1 Lessons for Project Planning and Implementation ............................................................................................ 18 5.2 Lessons for Implementers ...................................................................................................................................... 18 5.3 Lessons for USAID ................................................................................................................................................... 18 6. RECOMMENDATIONS ...............................................................................................................19 6.1 Materials Improvement ........................................................................................................................................... 19 6.2 Systems Improvement ............................................................................................................................................. 19 ANNEX A. SCOPE OF WORK .......................................................................................................21 ANNEX B. DATA COLLECTION INSTRUMENTS .....................................................................24 ANNEX C. MASTER DATA COLLECTION INSTRUMENTS ...................................................25 Performance Evaluation of TLMP – Senegal ii ANNEX D. SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES ....................33 ANNEX E. SUMMARY OF FINDINGS ..........................................................................................44 ANNEX F. EVALUATION OF SELECTED TLMS........................................................................46 ANNEX G. LIST OF CLARIFICATIONS ON THE TLMS FOR POSSIBLE MOE DISTRIBUTION ...............................................................................................................................49 ANNEX H. CONFLICT OF INTEREST STATEMENTS ..............................................................50 Performance Evaluation of TLMP – Senegal iii ACKNOWLEDGEMENTS The in-country team members would like to express our appreciation for the willingness of Ministry of Education officials to provide documents and information as well as ensure that we could access targeted schools, teachers, and students. Dr. Abdou Sene, TLMP Director, who was in Senegal for other work during our time there, facilitated our scheduling and was always willing to answer our questions. The international members also note the extremely important contributions made by national team members Alhousseynou Sy and Babacar Diouf, whose experience and contacts greatly facilitated our access to key informants and information. The IBTCI Research Team in Dakar: Left to Right: Eric Allemano, Alhousseynou Sy, Carol Benson, Babacar Diouf Performance Evaluation of TLMP – Senegal iv LIST OF ACRONYMS AND TERMS AEI President’s African Education Initiative CA Cooperative Agreement CPD Continuous Professional Development ECSU Elizabeth City State University EFA Education for All EFL English as a Foreign Language EGRA Early Grade Reading Assessment ESL English as a Second Language HBCU Historically Black College or University IA Inspection de l’Académie (Regional Education Office, Senegal) IBTCI International Business & Technical Consultants, Inc. IDEN Inspection Départementale de l’Education Nationale (Departmental Education Office, Senegal) INEADE Institut National d’Etude et d’Action pour le Développement de l’Education (National Institute for Applied Research in the Development of Education) Manuel School textbook (aligned with/representing the official curriculum) Manuel d’appoint Reference book (supporting but not aligned with the curriculum) MEN Ministère de l’ Education Nationale (Ministry of Education, Senegal) MOE Ministry of Education MSI Minority Serving Institution NIH National Institutes of Health PAC Program Advisory Committee PTA Parent-Teacher Association RFP Request for Proposal SIPS Société Industrielle de Papeterie au Sénégal SVT Sciences de la Vie et de la Terre (Life and Earth Sciences, Senegal) TL Team Leader TLM Textbooks and Learning Materials TLMP Textbook and Learning Materials Program UNICEF United Nations International Children’s Fund USAID/S United States Agency for International Development/Senegal USAID/W United States Agency for International Development/Washington Performance Evaluation of TLMP – Senegal v THE SENEGALESE SCHOOL SYSTEM École elementaire Elementary school (6 years, 3 stages of 2 years each) Collège Middle school (4 years, equivalent to grades 7-10) Sécondaire, lycée Secondary school (2-3 years) Directeur d’école School director = Person responsible for an elementary school Principal Person responsible for a middle school Proviseur Person responsible for a secondary school Elementary school Middle school Secondary school Stage Name Grade Name Grade Name Grade First CI 1 6ème 7 2ème 11 CP 2 5ème 8 1ère 12 Second CE1 3 4ème 9 Términal 13 CE2 4 3ème 10 Third CM1 5 CM2 6 MAP OF SENEGAL Field visit sites are marked in yellow. EXECUTIVE SUMMARY Overview TLMP was implemented in Senegal between 2009 and 2012 under a Cooperative Agreement (CA) awarded by USAID’s Africa Bureau to Elizabeth City State University (ECSU) in North Carolina. TLMP was envisioned as a program to generate high quality textbooks and learning materials (TLM) developed in partnerships between Minority Serving Institutions (MSI) of higher education in the U.S. along with ministries of education (MOE) in several African countries. The overall aim of TLMP was to provide high-quality textbooks to African students. According to the TLMP White Paper produced by USAID, well over 25 million children have gained access to the textbooks produced, which consist of more than 500 titles in 13 languages. According to the CAs awarded to each MSI, the materials produced were to be fully aligned with national curricula, to focus on primary education, to be culturally relevant, and to integrate important cross-cutting themes such as HIV/AIDS, gender sensitivity and equity, hygiene and youth leadership. An initial inquiry by ECSU determined that in Senegal the focus of the TLMs should be on middle school and lower secondary, and on providing French-language support materials (manuels d’appoint) in mathematics and the sciences rather than on curriculum-based textbooks (manuels). According to the CA, the contents of the TLMs should reflect the official curriculum of Senegal as well as recommendations of the National Education Reform Committee and international trends in curricula. In addition, the TLMs were to be culturally sensitive, research-based, and supportive of gender equity and grade-age appropriateness. This evaluation of TLMP in Senegal is intended to satisfy the following objectives:  Validate stated program goals and impacts;  Assess the results achieved in Senegal in relation to intended program targets as well as standardized and variable indicators (including quantitative and qualitative impacts on local materials production capacity, teaching and learning;  Determine if in-country institutions (with support from ECSU) were able to deliver services effectively in terms of coordinating material design, alignment, production and distribution;  Analyze usage of allocated USAID funding and overall cost effectiveness;  Highlight specific program accomplishments;  Explore the level of satisfaction on the part of the Senegalese Ministry of Education and educational stakeholders including teachers, parents and students;  Document challenges and lessons learned;  Make clear, explicit and actionable recommendations, suggesting options to expand the impacts achieved to date and determining the conditions under which scale-up and/or replication in other countries would be recommended in accordance with the new Agency Education Strategy http://www.usaid.gov/ our_work/education_and_universities/documents/USAID_ED_Strategy_feb2011.pdf The evaluation was launched with interviews conducted in December 2012 by the IBTCI Project Director on campus at ECSU. Between January 27 and February 9, 2013, a four-person team comprised of two U.S. consultants and two Senegalese education specialists, all with expertise in TLMP-related areas, conducted in-country field work, which included a literature review, interviews with senior MOE officials, school principals and teachers, school visits and observations of teachers using TLMs, and discussions with students. The team worked as a whole for one week and was divided into two for the second week, with one sub-team concentrating on research in the North (Saint Louis, Louga and Thies) and the other in the South (Fatick, Kaolack and Kafrine). TLMP in Senegal TLMP in Senegal was funded in two phases. Phase 1 covered October 1, 2005 to September 30, 2008, Performance Evaluation of TLMP – Senegal 2 with the aim of producing at least one million textbooks and learning materials (TLMs). The original CA was funded in the amount of $3,000,000 (October 1, 2005) and was later amended to include an additional $2,000,000 (October 1, 2007), bringing the total amount for the three-year period to $5,000,000, and producing 1,634,000 books. With the exception of two teacher’s guides, all the TLMs were student materials. Phase 2, the expansion phase, which is the subject of this evaluation, ran from September 1, 2009 to December 31, 2012 with a final budget of $8,066,000 and produced 2,650,000 books (three million was the target in the CA). During Phase 2, ECSU facilitated the development of 20 titles for various grade levels and subjects while supporting the printing of workbooks for three subjects at the primary level. No teacher’s guides were produced during Phase 2. There was reportedly heavy involvement from ECSU during Phase 1, but during Phase 2 ECSU’s involvement was primarily in reviewing the materials that Senegalese educators had prepared. Based on the “lessons learned” from Phase 1, the operating principle for Phase 2 was to institutionalize the production of materials, both content and physical production, in Senegal to the greatest extent possible. Research Methods While this evaluation attempted to gain evidence of learning outcomes, i.e. impacts of the TLMs on student learning, the amount of time and need to cover rural and urban departments of many regions meant that the data is impressionistic rather than comprehensive in nature. However, the team visited schools in seven of 14 regions of the country, and was able to triangulate findings thus producing stronger conclusions. The team spoke with some parents, observed classes when possible, and looked for impacts on learning when books were in students’ hands, but most of the data speak to the processes of TLM development and distribution. The data presented constitute the opinions and experiences of Senegalese educators, teachers and learners who had contact with the TLMP processes or the outputs, the materials themselves. In sum, the fieldwork focused on assessing the processes of materials development, storage and distribution, and use of the TLMs by teachers and students. Successes and Challenges From the perspective of the ECSU community, participation in TLMP made a very significant contribution to strengthening the university’s capacity to address international activities, involving both faculty and students. Specific instances for which TLMP served as a catalyst include the establishment of a Global Education Center on campus, the strengthening of international student exchange programs with Senegal and China, and participation in the USAID-supported American-African Universities Collaborative of the African Presidential Center. The TLMs were produced in collaboration with the MOE and a team from the curriculum development unit. The project fit well with USAID’s strategic objectives, and the Mission played an important role by creating an advisory committee on which all stakeholders, including ECSU representatives, reviewed and approved all aspects of the program, and by actively participating in the development and review of the materials. The process exposed the Senegalese subject specialists and other participants to new ways to go from concept to draft product within a limited time frame, to working under pressure and to following strict production requirements. The TLMs were printed by two different printing companies in Dakar, whose capacity was improved. The printers were responsible for delivery of the TLMs to the regional education offices (Inspections de l’Académie), from which the packaged materials were distributed to the schools. The achievements of the MOE in partnering with ECSU to implement TLMP include the improved ability of its staff and partner institutions to develop TLMs. Teams of Senegalese subject-matter specialists and review committees were created to develop and validate the draft TLMs before printing. However, little information is available about field testing of the materials. (We understand that some individual teachers used some chapters for some lessons, but the “validation” process seemed to involve Performance Evaluation of TLMP – Senegal 3 discussions among specialists rather than piloting with students.) Finally, the MOE now has a supply of support TLMs of a reasonable quality in mathematics and science for middle and secondary schools where books were not previously available. Several outcomes of TLMP have the potential to boost student learning. One of these is the provision of resource materials (manuels d’appoint) in mathematics and science, most of which are generally useful for teaching the national curriculum, especially by skilled teachers. The TLMs provide a variety of texts and exercises which could potentially save teachers and students learning time, provided they no longer feel they need to copy everything onto and from the blackboard. The evaluation team found that the science and mathematics materials were produced in numbers that would significantly improve the pupil-to￾book ratio in the lower and upper secondary schools where they were used in the classroom. In addition, TLMP helped the MOE by reproducing supplementary materials for the elementary level in 2009 and again in 2012, demonstrating flexibility in responding to the MOE’s need for TLMs. However, there were many challenges for the TLMP in Senegal. These have their basis in ECSU’s lack of experience when it came to TLM development, working with the Senegalese counterparts in French, dealing with the administrative constraints of USAID, consistently following the aims of the project, and following up on book development, distribution and measuring learning outcomes. First, the process of TLM development was very rushed, being based on U.S. fiscal years and having to observe (and miss) printing deadlines. TLMs went through a committee “validation” process but were not piloted with teachers or students. There were numerous inappropriate and non-African illustrations in the texts and highly limited gender awareness or inclusion of mandated cross-cutting themes. During Phase 2 (2009- 2012) there were no teacher’s guides to familiarize teachers with how to use support materials to teach their subjects. Instead, according to available ECSU reports, a program of in-service training using a “cascade model” was set up to guide teachers in how to use the TLMs. There were, apparently, some trainers who were trained; however, the research team was unable to locate any independent documentation of this process or its results. There were obstacles in the distribution process and limited follow-up to correct them, since the process was not monitored. When TLMs did reach the schools, there was limited understanding of how books, which are free, were to be distributed, and many remained in storage or in libraries rather than in teachers’ and students’ hands. There were very positive exceptions to these findings in individual districts and schools, where well-trained and motivated teachers were able to put the TLMs to good use with students, and where students were able to use the TLMs either in the library or (having signed them out) at home. Finally, since copyrights are in the names of individual former faculty members of ECSU, there is the potential for complications and delays as to the MOE’s right to make revisions, reprint materials, etc. It should be noted, also, that various materials incorporated content, primarily illustrations, from other sources without permissions. Lessons Learned There are a number of project-specific lessons to be learned. One major set relates to M&E deficiencies. The TLMP project would have benefitted considerably had a logical framework analysis been done at the beginning to develop appropriate, achievable goals, objectives and specific indicators that could be assessed during implementation. Baseline data on pupil-to-textbook ratios should have been collected but were not. There should have been both monitoring and evaluation at each step to ensure that the original goals of the project were being adhered to, or that there was good reason to diverge from them. There should have been follow-up activities (such as checking to see that the TLMs had actually reached the teachers and students) which would result in clear, budgeted actions. Another set of lessons relate to the materials themselves. The TLMs should have been properly piloted and criteria applied to check them for gender awareness, inclusion of cross-cutting themes, language level, grade/age level, and how usable they were for teachers. A third set relates to production. The process of procuring the printing of materials as separate procurements, rather than as a logical grouping, led to Performance Evaluation of TLMP – Senegal 4 inefficiencies, resulting in higher costs, and this was exacerbated by production schedules missed by TLMP. However, a more critical set of “lessons learned” relates to a fundamental design flaw. The mere distribution of TLMs does not ensure that they will be used at all, let alone that they will be used the way they were intended, especially in the effective absence of teacher training. Teachers who have not themselves had access to TLMs do not automatically know how to teach with them. Early among the school visits the team started finding books stuck in storage rooms full of dust or in libraries still in boxes or lying pristine on shelves, and it is fair to assume that a major reason for this is lack of a level of comfort. The most capable teachers were clearly able to make use of the TLMs; the less competent were not. Teacher’s guides, along with TLM-related training, are necessary elements of the teaching￾learning process, not optional luxuries. Key Recommendations 1. Rigorous planning and both systemic and systematic M&E, with follow-up, is critical to assuring that projects run smoothly. 2. Improving the ability of Senegalese teachers and students to use TLMs.  Senegal’s MOE should provide guidance to inspectors and schools as to the status of TLMs and how they should be used.  The MOE should be supported in orienting school inspectors so that they can follow up in all school districts to ensure that students have use of the TLMs. In Annex H we provide a set of clarifications that the MOE could use to communicate with its field staff.  USAID should support the development of a set of charts showing how the TLMs build the subject- and grade-specific competencies of the national curriculum.  Teachers need to get the skills they need in using TLMs to promote learning. The TLMs, along with the set of charts linking them to the curriculum, should be used in this teacher professional development.  In-service training could have given a temporary boost to help teachers use the new materials. However, this is still challenging in countries like Senegal which, due to rapidly expanding school systems, must hire new, often inadequately trained teachers every year. There is also attrition as teachers either retire or leave the profession. 3. Development of the materials  For any project involving the development of educational materials, teachers with typical capacity should be involved, along with specialists in the development of draft TLMs, contributing to ownership and lending realism to the task.  Materials development specialists should have been engaged so that important steps in TLM development would be followed.  TLMs should be submitted to analysis using criteria for gender awareness, languages and levels of language, content and sequence, cross-cutting themes, and other criteria important to the target users.  Piloting with classroom teachers and students should be planned and budgeted. Teacher orientation/training should be planned and budgeted. Teacher’s guides should be developed for future TLMs that are designed for the actual audience. The research team found that Senegalese teachers who were not well trained had difficulty in using the TLMs. 4. Other  USAID and/or the MOE should encourage the use of procurement mechanisms that are more efficient and more cost efficient than ordering the production of each TLM as an individual procurement. One better approach, for example, would be to group TLMs with similar size print runs together.  As is commonly the case in sub-Saharan Africa, there were snags in getting materials from Performance Evaluation of TLMP – Senegal 5 district offices to schools. Rather than hoping that principals will figure out their own ways to get TLMs, the MOE should strive to negotiate with other agencies which have vehicles, such as the police, to assist with distribution.  Insofar as copyrights that were assigned to ECSU faculty members, the right of the MOE to revise and/or reprint TLMs should be clarified.  TLM developers should obtain the rights to use intellectual property, such as illustrations, that belongs to others. (Preferable would be for Senegalese developers to develop their own.) Performance Evaluation of TLMP – Senegal 6 1. EVALUATION PURPOSE, QUESTIONS AND OBJECTIVES 1.1 The Background to TLMP The overall aim of the TLMP was to provide high-quality textbooks to African students. According to the TLMP White Paper produced by USAID, well over 25 million children have gained access to the textbooks produced, which consist of more than 500 titles in 13 languages. The Cooperative Agreements (CA) awarded to each MSI state that the materials produced are to be fully aligned with national curricula, to focus on primary education, to be culturally relevant and to integrate important cross-cutting themes such as HIV/AIDS, gender sensitivity and equity, hygiene and youth leadership. The extension of TLMP between 2009 and 2012 built on the President’s African Education Initiative (AEI) launched by USAID’s Africa Bureau in 2005. It was designed to contribute directly to USAID’s effort in the development and distribution of learning materials. A related goal was to improve the management capacity of sub-Saharan African (SSA) host country partners in the education sector. CAs were awarded to Elizabeth City State University (ECSU) and four other Minority-Serving Institutions (MSI), each of which was responsible for managing and implementing the TLMP in a specific country – for ECSU, to continue work in Senegal – and with achieving specific outputs of TLM production. (The University of Texas, San Antonio was awarded separate CAs to finish work in South Africa and to replicate it in Malawi.) These materials were to be developed and/or adapted in partnership with the host partner country’s Ministry of Education (MOE) and other relevant local specialists. The main objectives of the TLMP were to: 1) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities in primary schools within SSA, 2) strengthen the capacity of U.S. MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). Originally a program to develop TLMs for primary schools, ECSU’s initial needs assessment determined that MOE priorities in Senegal were for French language middle and secondary school TLMs in history, geography, mathematics and science. This notwithstanding, in the AEI phase a total of 1.63 million primary school textbooks was produced for the MOE before 2009. While most were reprints of existing books, the project produced a French grammar book and two teacher’s guides for environmental science, the latter using translated materials. The project also helped the MOE by reprinting several cahiers d’intégration, which were workbooks produced by a different project that were designed for use in a single school year and had to be reprinted every year. 1.2 Evaluation Objectives This Performance Evaluation is intended to satisfy the following objectives with respect to the 2009- 2012 period of the implementation of the Textbooks and Learning Materials Program (TLMP):  Validate stated program goals and impacts;  Assess the results achieved in Senegal in relation to intended program targets as well as standardized and variable indicators (including quantitative and qualitative impacts on local materials production capacity, teaching and learning;  Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution;  Analyze usage of allocated USAID funding and overall cost effectiveness; Performance Evaluation of TLMP – Senegal 7  Highlight specific program accomplishments;  Explore the level of satisfaction on the part of the Senegalese Ministry of Education and educational stakeholders including teachers, parents and students;  Document challenges and lessons learned;  Make clear, explicit and actionable recommendations, suggesting options to expand the impacts achieved to date and determining the conditions under which scale-up and/or replication in other countries would be recommended in accordance with the new Agency Education Strategy http://www.usaid.gov/ our_work/education_and_universities/documents/USAID_ED_Strategy_feb2011.pdf The overall purpose of this evaluation provides USAID/Africa Bureau and USAID/Senegal with answers to these questions:  What documentation can be provided on the impact, if any, of TLMP on the way that students learn and teachers teach in Senegal?  What conditions have facilitated or constrained the impact of TLMP in Senegal?  Is there evidence that successes and lessons learned during the implementation of TLMP in Senegal have been institutionalized or incorporated into national education sector activities? If so, by which entities? This Performance Evaluation also assesses the impact of TLMP on the U.S. Minority Serving Institutions as well as on host country partners. (See ANNEX A – SCOPE OF WORK) 2. PROGRAM BACKGROUND 2.1 ECSU Responsibilities ECSU’s vision for TLMP was not only to increase the number of textbooks to which students have access, but also to enhance the teaching and learning of mathematics and the sciences, particularly at the middle and lower secondary school levels, by integrating the most effective instructional methods into books and teacher’s guides. However, the decision was made to develop support materials (manuels d’appoint) rather than curriculum-based textbooks (manuels), because of the length of time and the bureaucratic obstacles impeding textbook approval in Senegal. Only two teacher’s guides were produced, and they were designed to accompany primary school textbooks on science and the environment. TLMP in Senegal was funded in two parts. The first covered October 1, 2005 to September 30, 2008, with the aim of producing at least one million textbooks and learning materials (TLM). The original CA was funded in the amount of $3,000,000 (October 1, 2005) and was later amended to include an additional $2,000,000 (October 1, 2007), bringing the total amount for the three-year period to $5,000,000, and producing 1,634,000 million books. The second part ran from September 1, 2009 to December 31, 2012, with a final budget of $8,066,000, and produced approximately 2,650,000 books. During this part, ECSU facilitated the development of 20 titles for various grade levels and subjects, while supporting the re-printing of workbooks No teacher’s manuals were produced in Phase 2. There was heavy involvement from ECSU during the first part, but during the extension ECSU’s involvement was primarily in reviewing the materials that Senegalese educators had prepared. Based on the “lessons learned” from part 1, the operating principle for the second was to institutionalize the production of materials, both content and physical production, in Senegal to the greatest extent possible. The focus of the TLMs developed by TLMP in Senegal was on middle and lower secondary school, and on providing support materials (manuels d’appui). The reason cited for focusing on post-primary education was that the curriculum of the primary grades was undergoing revision at the time of the project. While ECSU helped develop a few TLMs for the elementary level (three titles), the bulk of the Performance Evaluation of TLMP – Senegal 8 elementary materials had already been developed by the MOE and were simply reprinted by the TLMP project. ECSU affirmed in the CA that the contents of the TLMs should reflect the official curriculum of Senegal as well as recommendations of the National Education Reform Committee and international trends in curricula. In addition, the TLMs were supposed to be culturally sensitive, research-based, and supportive of gender equity and grade-age appropriateness. TLMP required significant USAID/Senegal involvement to bridge the cultural and linguistic gaps between ECSU and Senegalese government education officials. To this end, a Program Advisory Committee (PAC) was set up in Senegal.1 The role of this committee was to guide TLMP and ensure that potentially helpful entities and individuals were aware of and participated (as needed) in the project. The specific functions of the PAC were the following: 1. To provide quality control for program management; 2. To ensure that the best-qualified national specialists were contracted in the areas of curriculum, writing, illustrating and graphic production; 3. To ensure cost-effective choices of strategy, administration and logistics in the TLMP; 4. To assist in establishing vetting and revision procedures for the TLM drafts before printing; 5. To set standards of durability and shelf life of the TLMs in light of local conditions and costs. 2.2 MOE Context and the Educational System Senegal has an educational system largely based on the French colonial model. Although there is experimentation with the use of mother tongues in formal education, French remains the major language of instruction at all levels of the educational system. Since 1996, educational governance has been delegated to the 14 administrative regions of the country, which have considerable legal responsibilities and financial autonomy. Communes and communautés rurales have jurisdiction over the management of basic education services, literacy training, vocational training and promotion of national languages. Central services, in their reduced role, concentrate on policy design, monitoring and evaluation, production of teaching and learning materials and coordination of the various levels of the education system. However, few local governments adequately fulfill their legal obligations because the budgets allotted are insufficient in relation to their needs. It is very often parent-teacher associations (PTAs) that pay school water, electricity and telephone bills.  The Inspections d’Académie (IAs), the regional inspectorates responsible for implementing national education policy, are in charge of supervising the implementing national policy on curriculum, teacher training and other functions. However, they do not have the tools and human resources needed to fulfill their mission, nor are they equipped to evaluate future needs for learning inputs.  The Inspections départementales (IDENs) play a strategic role in policy implementation. They provide technical support and pedagogical facilitation for the schools in their department. However, considerable efforts are needed for the IDEN to properly fulfill these functions. Local government involvement in the development of education has significant gaps. According to a separate analysis of the education system done by a team involving our two national consultants, local government resources are not allocated on the basis of objective criteria and it is difficult to obtain precise figures on the total amount of resources stipulated in the budget or effectively made available to the education system. The analysis found that among the many weak points in the decentralized management of the education system is a lack of transparency, administrative red tape, inefficient resource management, and problems of access and poor quality of services. This was reflected in the 1 PAC members were representatives of the MOE including the Minister or his/her delegate, directors of Elementary and Middle/Secondary Education Departments, the IGEN director, president of the national PTA, and other management and communication officers. Performance Evaluation of TLMP – Senegal 9 confusion over the distribution of the TLMs that the printers delivered to the IAs. The IDEN were sometimes unable to find the resources to take the cartons of books to the schools. Similarly, according to some school principals interviewed, the MOE had not disseminated clear guidelines about the status and intention of the TLMs. Still, progress has been made in the financing of schools by local government: their contribution is larger than the subsidies they receive from the state in the form of endowment funds and assistance.2 Development of new textbooks or choice of foreign textbooks is still a centralized process in Senegal. The Ministry of Education relies on the Institut National d’Etude et d’Action pour le Développement de l’Education (National Institute for Applied Research in the Development of Education) or INEADE, a special unit to develop textbooks that conform to the approved curriculum. This unit also approves commercially-available textbooks that are marketed in Senegal, primarily by French publishers. Some of the experts in this unit, most of whom work as education professors at the national university, have been writing and editing texts since 1984. 3 3. EVALUATION METHODS AND LIMITATIONS 3.1 Methods The team used a “mixed-methods” methodology that emphasized qualitative approaches. It consisted of review of project documents (including a number of the TLMs); interviews with senior officials of ECSU and the MOE, Senegalese and U.S. TLMP implementing staff and administrators, materials developers, MOE officials at district and regional level, particularly those responsible for training and pedagogical support in each region visited, and the printers; classroom observations with discussions with principals, teachers, and students; and assessments of student performance. 3.2 Research Conducted at ECSU The evaluation project director conducted a site visit to ECSU between December 4 and December 6, 2012. This visit was organized by Dr. Abdou Sene, who succeeded Dr. Johnny Houston as TLMP project director in December 2009, which included group and individual meetings with University Chancellor Willie Gilchrist and top university academic officials, Dr. Sene, Dr. Houston, and other TLMP staff, plus other university administrators. In addition to review of project documents and some materials and discussion of the mechanics of project implementation, a key focus of the sessions was on the impact of TLMP on the campus community. These are discussed in section 4.1 below. 3.3 Research Activities in Senegal The in-country evaluation activities were conducted by Eric Allemano (Team Leader), Carol Benson, Alhousseynou Sy, and Babacar Diouf between January 27 and February 9, 2013. The field visits included interviews at USAID/Senegal, MOE officials at various levels, school principals and teachers, and classroom observations and quasi-focus group discussions with students and some collective interviews with teachers and trainers. The team also had the opportunity to talk with some parents. Sites were selected to reflect both capital city and distant settings with different demographics. In addition to visits to schools in the greater Dakar area, two-person sub-teams, comprised of one U.S. specialist and one Senegalese specialist, visited sites to the north (Saint-Louis, Louga and Thies) and to the southeast (Kaffrine, Kaolack and Fattick). Please see the map of Senegal above for site locations. School sites were selected to reflect capital city, peri-urban and remote rural settings with different demographics. (See 2 Senegal. Effective delivery of education services: A review by AfriMAP and the Open Society Initiative for West Africa by Hady Guèye, Lamine Kane, Babacar Diop and Amadou Abdoul Sy (Nov. 2010). 3 An Outcomes and Impacts Evaluation of the President’s Africa Education Initiative. Country Study Report: Senegal. Aguirre Division of JBS International (2009). Performance Evaluation of TLMP – Senegal 10 ANNEX D – SCHEDULE OF DATA COLLECTION ACTIVITIES.) As a result of the focus on creating middle and lower secondary school TLMs as support materials (manuels d’appui) in mathematics and sciences rather than on curriculum-based textbooks (manuels), the questions posed at the MOE and in schools had to be adapted. While there were materials developed for elementary (three titles), the bulk of the elementary materials was developed by the MOE and simply printed by TLMP. This meant that the team needed to focus on middle school visits, while trying to check on the other two levels. The team’s initial interviews with stakeholders disclosed that the project had experienced challenges in the delivery and acceptance of materials, resulting in the desirability of exploring this challenge more deeply than initially envisioned. In order to obtain perspectives of the reach of TLMP to schools at some distance from the capital, a day of fieldwork was added to the original plan, enabling each team to work three full days outside Dakar and to see schools in seven of Senegal’s 14 regions. Nonetheless time for observation of classes in session was still limited, a factor that was complicated by a teachers’ strike at one point, so as an alternative the team opted to talk to available teachers and students and to ask students to show us the books they were using. In sum, the fieldwork focused on the processes of materials development, distribution, teacher training and use of the TLMs. The team did talk to some parents, observed classes when possible and looked for impacts on learning when books were in students’ hands, but most of the data say more about the processes of TLMP than any impacts, because of a number of challenges. For a description of the stakeholders targeted in the field and the themes on which they were questioned, see ANNEX B – DATA COLLECTION INSTRUMENTS. 3.4 Limitations of the Study One inherent limitation was the time available, given the great desirability of visiting schools located away from Dakar. While the task order anticipated that a country visit would last for ten calendar days, the team spent 11 working days in Senegal, a duration which IBTCI deemed was needful in order to accommodate travel to different parts of the country. Complicating this research was a teachers’ strike, limiting our access to schools, teachers and students in the north-east, and classroom observation time was constrained. Because of time and data limitations, the use of formal quantitative methodologies was not feasible. For example, because of multiple confounding factors it would not have been possible to credibly come to any rigorous conclusions as to the effect of TLMP materials on learning, given the large number of titles and grades covered. The team does, however, report on the opinions and experiences of Senegalese educators. The TLMs developed in 2012 had not all arrived at the schools by the time of the in-country activities. This caused us to redirect our methodology somewhat to focus on questions that would be central to the project and processes as implemented in Senegal. 4. FINDINGS AND CONCLUSIONS 4.1 ECSU Achievements and Challenges According to the Chancellor of ECSU and others at ECSU, participation in TLMP made a very significant contribution to strengthening ECSU’s capacity to address international activities involving both faculty and students. Specific campus instances for which TLMP served as a catalyst include:  The establishment of a Global Education Center equipped for teleconferencing, which serves as a campus focal point for a number of international activities.  Increasing internationalization of the curriculum and improving student and faculty awareness of Performance Evaluation of TLMP – Senegal 11 international issues.  The establishment and strengthening of international student exchange programs with Senegal and China.  Funding of a scholarship for a student from Dakar.  The establishment of National Institutes of Health (NIH)-funded research linkages with Senegal, such as a study on hypertension and diabetes.  Participation in the USAID-supported American-African Universities Collaborative of the African Presidential Center, hosted by Boston University. The Collaborative provides the opportunity for ECSU to engage with dialogue with presidents of other HBCUs and of African universities and also to participate in colloquia on major issues with former heads of state of African nations. ECSU was also able to develop and implement acceptable financial procedures that made it possible for the printers to address the “Catch 22” elements associated with their need to procure large quantities of paper, etc. in order to print the materials on the basis of which they would ultimately be paid. Finance office staff also pointed to the ability of ECSU to serve as a source of information for area businesses seeking to work with potential international suppliers or customers. 4.1.1 Specific Achievements In evaluating the work of ECSU in the context of the work of all TLMP implementers, it is important to keep in mind that ECSU was the only MSI not to be conducting essentially all of its work in English. Based on the perceived needs of Senegal’s MOE, TLMP’s work focused from the beginning on post￾primary education; however, significant attention was given to the TLMP and Goal 1 emphasis on primary education. The project produced or reprinted 750,000 literacy development and/or reading books for grades 1 to 6 between 2009 and 2012, although most of these materials were not developed by TLMP itself. It is not known, however, how many of these books were actually delivered to schools and in use by teachers and students. Section 4.2 discusses the overall output of TLMs. Jointly learning from experience, the development process changed considerably since the inception of the project: Table 1: Transfer of Ownership of the TLM Writing Process Time period ECSU Senegalese Explanation/examples 2003 100-90% 0-10% Book from USA machine translated and sent to Senegal for interpretation and adaptation by national specialists. 2004-2006 80% 20% Senegalese team(s) went to USA to work with ECSU counterparts 2007-2008 50% 50% Some STTA trips 2009-2011 20% 80% Some STTA trips 2012 0-10% 90-100% Some STTA trips ECSU strived to be responsive to the needs of its Senegalese partners holistically, in ways beyond professional development. This is exemplified in part through support provided by TLMP to print and distribute elementary school learning materials developed by the Ministry, which was short of money. Another TLMP success was the reinforcement of capacity at two local printing companies, SIPS (Société Industrielle de Papeterie au Sénégal) and Polychrome, both based in Dakar. Further, both printing companies reported hiring additional local staff at least on a temporary basis during print runs. Printers rather than publishing companies were chosen to avoid overhead costs and the outsourcing of printing and via competitive bidding to eliminate the potential for favoritism towards publishers, who were Performance Evaluation of TLMP – Senegal 12 known to have relationships with MOE/INEADE staff. Distribution of TLMs to the regions was done by the printing companies, reducing the risk that books would “escape” to the market, and enabling proactive IAs and IDENs to facilitate distribution to schools. 4.1.2 Specific Challenges As mentioned above, TLMP staff at ECSU made some serious errors at the beginning of AEI, e.g. by starting with U.S. materials and using simple internet translation; however, in time, they realized that Senegalese authors should do the development and writing of the materials, and the preparation of manuscripts was delegated to Senegalese teams. Such a costly “lesson learned” could have been avoided if ECSU had been given advice by specialists in curriculum/materials development. Once the main work was moved to Senegal, which was a key aspect of the expansion phase, technical advice was still needed, particularly on appropriate methodology and activities to include in the materials. The advice was given by local subject-matter specialists. It is not clear that these materials actually did represent effective participatory pedagogy, as TLMP aimed to do, since few teachers’ manuals were developed and teachers who had been trained on the TLMs were not observed. It appears that most of the technical support given by ECSU took place during materials development and finalization workshops, but these did not appear to take place systematically, and the extent to which U.S. and Senegalese subject-matter specialists collaborated in the meantime, especially given language issues, or the extent to which U.S. curriculum specialists were at all involved, was by no means clear. According to a group of author/adapters with whom we met, the entire process of TLM development was rushed. They wished they had been given more training. They also felt their opinions were not always listened to; for example, one said the content of a mathematics TLM was too difficult and was not adapted to the Senegalese context; another said he did not see the connection with the curriculum; and all of them said they were against combining two grade levels in one book, but their opinions were overruled in the name of cost-saving. (The team heard a number of complaints in the field about the materials that combined grades, which would indicate that the project staff should have listened more closely to the Senegalese authors.) Regarding the development and validation of the TLMs, some do not seem to take into account the abilities of average Senegalese teachers (many of whom are poorly trained) or the level of students outside of the better urban schools. According to interviews conducted by the IBTCI research team, the TLMs were pilot-tested for 15 days in schools in Thies and Mbour. It is unknown how many schools participated or whether a balance of urban and rural schools was followed. The education curriculum advisor in these two cities worked closely with designated master teachers to collect data from the pilot test to pass back to the Senegalese editors. The data were collected by questionnaire and observations. (Apparently, no testing of learning results was done, as the details of the validation methodology were not available to the research team.) The sketchy information available on the validation process indicates that the drafts were finalized by a committee which did include one or two teachers from the Dakar area, but not from rural areas or other parts of Senegal. In any event, this does not constitute adequate piloting, which would have picked up on some of the issues we encountered with inappropriate, foreign photos and illustrations. One of the most blatant examples, which was referred to as problematic at least twice when we were in the field, was a set of drawings of naked non-African female and male children and adults in a description of maturation (see Annex F for examples of inappropriate content). The ECSU team should have advised the Senegalese writers about these issues. Furthermore, some of the principals and teachers we interviewed found the TLMs “too difficult”. They found the level of the mathematics books, particularly Mathématiques: Raisonnement quantitative (2008), is challenging for 7th and 8th grade students. For example, it contains a chapter entitled Nombres rationnels Leçon 5: Les Binomiaux, le triangle de Pascal et la théorie binomiale (Rational Numbers Lesson 5: Binomials, Pascal’s triangle and binomial theory). While the book, which had its origins in the AEI phase, is well- Performance Evaluation of TLMP – Senegal 13 written and laid out, it would take a creative and well-trained teacher to use this book effectively in an average Senegalese classroom, which raises questions about how the draft of the book was pilot tested. From the point of view of the original aims of TLMP as expressed in the CA, it is disappointing to see so much unrealized potential of ECSU to increase access to high-quality teaching and learning materials in Senegal. The materials produced were to be fully aligned with national curricula, to focus on primary education, to be culturally relevant and to integrate important cross-cutting themes such as HIV/AIDS, gender sensitivity and equity, hygiene and youth leadership. Regarding the level, as already noted, the decision was made to support middle school/lower secondary mathematics and sciences due to expressed needs and the fact that the primary curriculum was undergoing revision at the time. The printing of a few primary TLMs was positive in terms of project flexibility and support to national structures, but the quality of those materials was not high and there is no evidence that TLMP specialists had any input; they were merely printed by the project. The TLMP middle school/lower secondary support materials had as much alignment as possible to existing curricula, but these curricula were revised and at least some teachers felt that project TLMs were not completely consistent with what they were required to teach. Unfortunately there were no accompanying teachers’ guides developed for the math or science TLMs. While there was no requirement to run orientations or trainings related to the TLMs, ECSU did train some trainers and had a plan for a “cascade” model. (According to the ECSU newsletter, 350 teachers from the regions where books were to be distributed, along with some local/regional MOE trainers, were trained as Master Trainers.) Overlooking the fact that “cascades” are not known to be realistic nor sustainable, we were not able to verify how many actual trainings were done. We asked everyone in the field about participation in training related to the TLMs, but were not able to meet anyone who had been trained. Thus it appears that even if trainers were trained, there was no follow-through with the plan. Regarding cultural relevance, we have already mentioned the images of non-Africans; in addition, there were American/European stereotypes evident in the TLMs. There is no discernible integration of gender awareness; in fact there are few people represented in the texts, and when there are they are mostly men—girls and women are almost never depicted, nor are there many young people shown with whom learners can identify. Finally, we found no integration of other cross-cutting themes like HIV/AIDS, other than in one book (which the evaluation team did not see in the field) which seems to have been written to include HIV. There were a number of inefficiencies in print production. Bids were invited for each print run/title, which was not the most efficient way to work with printers, since costs for paper and ink would be lower if large print runs could be guaranteed. Dependence on U.S. fiscal year budgeting caused many aspects of the process to be unnecessarily rushed. For example, at least one title was printed by both companies to enable the TLMs to be produced in a short time frame, but this caused quality differences and other inefficiencies. Further, failure to submit manuscripts by the deadline caused cost overruns. Polychrome said they even invited TLM author-editors to their offices to facilitate the editing of final manuscripts. The printers said that longer-term planning (over two to three years) and guarantees of larger print runs, e.g. for entire series, would have allowed them to invest in permanent staff, better equipment and storage of extra paper and materials. Although contrary to the wishes of Senegalese educators, as expressed to the team, in a number of cases the text for two years of a particular subject was printed as a single (but bulky) volume, largely for cost reasons. The evaluation team did not have the data to do a cost-benefit analysis of this project, but from our meetings with the two printing companies and our general knowledge, we have reason to believe that printing costs were higher than necessary, for a variety of reasons. For example, a separate bidding process was initiated for each title, which meant that neither printer could be confident that it would have a sufficient amount of work to hire and train personnel. Delays in finalizing the TLM manuscripts meant that the printers had to work overtime to produce the books they had agreed to print, increasing Performance Evaluation of TLMP – Senegal 14 costs. The printers said that if they had had contracts for a number of titles to be produced over a certain time, and if they had received the manuscripts as promised, the printing process would have been much more cost-effective and efficient. One result of receiving rush orders and requests for a different format (e.g. the primary TLMs, which used smaller paper and cover sizes) was that special types of paper had to be ordered by air, thereby increasing unit costs. 4.2 TLMP Output From the end of Phase 1 through the end of Phase 2, there were over 2.5 million TLMs published by the project, according to the project newsletter. Table 2 shows the distribution between levels of education: Table 2: Summary of TLM Production 2009 – 2012 YEAR Elementary School Middle School Lower Secondary Total 2009 500,000 500,000 2010 600,000 250,000 850,000 2011 200,000 800,000 1,000,000 2012 300,000 300,000 Total 1,000,000 1,400,000 250,000 2,650,000 Most of the titles developed by the project were for middle school and lower secondary mathematics, geography/history and the sciences, as shown in Table 3: Table 3: Summary of TLM Titles Produced 2010 – 2011 English title Grade Year Copies Life & Earth Sciences (SVT) 9-10 2010 250,000 Physical Science 9-10 2010 150,000 Life & Earth Sciences (SVT) 7 2010 200,000 Mathematics 11 2010 125,000 Physical Science 11 2010 125,000 History & Geography 10 2011 200,000 Life & Earth Sciences 8 2011 250,000 Life Skills (gen) 2011 350,000 The project also produced one elementary social studies title (Discovering Our World) in 2009 and three titles for elementary French language arts, as shown in Table 4: Table 4: Summary of Elementary French Literacy TLMs Produced 2011 – 2012 English title Year Copies Learning French 2011 100,000 Language & Communication 2011 100,000 Between Sounds (French) 2012 150,000 In addition, TLMP helped the MOE by reproducing supplementary materials for the elementary level in 2009 and again in 2012, according to Table 5: Performance Evaluation of TLMP – Senegal 15 Table 5: Summary of TLM Supplementary Titles for Elementary Grades in 2009 and 2012 No. of titles Year Copies per title Total 5 titles 2009 50,000 250,000 3 titles 2012 50,000 150,000 On the positive side, it may be seen that the science and mathematics materials were produced in numbers that would significantly improve the book-to-pupil ratio in the middle schools and lower secondary. Regarding the elementary materials, the team felt it was significant that TLMP could be flexible enough to respond to the MOE’s need for resources in printing elementary school materials. The titles printed for the MOE were elementary language arts and teachers’ guides, which we found were well distributed and in use, as there was no question about ownership. Certain anomalies arise when considering the TLMs. First, the numbers of TLMs, when cross￾referenced, differ when totaled. The information comes from different newsletters and reports, and unfortunately could not be aligned or verified across sources. Second, the content and images in both the MOE materials (reproduced by the project) and the TLMP-developed materials are often inappropriate. 4.3 Project Management A central role of ECSU TLMP staff was to serve as intermediaries between U.S. and Senegalese culture. With the retirement of the original TLMP Director, who continued to serve as a consultant in mathematics, ECSU recruited and hired a Senegalese education specialist resident in the area at the beginning of the expansion phase. An additional Senegalese specialist was later hired as Program Manager. These and other members of the ECSU team were able to effectively serve as bridges and skills transfer agents between the U.S. and Senegal, ECSU and the Ministry of Education, working in English and in French, Wolof, and other Senegalese languages. 4.4 Project Implementation 4.4.1 Materials Development Conceptualization and Development: The TLMs were produced in collaboration with the MOE and a team from the curriculum development unit. The project fit well with USAID’s strategic objectives, and the Mission played an important role by creating the Project Advisory Committee (PAC) on which all stakeholders, including ECSU representatives, reviewed and approved all aspects of the program. Although there was an attempt to use Senegalese examples, photographs and references, many of the final materials lacked cultural sensitivity, and many items appear to have been downloaded in haste from the Internet without citing a source or permission given. On the other hand, one of the later books, the 9th grade history and geography book lists the source of borrowed photographs and other illustrations. However, this book is an exception to the general practice of the project. Nevertheless, despite these flaws in execution, the process exposed the Senegalese subject specialists and other participants to new ways to go from concept to draft product within a limited time frame, to working under pressure and to following strict requirements. The usual method of working was from a distance, with an occasional meeting at a hotel to work over the weekend. Authors said they learned a great deal during that time, though it was challenging and they could have used more training. The TLMP Director said that they gained capacity to use laptops, which were bought for the task by the project. In the 2010 fiscal year the technical teams in Senegal asked to develop each book jointly with the technical team in the United States. They felt that this would improve their ability to design and develop textbooks independently. One result of the collaboration was a seven-day workshop in December 2009, held in Senegal and attended by four people from the ECSU-Senegal Program in Elizabeth City (the TLM Performance Evaluation of TLMP – Senegal 16 Program Director; the former Program Director/Mathematics specialist, and specialists in Life and Earth Sciences (Sciences de la Vie et de la Terre, SVT), and Physical Sciences), 12 Senegalese educators for each of the three subjects, and eight members of the PAC. The resultant manuscripts were to be completed in a workshop held in March 2010. This presented the authors with a very tight timeframe to produce the materials, requiring a lot of rushed and last-minute work. Some books have a list of more than 25 Senegalese authors; we do not know how the writing and editing tasks were coordinated. At the week-long March 2010 Validation Workshop, which engaged over 100 individual teachers and educators from around the country to review the materials, the review unfortunately missed culturally inappropriate illustrations such as scarecrows and Thanksgiving turkeys. Had they piloted the materials with teachers, they might also have discovered that teachers did not appreciate the merging of two school years in one book. This concern had been expressed by some authors earlier in the process, but was apparently overruled for reasons of cost-effectiveness. ANNEX F – EVALUATION OF SELECTED TLMS presents an analysis of eight TLMs prepared by the project. Printing: As increasing local printers/publishers capacity was one of the goals of TLMP, the TLMs were printed by two different printing companies, SIPS and Polychrome, both based in Dakar. Publishers were avoided due to high costs and the potential for favoritism, as publishers were known to have relationships with insiders at the MOE. Bids were invited for each print run, i.e. each title, an inefficient system. The printers were responsible for delivering the TLMs to the IAs at the regional levels; to do so, they called ahead to be sure that someone at the IAs would be prepared to sign for delivery. (This system seemed to have functioned to the extent that someone was required to receive the materials; however, there was criticism of this practice at the IA level since the usual procedure would have been to receive word of the delivery from the Ministry.) There was no systematic follow-up by ECSU or by the MOE. Distribution was not monitored, nor was the use of TLMs verified in the schools. There was no monitoring and evaluation component in the TLMP CA, and no indicators upon which to evaluate project implementation. The team did see some students and some teachers using TLMs effectively in classrooms in some research locations (e.g. Kaolack, Kaffrine and Fatick), but in other locations the TLMs were not in use for a variety of reasons. Training: According to an ECSU newsletter, 350 teachers from the regions where books were to be distributed, along with some local/regional MOE trainers, were trained as Master Trainers. A plan was made to “cascade” the training to other teachers at the school level; however, the team found no evidence that further training was ever undertaken, there was no budget item for this, no apparent follow-up by the project, and none of the teachers we met in the field had been trained to use the TLMs. 4.4.2 Achievements and Challenges for the Senegalese MOE The MOE indicated that its staff and partner institutions improved their skills in TLM development, especially in content, choice of images and validation. Equally important is the creation of teams of Senegalese subject specialists and review committees to develop and validate TLMs. The MOE also now has a supply of support TLMs of a reasonable quality in mathematics and science for middle and secondary schools where books were not previously available. The Senegalese author-editors who participated in developing the TLMs said their own capacity was raised through the process. Authors gained capacity in materials development as well as computer skills, according to the TLMP director. Several outcomes of TLMP have the potential to boost student learning. One of these is the provision of resource materials (manuels d’appoint) in mathematics and science, most of which are generally useful for teaching the national curriculum, especially by teachers who have some training. Also important is the Performance Evaluation of TLMP – Senegal 17 opportunity for students in some schools we visited to take books home for periods ranging from a week to the entire school year. Another asset is the numerous images (particularly the colored graphs, maps, illustrations and photos), which some teachers felt improve the quality and efficiency of teaching. In addition, the TLMs provide a variety of texts and exercises, which could potentially save teachers and students learning time – providing they no longer feel they need to copy everything onto and from the blackboard. There was one possible “halo” effect of TLMP seen at the Regional Inspectorate of Thies, where additional resources were mobilized to train users of the TLMs. Also of interest are (unsubstantiated) reports of improved student learning in classes with trained teachers that use the TLMs. As for the challenges for the MOE, the project was operated largely through ad hoc advisory committees (separate from the PAC) and specially-hired consultants. There does not appear to have been a strategy to integrate the methodologies of the TLMs into the formal curriculum or textbook development bodies of the MOE; this integration could have increased collaboration and capacity for both U.S. and Senegalese institutions and improved the sustainability of the TLMP interventions. It is unfortunate that the MOE did not take steps to ensure that the TLMs were fully aligned with the curriculum and able to serve as required textbooks rather than support materials. It was apparently not only the primary curriculum that was a moving target; the middle school and secondary curricula were also in flux during TLMP. Some principals did not distribute the TLMs because of doubts that they conformed to the official curriculum. There was no evidence that MOE-supported pre-service or in￾service teacher training integrated the TLMs into their methodology instruction. Delivery and distribution of the TLMs was highly problematic. Some IA and IDEN heads reportedly rejected the materials or objected to the unusual delivery procedures (e.g. representatives of the printing companies calling and requiring signatures without official word from the MOE). A number of interviewees commented independently of each other that the TLMs had “fallen on our heads,” meaning that they were not forewarned of the arrival of the materials nor of the fact that they would be responsible for facilitating delivery to the schools. There was reportedly no clear message from TLMP management or the MOE about how the TLMs were to be distributed “free” to teachers and students, which meant that the response ranged from keeping the TLMs sealed in their cartons, taking some out and selling them in the market (although they were marked Not For Sale), storing cartons or books unused in storage spaces, putting them in libraries for use by teachers and students in school, borrowing to use in the classroom, or borrowing to take home for shorter or longer periods. During school visits we saw many books stored in places where they would not be available to students. All of these different behaviors depended on the local understanding of the potential of the TLMs and the capacity of the receivers to effectively exploit the materials for student learning. Finally, there was limited follow-up of distribution to the IAs and no budget or plan was in place to ensure that TLMs were in fact distributed to the school level and to the student level. 4.4.3 A Vignette on How Teachers Actually Used the TLMs It would be informative to describe how teachers in two 10th grade classes used the TLMs; we believe this to be broadly representative of the classes we observed. The first was a physics and chemistry lesson in a rural school; the other was a life sciences lesson in an urban school.  Neither teacher had been trained on the TLMs he was using.  The pedagogy was teacher-centered, particularly in the urban school, where a neophyte teacher was observed.  The teachers tended to fall back on traditional methods of writing texts and diagrams on the blackboard for students to copy, rather than engaging in a discussion of the material in the TLMs, even though there was at least one copy for every two students.  The more experienced teacher (in the rural school) asked students to come up to the board and Performance Evaluation of TLMP – Senegal 18 write answers to certain questions. The other teacher wrote the answers himself after students had responded orally.  There was some good use of questioning techniques, such as asking “why” questions that required the students to refer to their TLMs.  The newer teacher, in the urban school, gave a short dictation to give the students practice in listening and writing texts with vocabulary such as proteids vs proteins. 5. LESSONS LEARNED 5.1 Lessons for Project Planning and Implementation To be noted both by USAID and by implementers, TLMP would have benefitted considerably from following essential planning and implementation elements which are normally part of a development project. First, at the planning stage, there should have been a logical framework analysis to develop appropriate aims, goals and specific indicators that could be assessed during implementation. Then, as TLMP was rolled out in Senegal, there should have been both monitoring and evaluation at each step, and a mid-term review not limited to financial reports. This would have ensured that the original goals of the project were being adhered to, or that there was good reason to diverge from them. Likewise, it would have helped ECSU and the MOE as well as other stakeholders to learn from their experiences and adapt so that follow-up activities (such as checking to see that the TLMs had not only reached the schools but were indeed being used by teachers and learners) would result in clear, budgeted actions. Adequate planning and regular monitoring would have uncovered the flaws of the project, including but not limited to: failure to develop TLMs that could serve as designated textbooks rather than support materials; failure to thoroughly pilot-test the materials; and failure to consider the needs of most teachers, either through development of teachers’ guides or follow-up of training or both. These flaws resulted in what we see as a failure of TLMP in Senegal to reach its full potential. Another lesson learned is that the project budgeting, financing, and procurement processes were not efficient from a planning point of view, particularly in the development and piloting of materials, which when done responsibly takes well over one year. It was also inefficient for the planning of print runs, whose cost-efficiency (and adherence to the initial bid) depends on advance acquisition of paper, toner and other supplies and the meeting of deadlines from the book development side. 5.2 Lessons for Implementers The most important lesson was that it was not a good idea to begin with materials for American students. This was exacerbated by the fact that the early TLMs, produced under AEI, were machine￾translated from English to French. This may have seemed efficient but introduced too many inaccuracies, resulting in hard work for the Senegalese specialists and misunderstandings between American and Senegalese collaborators. Further challenges inherent in this U.S.-to-Senegal direction were that images were often culture-bound (not relevant to Senegalese learners), that books were not fully aligned with the national curriculum, and that the levels were too high. There were also issues associated with quality control of the TLMs. Later materials were created by Senegalese specialists, who received limited feedback from individuals at ECSU, and there was no piloting; meanwhile, the validation processes used failed to capture and correct these flaws. The rushed process from conception to distribution did not facilitate thoughtful application of project goals (such as raising capacity among stakeholders) nor did it cultivate effective practices (such as developing criteria for textbook evaluation such as gender balance, culturally appropriate illustration, etc.). There are also intellectual property issues. 5.3 Lessons for USAID There was a fundamental flaw in the development hypothesis – an assumption that production and distribution of textbooks by itself will make much of a difference. Performance Evaluation of TLMP – Senegal 19 The mere distribution of TLMs does not ensure that they will be used at all, let alone that they will be used the way they were intended. Teachers, who have not had access to TLMs, whether as students themselves or through training, do not automatically know how to teach with them. During the course of the field work, TLMs were found in storage rooms full of dust or in libraries still in boxes or lying pristine on shelves. The most capable teachers were clearly able to make use of the TLMs; the others were not. Teachers’ guides along with TLM-related trainings are necessary elements of the process, not optional luxuries. 6. RECOMMENDATIONS 6.1 Materials Improvement The implementation of the expansion phase of TLMP in Senegal suffered from the lack of adequately rigorous and systematic planning processes as well as from lack of adequate monitoring and evaluation as the project progressed.  Ensuring high-quality design and lay-out. Materials development specialists should be engaged so that important steps in TLM development are all followed—including, e.g., piloting with students, checking for gender balance, stereotyping, etc. in content and illustrations. In addition to subject matter experts, graphic design assistance should be called upon for expertise in choosing or developing illustrations, photographs, maps and charts.  Realistic timelines. Frequently, there was a very tight timeframe within which materials were to be planned, developed, and finalized. This resulted in work that was rushed and not adequately reviewed plus costly delays in production.  Piloting and Usability. It was not clear how many TLMs actually were piloted in classrooms, or whether schools operating in different types of environments were used for the piloting. Teachers of representative ability should be involved as part of the materials development and piloting process in order to identify areas that could present challenges to comparable teachers.  Teacher’s guides. It is important to develop a teacher’s guide for each TLM, since not all teachers will be familiar with the TLMs through pre- or in-service training or with the pedagogical principles being promoted. In Senegal, as in other developing countries, many teachers have inadequate pre￾service training and some may not even be certified to local standards. Such teachers tend to fall back on the methods by which they were taught as students; moreover, in many countries the teachers may themselves have gone through much of their own schooling with little contact with textbooks.  Workbooks. Some materials were developed with the expectation that they would be accompanied by workbooks in which learners would write. This inherently makes the workbooks usable only once, with a need to replace them every year. Materials should be developed instead such that learner responses can be put into regular copybooks, which are relatively readily replaceable, including through the use of plain paper if need be. 6.2 Systems Improvement  Ensure closer involvement of MOE bodies concerned with learning material development and approval. It appears that there was not a very close collaboration between INEAD and other MOE entities concerned with curriculum and textbooks. In fact, the choice of producing supporting manuels d’appoint was a way of circumventing the admittedly complex procedures of formally approving textbooks as part of the official, examinable curriculum. While this saved time, it appears that the MOE was not proactive in supporting the distribution process or in clarifying to its personnel what the purpose of the TLMs was for teachers and students. This would be avoided with better collaboration and greater willingness to follow the appropriate procedures.  Support the MOE in orienting school inspectors so that they can follow up in all school Performance Evaluation of TLMP – Senegal 20 districts to ensure that students have use of the TLMs. Whatever information was provided at the time of distribution to the school districts was inadequate to ensure that school directors and teachers understood the aim of getting materials into students’ hands. A clear directive from the MOE, along with a budget for orientations, will help get TLMs out of cartons, off shelves and into teachers’ and learners’ hands, to be used as resources in and outside of classrooms. As a reference, we have listed some points that could be clarified in ANNEX G: List of Clarifications on the TLMs for Possible MOE Distribution.  Support development of a chart that links the TLMs to the national curriculum. To provide added value to the TLMs as support materials, a chart showing teachers how to link the TLMs (as well as other available resource materials) to the competencies in the national curriculum would show that they are part of an officially sanctioned program of the education system. Such an activity could also raise capacity among subject specialists at the MOE to help teachers implement the latest curriculum reform. Doing this for each subject at each level of schooling would help educators to understand progression/sequencing of skills and would address the challenge some teachers expressed with having one TLM for more than one grade level. The chart for each subject and grade level could use a template with columns for: competencies in curriculum, learning indicators, pages in TLM, lesson name, and assessment.  Link TLMs to teacher professional development. Since the TLMs for middle and secondary school levels were all support materials rather than standard textbooks, since some of their content was considered challenging, and since not all teachers knew how to use books as resources in the first place, a link to teacher in-service and pre-service courses could bring the TLMs into better circulation and raise teacher capacity. Even if focused simply on using resource materials, training is justified for all teachers and could be made part of USAID/Senegal’s education strategy.  Support school-based management of resources. USAID might consider developing support programs at Senegalese middle and secondary schools where subject-based departments (such as mathematics) collaborate on the utilization of TLMs and other resources. USAID support to secondary school directors and department heads might help the MOE to create criteria and models for future school-based decision making. Performance Evaluation of TLMP – Senegal 21 ANNEX A. SCOPE OF WORK DESCRIPTION/ RESULTS-ORIENTED STATEMENT OF OBJECTIVES (SOO) Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania I. BACKGROUND TLMP contributed directly to USAID’s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners‘ management capacity in the education sector. Each Minority-Serving Institution (MSI), based upon the provisions noted in their Cooperative Agreement (CA), was responsible for managing and implementing the TLMP in a specific country and with achieving specific output results. Each MSI was also responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in primary schools in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MOE) and other local specialists. The main objectives of the TLMP were to: 1) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities in primary schools within SSA, 2) strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). TLMP Cooperative Agreement History  TLMP Ethiopia: USAID Cooperative Agreement RLA-A-00-09-00035-00; In coordination with local entities, Alabama Agricultural and Mechanical University (AAMU), over 3 million English for Ethiopia textbooks were produced and disseminated for grades 1, 6, 7, and 8. Over 132 teachers were subsequently trained to use the materials in classroom settings.  TLMP Ghana: USAID Cooperative Agreement RLA-A-00-09-00036-00; In coordination with local entities, Chicago State University (CSU) has trained 260 teachers in using the developed materials. Over 6 million materials and textbooks have been created and distributed for students up to grade 3 in mathematics, environmental science, and English.  TLMP Malawi: USAID Cooperative Agreement RLA-A-00-09-00033-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over five million supplemental reading books, teachers’ guides and training materials and trained nearly four thousand teachers on methodological classroom usage.  TLMP Senegal: USAID Cooperative Agreement RLA A 00-09-00037-00; In coordination with local entities, Elizabeth City State University (ECSU) produced and distributed over 2.6 million materials in French for grades 1-11 in a variety of subjects, Performance Evaluation of TLMP – Senegal 22 including science, mathematics, and language arts. Over 160 teachers were said to have been trained on utilizing the materials as part of their curriculum.  TLMP South Africa: USAID Cooperative Agreement RLA-A-00-05-00079-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over 1.4 million materials in 11 languages for grades 4, 5, and 6, as well as trained over 6,000 teachers. The work was completed in 2009.  TLMP Tanzania: USAID Cooperative Agreement RLA-A-00-09-00034; In coordination with local entities, South Carolina State University (SCSU) created and disseminated over 1.1 million materials for secondary level usage in the fields of science and mathematics. Over 1,200 teachers were trained. The Contractor will be provided with each institution’s Cooperative Agreement by each individual institution, which will include the relevant scope of work. The Contractor will be required to obtain other pertinent documents as necessary. II. PURPOSE The purpose of this SOO is to support the Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania. III. SCOPE OR MISSION Task 1 – Data Collection Task 2 – Data Review Task 3 – Coordination and Management Task 4 – Site Visit Task 5 – Data Analysis IV. PERFORMANCE OBJECTIVES / DESIRED OUTCOMES The Contractor shall provide all labor, equipment, supplies and materials, and travel necessary to conduct Textbooks and Learning Materials Program Evaluation (PE). The PE is intended to satisfy the following objectives:  validate stated program goals and impacts;  assess the results achieved for each host partner country in relation to intended program targets, as well as standardized and variable indicators by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions, Ministries of Education (MOs), etc.), student achievement, teacher performance, amongst other criteria, in each host partner country;  determine if in-country institutions (with support from U.S.-based Minority-Serving Institutions [MSIs]) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution;  review allocated USAID funding in terms of usage and overall cost effectiveness;  highlight specific program accomplishments per MSI-host country partnership; and Performance Evaluation of TLMP – Senegal 23  document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the New Agency Education Strategy http://www.usaid.gov/our_work/education_and_universities/documents/USAID_ED_Str ategy_feb2011.pdf 6 V. OPERATING CONSTRAINTS / LIMITATIONS We anticipate that Awardee would complete one site visit per country and that the site visits would take no longer than 10 days each. There is not a requirement for specific key personnel or a combination of key personnel to complete the site visits; however consistency in terms of personnel for the site visits is preferred. The Contractor shall perform the PE in accordance with USAID ADS 203 and the new USAID Evaluation Policy published in January 2011. The USAID ADS 203 Performance and Monitoring Guidance can be found here: http://www.usaid.gov/policy/ads/200/203.pdf. The new USAID Evaluation Policy can be found here: ttp://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf Monthly Status Reports. The Contractor shall provide written reports to the USAID COTR or his/her designee on the progress of the work, contacts made, and problems encountered on a monthly basis. They should be submitted by the last business day of every month. Comment Reponses. Comments will be provided to the Contractor electronically. The Contractor shall prepare comment responses that clearly state the actions taken to incorporate the comment or show the changes in a redline and strikeout version of the revised report. The Contractor may contact the reviewers for clarification. Unresolved technical issues shall be coordinated with the COTR. ANNEX B. DATA COLLECTION INSTRUMENTS The following is a description of the stakeholders the in-country team targeted in the field and the themes on which we questioned them: Regional education heads 1. Regional education inspectors (Inspecteurs d’Académie)  Their evaluation of the TLMP school books with regard to their alignment with the national curriculum  Management of the books (reception, storage and distribution) Departmental education heads 2. Departmental inspectors (Inspecteurs départementaux) - rural - urban  Their evaluation of the TLMP school books with regard to their alignment with the national curriculum  Management of the books (reception, storage and distribution)  Impact of the books on educational quality and student learning Subject specialists/Trainers 3. Specialized inspectors (Inspecteurs des spécialités) in language, math and science 4. Heads of regional training centers (Responsables des Pôles régionaux de formation) 5. Travelling pedagogical advisors (Conseillers pédagogiques itinérants)  Their evaluation of the TLMP school books with regard to their alignment with the national curriculum  Management of the books (reception, storage and distribution)  Impact of the books on educational quality and student learning  Participation in any training in the use of the books with regard to the reported cascade model  Their evaluation of the capacity/level of teachers using the TLMP books School heads 6. Middle school principals (Principaux de collèges) 7. Elementary school principals (Directeurs d’écoles)  Their evaluation of the TLMP school books with regard to their alignment with the national curriculum  Management of the books (reception, storage, care and lending policies)  The impact of the books on educational quality and student learning  Their evaluation of any training given to the teachers Teachers 8. Elementary school teachers 9. Middle school teachers 10. Secondary school teachers  Their evaluation of the TLMP school books with regard to their alignment with the national curriculum  Distribution, care and lending of the books  The impact of the books on educational quality and student learning  Their evaluation of any training given to the teachers Parents 11. PTA representatives 12. Individual parents  The impact of the books on educational quality and student learning (if they have seen the books) Students 13. Elementary students 14. Middle school students 15. Secondary students  Their evaluation of the attractiveness of the books, the images and the readability of the texts  The impact of the books on their learning and school results Others (added in the field) 16. IA storage heads (comptables magasiniers) 17. IDEN storage heads 18. School librarians  Management of the books (reception, storage, distribution, care and lending policies) Performance Evaluation of TLMP – Senegal 25 ANNEX C. MASTER DATA COLLECTION INSTRUMENTS This annex contains the revised master set of instruments used as a model for the TLMP evaluation. These questions were adjusted based on the circumstances of TLMP implementation for each country. In Capital and Adjacent Locations 1) USAID Mission – Education Team  What has been the mission’s role in implementing TLMP? What types of support activities have you provided to the project?  How does TLMP fit in with other USAID education program/priorities in this country? How do you see the expertise developed in textbook production by the MOE being leveraged to obtain other, similar grants? What would prevent this from happening?  Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbook and learning material development?  How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable?  What specific challenges has TLMP faced in Ethiopia? How were they addressed?  How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outputs and outcomes of TLMP? What would you change?  How satisfied were you with your relationships with Alabama A&M and the work they did? What suggestions do you have for overall improvement?  What were the lessons learned for the mission in overseeing the TLMP? Would the mission support a similar project in the future? 2) MOE Administrators (triangulate with different administrators at each level)  What is your current position? How long have you been in this position? For how long have you been working in education? In what positions?  What was your particular involvement in TLMP? During what period?  What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice.  How did you decide which staff members/departments were to work on the TLMP? Were they seconded to the project or were project responsibilities added to their normal tasks? Were any incentives provided for participation? What?  How was it decided which schools would receive the TLMs? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training?  What types of policy change has the MOE instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training?  What other types of teacher training does the MOE provide? How frequently?  How do the woreda inspectors assess teachers? Were they trained in the use of TLMs? How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLMs included in teacher training (either pre-service or INSET)?  How was the decision made to include supplementary/complementary readers in the TLMP? Who made the decision that these materials should be based in folk tales? How did the process of developing these materials differ from that used in developing the TLMs for grades 1-4 English classes (textbooks)? Was there an advantage of one approach over the other? Please explain.  What will the MOE do to continue the production of TLMs now that the project has ended? Have any other donors been found to continue this activity? What new textbook policies have been developed as a result of TLMP? Performance Evaluation of TLMP – Senegal 26  Is the MOE ready to use its own funding in the creation of TLMs in the future? In the redevelopment and printing of the grades 6, 7, and 8 materials? What would prevent this from happening?  How has the material presented in the TLMs been included in national exams? Has performance on exams improved with the use of TLMs? How do you know?  How was TLMP monitored by the MOE? What indicators did you use? How often did you go to schools to observe the use of TLMs? How was TLM production managed and monitored?  How has the MOE benefited from TLMP? How has it been challenged? Were there any negative effects of TLMP? If so, what were they and how were they addressed?  How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? (Why?) If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different?  How satisfied are you with the collaborative relationships established with AAMU? How could they be improved? 3) Material Developers/Curriculum Specialists  What is your current position? For how long have you had this position? For how long have you been working in this area (e.g., subject matter, curriculum and instruction, grade level)? What is your educational/training background in this area?  How did you become involved in TLMP and at what point in the process? What was your specific role at the outset? At the end of the project?  What was the composition of the writing/production team? What types of expertise was represented? What other expertise was needed, in your view? How were the members of the team compensated for their activities?  How did the production process and personnel differ between the development of the textbooks and the development of the supplementary reading materials (folk tales)? Did one process have an advantage over the other? If so which one, and how were results different? Would you recommend one process over the other for future book production?  How often did the two US- and Ethiopia-based teams meet? What were the results of these meetings?  In developing TLMs, how did you ensure conformity with the national curriculum in terms of subject matter and grade level? What cross-cutting themes did you include?  What type of local and international review process did the production team have to go through?  How did you obtain illustrators for the TLMs?  How satisfied are you with the collaborative production process between yourselves and AAMU? What worked well/did not work well? What would you change to improve the process?  How do you think the production process can be improved in the future? 4) TLMP Program Administrators (Field Offices) Background Information  Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? Did you have any assistance doing this? From USAID? Other stakeholders? Materials Development and Distribution Process  What process was used in the creation of TLMs? (describe both the textbooks and the supplementary reading materials) What did the MOE do (specify unit)? What did the project Performance Evaluation of TLMP – Senegal 27 do? What challenges emerged in your work with the MOE? How were they resolved? How did you liaise with all stakeholders?  In implementing the project, what role did the MOE play (specify unit)? What roles did your office play? What guidelines did the MOE provide?  How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? What work are they now able to take on with other clients? What other services did the printer provide?  How was the decision made about which districts/schools would receive the materials? Who was responsible for distribution? What was the distribution chain? How did you monitor distribution?  How was teacher training conducted? Who provided the training? For how long? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Gender distribution of those who attended? Did TTC faculty attend? From which TTCs? Did university faculty attend? How many? From which universities?  What other in-service teacher training is provided by the MOE? In what format? How did the TT for TLMP differ from the TT for other areas?  In conducting TOTs, were teacher salaries supplemented? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much?  How successful was the cascade training model? How many teachers did those who participated in the TOT actually train on the use of the TLMs? Project Management and Outcomes  What was the TLMP management structure in Ethiopia? What types of services did you provide to the MSI and other stakeholders?  What was the composition of the Ethiopia PAC? How often did they meet? What decisions did they make? How did these decisions affect the project?  How often did you visit project implementation sites? What types of monitoring did you perform? How frequently?  What types of assistance did USAID provide to you?  What kind of networks and/or public-private partnerships did you create? How are you collaborating with other stakeholders?  From your point of view, as a result of the TLMP project, what has changed either positively or negatively? How have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials?  What types of policy changes, if any, have you observed as a result of project work?  What accomplishments are you most proud of? If a TLMP-type project were to be undertaken again, what would you do differently? 5) Printers/Publishers  Tell me about your operations before you were granted the TLMP contract and how they changed (either positively or negatively) as a result of TLMP participation?  How did the contracting occur with AAMU? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these?  What instructions were you given on how to distribute the TLMs? From whom? What kind of difficulties did you encounter in keeping to the distribution schedule?  When/how did you distribute the TLMs after they were produced? To whom did you distribute them? How many TLMs were delivered to each receiver? What kind of tracking/delivery system did you establish? What kind of challenges did you have in distributing the materials? Performance Evaluation of TLMP – Senegal 28 How were these overcome?  How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have your hired? What new equipment have you purchased? What other inputs would you require to take on more textbook production projects?  How satisfied were you with the relations established with AAMU? How could they be improved? In Field 6) Regional Education Offices/Primary Education Advisors  What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? What is your educational/ training background related to this work?  What was your particular involvement in TLMP?  What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice.  How many of each TLM did you request for your district/region/province? (Subjects, languages, levels?) How did you calculate this number for appropriate grade level students? If you had any surplus, what did you do with the materials? If you had any shortfall, what did you do?  What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received?  What instruction did you give to each school about how the TLMs were to be used? How many teachers in your district attended the TOT? How were these teachers chosen? How many of these teachers went on to teach others through the cascade model? How many others were trained? How would you rate the quality of the training they provided? Based on what evidence?  How were inspectors instructed on how to evaluate teachers using TLMs? Do all children have/use the books produced?  From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank the TLMs produced for this project? What do you recommend for materials improvement? Program improvement? In Schools: 7) Principals  What is your current position? How long have you been a principal at this school? How long have you been a principal? In how many schools?  What is the overall economic status of the people in this community? How do they generate income? What is the composition of most families/households? How big a problem is HIV/AIDS in this community? About what percentage of your students are Orphans or Vulnerable Children (OVC)?  Do families send their girls to school as often as their boys? What gender-based trends do you see in enrollment? Has your school done anything to make teachers or families more aware of gender disparity in enrollment/attendance? If so, what have the results been?  What is the linguistic background of the learners at this school? What language(s) do children speak when they enter school? Is this language the language of instruction? If yes, until which grade? At what grade does English become the language of instruction? Do you believe your students are adequately prepared in English to learn entirely in English? What needs to be done to prepare students better?  In terms of teacher mobility, has there been any increase or decrease in the rate of teacher Performance Evaluation of TLMP – Senegal 29 transfer since they attended a TOT or were trained in the use of the TLMs? What are the most common reasons why teachers request a transfer? [If appropriate, you can prompt, e.g., “Does this have to do with obtaining a higher salary, improving living conditions, or other factors?”]  How many of your teachers/administrators participated in the development of TLMs? Where was the work undertaken? For how long?  How many of each textbook and workbook did you request for the school? How many of each text/work books did you actually receive per grade level? If you had any surplus, what did you do with the materials? If you had a shortage, what did you do? When during the term were the books received?  How many of your teachers attended the TOT in the use of the TLMs? How did you choose these teachers? How many of these teachers went on to teach others? How many other teachers received the training from a teacher who attended the TOT? What were the teachers’ reaction to/opinion of the training? Did you attend the TOT yourself? If so, what was your opinion of the training?  What other types of teacher training does the MOE provide? How often? Are those who attend expected to pass on (cascade) what they have learned to their colleagues? Do they receive any incentive to do this?  How has the cluster center training and resource system enhanced the ability of teachers to be more learner-centered? How has the TLMP enhanced the adoption of learner-centered teaching practices?  What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? 8) Classroom and Head Teachers  What is your current position? How long have you been teaching this subject at this grade at this school? How long have you been a teacher? What other classes have you taught before? At what grade level? What is the level of education you have achieved? What qualifications do you have to be a teacher? (certificate, diploma, degree)  In this Region, which languages are used for instruction at which grade levels? In which language is initial literacy (reading and writing) learned? At what grade do children start learning English? At what grade does English become the language of instruction?  What is your greatest challenge in teaching English? In any other Mother Tongue languages? What would you like to improve?  What role, if any, did you play in producing the TLMs? Please explain.  How many students do you have in your classes? Specify class and number of students. What is the age range of your students in each class?  What non-TLMP textbooks do you have to teach? What non-TLMP workbooks do you have to teach? Does every child have a textbook? Workbook? What do you do when you don’t have enough textbooks or workbooks for each child? Do you have a teacher’s guide for each of the textbooks/workbooks? If not, what do you use?  What TLMP textbooks do you have to teach? What do you do when you don’t have enough TLMs for each child? Do you have a teacher’s guide for each of the books? If not, what do you use?  When did you receive the TLMs for your classes? How many were you provided? From whom did you receive them? How did you distribute them to your students? How many students must share a textbook? A workbook? Are students allowed to write in their workbooks?  When did you receive training on the use of TLMs? How long did it last? Did someone from the TLMP project or another teacher deliver the training? What is your impression of the TLMP training? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank Performance Evaluation of TLMP – Senegal 30 the TLMP training you attended? Please explain your reason for this choice and identify areas where it could be improved. If you did not attend any training related to the materials, how did you learn how to use them?  Were you able to use the textbooks/workbooks after the training? Did you feel you needed more training? In what?  Do you believe the TLMs were aligned with the curriculum? If not, how should the materials be changed?  Do you believe the TLMs were properly sequenced (go from easiest to hardest)? What would need to change if they were not?  For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLMs?  What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not?  How “ready” were your students to use the materials distributed? Was the grammar and vocabulary at a level that could be understood by students? What type of difficulties do the students have in using the materials? How should the program overcome these difficulties?  What changes (either positive or negative) have you observed and recorded in girls’ and boys’ achievement on annual or national examinations since the TLMP workbooks/ materials were introduced? Do you think these changes are attributable to the use of the TLMs? What evidence can you give for this?  What is your opinion of the TLMs in so far as their attractiveness to students? On a scale of 1- 4, with 1 being very attractive, and 4 being not very attractive, rank the materials. Please explain your reason for this choice.  What is your opinion of the TLMs in the ways that they depict girls and boys? Do they represent them in non-traditional /traditional roles?  Is there anything about the TLMs that you would change? What? Why?  In using the TLMP workbooks/materials, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice.  What is the greatest challenge your students experience in using the TLMs?  What do you think is the overall impact of the program on your students? What kind of difference does it make in learning for a child to have textbooks/workbooks? What do you think could be improved to have an even larger impact?  What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the supplementary readers? 3) In the delivery of the program? IF ALSO PROVIDED TOT, ANSWER THE FOLLOWING ADDITIONAL QUESTIONS:  If you attended the TLMP TOT training, how many other teachers did you teach afterward? Where did you conduct this training? What worked well? What difficulties did you encounter in doing this? Were you provided with any follow-up support after you received the training? What type? How often? By whom?  Why do you think you were chosen to be a trainer?  On a 1-4 scale, with 1 being very satisfied and 4 being not satisfied, how would you rank the training you received? Please explain your reason for this choice. What feedback, if any, did you receive from observers on your training style and approach? How did this feedback improve your own teaching? Performance Evaluation of TLMP – Senegal 31  How familiar were you with the TLMs before you delivered the training? What materials were you provided to be a trainer? What materials did you provide to your trainees? How confident were you after the TOT that you could teach others in how to use the TLMs? What else did you need?  Was the length of training adequate for you to cover all topics well? What area required more time?  Did you receive any compensation for conducting this training? 9) CLASSROOM OBSERVATIONS OF TEACHER USE OF TLMs/SENEGAL Used in the Regions of Kaolack, Kaffrine and Fatik only School: _________________________ Village/Town/City: ________________________ Teacher Sex: _____ M _____F Grade Level: ______________________________ No. of Students: ______ M ______ F ______Total Languages spoken in this community: __________________________________________ Home Language/Mother Tongue of learners in the class: ___________________________ Home Language/Mother Tongue of teacher: _____________________________________ No. of books: _______________ No. & type of learning materials: __________________ Indicator Observed Not Observed Other Teaching Using TLMs Instruction 1. The teacher has prepared an authentic lesson that uses the TLMs for the class period. 2. Students have the appropriate TLMs and are ready to use them in class activities. (Note ratio of materials to learners.) 3. The teacher explains the goal and purpose of the class lesson to the students. 4. The teacher identifies, pronounces and defines any difficult vocabulary before teaching the lesson. 5. The teacher begins the class activity with questions that review previous activities using the TLMs and draws on the prior knowledge of the students. 6. The teachers uses learning aids/materials produced by TLMP 7. The teacher can read and explain TLM content to the students 8. Students can read and understand the subject matter in the TLMs. 9. Students are actively and interactively engaged with the teacher in the use of TLMs (Q&A, group work, workbook practice, continuous assessment). 10. The teacher gives and corrects homework using the TLMs. 11. The teacher shows evidence of having used the Teacher’s Guide in Performance Evaluation of TLMP – Senegal 32 Indicator Observed Not Observed Other presenting the lesson 12. Students and teachers use mother tongue/English (French) when asking and responding about TLMs (circle which language) 13. Teacher demonstrates personal mastery of English TLMs/Artifact Inventory 14. Lesson objectives are written on the board in English 15. Learning aids/materials are posted in the classroom (TLMP produced and others). 16. TLMs are locked up in the cupboard. 17. Word walls display key words in English 18. Sentences appear on the chalkboard or on a chart 20. Students write words and sentences in their exercise books (demonstrating evidence of having pencils/pens and exercise books) 21. There is evidence that teachers (or peers) mark exercise books in a process of continuous assessment Comment____________________________________________________________________________ ____________________________________________________________________________________ Performance Evaluation of TLMP – Senegal 33 ANNEX D. SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES People Interviewed at Elizabeth City State University Name Title Date(s) Mr. Abdou Maty Sene TLMP Project Director 4-6 Dec. Dr. Willie Gilchrist Chancellor 5-6 Dec. Dr. Anthony Brown Vice-Chancellor 5 Dec Dr. Ali A. Khan Provost 5 Dec Dr. Johnny L. Houston Former TLMP Director; Consultant 5-6 Dec Dr. Chérif Seck Director of International Programs & TLMP Coordinator 5-6 Dec. Ms Eundene Faulks Asst. Director, Sponsored Programs 5 Dec. Mr. Joshua Lassiter Finance Dept. 5 Dec. People Interviewed in Senegal Title Name Date(s) Place Project leadership ECSU TLMP Project Manager Mr Abdou Maty SENE 28-30 Jan, other days Dakar USAID representatives Chief Education Officer Mr Pape SOW 28 Jan, Dakar Education Officer Ms Michelle CHEN 28 Jan, Dakar MEN (national level) MEN Secretary General Mr.Baba Ousseynou LY 28 Jan Dakar Former Secretary General Mr Mafakha TOURE 28 Jan Dakar Dir Elementary Ed Mr Abdou DIAO 28 Jan Dakar Dir Middle School Ed Mr Ibrahima NDOUR 28 Jan Dakar Coord DEMSG Mr Papa SENE 28 Jan Dakar Director of Inspection (IGEN) Mr Ndiogou FAYE 31 Jan Dakar Printing company representatives SIPS Technical Director Mr Maher GHANDOUR 29 Jan Dakar Polykrome Director General Mr Walid ATTIEH (and staff) 29 Jan Dakar Authors/validators Science expert Mr Cheikh Tidiane DIOP 29 Jan Dakar Math expert Mr Samba DABO 29 Jan Dakar Middle school math teacher Mr Oumar SY 29 Jan Dakar Parent/teacher association (national) President of FENAPES Mr Bakari BADIANE 31 Jan Dakar Secretary general of federation Mr Dam SEK 31 Jan Dakar Secretary of communications Mr Mohamed Diem SESAY 31 Jan Dakar Regional (IA) directors and staff (including inventory/storage chiefs) IA Dakar Whole team Academy inspector 1 Feb Pikine Assistant inspector (IA Pikine) Mme DIABY 1 Feb Pikine Mr Musa Diaba HASSAN 1 Feb Pikine IA Saint-Louis CB & AS Performance Evaluation of TLMP – Senegal 34 Academy inspector Mr Ndar FALL 4 Feb Saint-Louis Storage Chief Mr Abdou Karim KA 4 Feb 4 Feb Saint-Louis 4 Feb IA Kaolack EA & BD Academy inspector 4 Feb 4 Feb 4 Feb IA Louga CB & AS Academy inspector Mr Lamine SARR 5 Feb Louga 5 Feb 5 Feb IA Kaffrine EA & BD Academy inspector 5 Feb 5 Feb 5 Feb IA Thiès CB & AS Assistant academy inspector Mr Ibrahima BAR 6 Feb Thiès Comptable matières de l’IA Mr Mamadou KANE 6 Feb Thiès President of the APE (Association of students’ parents) of Thiès Mr Mama KANTE 6 Feb Thiès IA Fattick EA & BD Academy inspector 6 Feb 6 Feb 6 Feb IS, CPI, etc. (subject specialists/trainers) Saint-Louis region (they cover Louga as well) Science Inspector (CPI) Mr Déthie Chiendella FALL 4 Feb Saint-Louis Science Inspector (Regional training center) Mr Mactar FALL 4 Feb Saint-Louis Coordinator of the regional training pole (for middle & secondary ed) Mr Amadou NDIAYE 4 Feb Saint-Louis Thiès region Trainer (former CPI) in sciences (SVT) Mr Papa Birane THIANDOUM 6 Feb Thiès Departmental (IDEN) directors and staff (including inventory/storage chiefs) IDEN Saint-Louis Departmental inspector Mr Samba Laobé DIOP 4 Feb Saint-Louis 4 Feb Saint-Louis IDEN Performance Evaluation of TLMP – Senegal 35 Departmental inspector 4 Feb 4 Feb 4 Feb IDEN Louga Departmental inspector Mr Amady KEBE 5 Feb Louga Comptable matières Mr Ismaïla TALL 5 Feb Louga 5 Feb IDEN 5 Feb 5 Feb 5 Feb IDEN Thiès Departmental inspector for Thiès city Mr Talla FAYE 6 Feb Comptable matières Mr Khaty DIAGNE 6 Feb 6 Feb IDEN 6 Feb 6 Feb 6 Feb Schools, teachers, students and parents In Saint-Louis IDEN Mpal Elementary School Director: Mr Sidate DIOP Teachers Mr Insab BADJI, Ms DIABO, Ms Bintu (bilingual) 4 Feb Mpal Ngom Middle School Principal: Mr Ngange GUÈYE Grade 9 class Grade 10 class 4 Feb Ngom 4 Feb In __ IDEN Inspector 4 Feb In Louga IDEN Director of Keur Serigne Bara Elementary School Ms Khady Ndiaye GNIGUE 5 Feb Louga CM1 teacher at KSB Elementary School Mr Mamadou DIAW 5 Feb Louga Students at KSB Elementary School CM1 students 5 Feb Louga Principal of Elhadji Djily Mbaye Middle School Mr Abdoulaye DIALLO 5 Feb Louga Teachers responsible for EDM school library Mr Babacar NDOYE Mr Alias Ismaïla SYLLA Louga Teachers of EDM math & science (SVT) Mr Mamadou THIAM Mr Abdoul CISSE 5 Feb Louga Father of middle school student in 4ème Mr Saliou SEYE 5 Feb Louga In Kaffrine IDEN 5 Feb Performance Evaluation of TLMP – Senegal 36 5 Feb 5 Feb In Thiès IDEN Director of Saïb Ndoye Elementary School Mr Cheikh MBENGUE 6 Feb Thiès Gr 6 teacher at SN Elementary School Mr Modou NDIAYE 6 Feb Thiès Gr 5 teacher at SN Elementary School Ms THIAM 6 Feb Thiès Gr 5 and Gr 6 students at SN Elementary school One gr5 class and one gr6 class 6 Feb Thiès Principal Mamadou Diaw Middle School (formerly Château d’Eau) Mr Amary MBAYE 6 Feb Thiès Math & science teacher at MD Middle School Mr Paul DIOP 6 Feb Thiès Librarians at MD Middle School Ms Pallé DIOP (resp) Ms Didé Mbodji COULIBALY 6 Feb Thiès IDEN 6 Feb 6 Feb 6 Feb Performance Evaluation of TLMP – Senegal 37 Schedule of Activities in Senegal, 27 Jan – 9 Feb 2013 Date/Time Schedule Meeting with Accompanied by 26 Jan, Sat (evening) Eric: arrival in Dakar Hotel Ibis 27 Jan, Sun (morning) Carol: arrival in Dakar Hotel Ibis 16:30-18:00 First team meeting at Hotel Ibis Dakar: self￾presentations, introduction to evaluation, methods and schedule. Eric ALLEMANO Carol BENSON Alhousseynou SY Babacar DIOUF 28 Jan, Mon Activities (whole team: Eric, Carol, Babacar and Alhousseynou) 9:00-11:30 Meeting at USAID with the Chief Education Officer: Background to TLMP and this evaluation, logistics Pape SOW Abdou SENE Former Chief of Party TLMP 12:00-12:20 Meeting at MOE with Secretary General: We presented evaluation, he gave impressions. Secrétaire Général : Mr. M. Baba Ousseynou LY “ 12:35-13:35 Meeting at MOE with Director of Elementary Education: Impressions, materials published under TLMP Directeur de l’Education Elémentaire : Mr. Abdou DIAO “ 14:10-15:10 Meeting at MOE with Director of Middle School Education (Dir de l’Enseignement moyen sécondaire général/DEMSG/collège): Impressions, recommendations. Dir DEMSG Ibrahima NDOUR “ (Lunch) 16 :30-17 :25 Meeting at home of retired Secretary General of MOE: His impressions of national textbook policy, recommendations. Mr. Mafakha TOURÉ “ 29 Jan, Tues Activities (whole team) 9:00-9:30 Team meeting at Hotel Ibis. Abdou SENE 10:25-11:20 Meeting at SIPS (Société Industrielle de Papetérie au Sénégal/printing company) with Technical Director: Impressions of working with TLMP, recommendations. Dir Technique Maher GHANDOUR “ 11:50-12:55 Meeting at Polykrome (printing company) with Director General and staff: Their experiences with TLMP and recommendations on printing, distribution. Dir Général Walid ATTIEH “ (Lunch with team and authors at Hotel Novotel) 14:25-17:30 Meeting at Hotel Novotel with authors/validators/teacher: Process, collaboration, impressions, recommendations. Cheikh Tidiane DIOP, science expert Samba DABO, math expert Oumar SY, middle school math teacher in Dakar 17:30-19:00 Team meeting to synthesize findings, challenges, recommendations and questions Eric ALLEMANO Carol BENSON Performance Evaluation of TLMP – Senegal 38 Alhousseynou SY Babacar DIOUF 30 Jan, Wed 8:00-8:45 Breakfast meeting at hotel (EA&CB) 8:45-13:00 Work at hotel on report, translations of instruments, bank issues, planning of trips (EA&CB) while preparing school visit permissions and travel (AS&BD). (Lunch) 14:00-19:00 (Continued, team meetings to plan fieldwork) 31 Jan, Thurs 8:00-9:30 Team work at hotel. 10:15-11:15 Meeting at Inspectorate headquarters (Inspection Générale de l’Education Nationale) in Dakar. Mr. Ndiogou FAYE, Inspector￾General of the MoE 12:00-14:30 Lunch meeting – team – and planning 15:00-16:15 Meeting at Fédération Nationale des Parents d’Elèves et des Enseignants du Sénégal (National Federation of PTAs). Bakari BADIANE, President 2 Mr. Dam SECK, Secretary General Mohamed Diem SESAY Secretary for Communication, Training and Research 1 Feb, Fri 8:00 Team: Depart hotel for outer Dakar (Dept. of Pikine) 9:00-10:00 Meeting at IGEN (Inspection Générale de l’Education Nationale) in Guédiawaye Mme Ami Sene DIABY Deputy Inspector Musa Diaba HASSAN, Assistant 10:15-12:20 Meeting with Guédiawaye middle school principal and staff. Visit to library. Observation of SVT demonstration lesson by Mr DIAW. Cheikh Tidiane SY, Principal Cheikna FALL (Math teacher) Abdou SENE (Math math teacher) Demba DIAW (SVT teacher & librarian) Lunch Team lunch and planning of field visits. 14:00-17:00 Planning, documentation of meetings. Eric ALLEMANO Carol BENSON Alhousseynou SY Babacar DIOUF 2 Feb, Sat 8:00-16:00 Eric & Carol: Breakfast and planning, bank and copying, reporting and methodology of upcoming visits. Calls to Babacar and Alhusseynou, Abdou. 3 Feb, Sun Morning (Check out and arrangement of cars) 14:00-15:30 Team lunch near crossroads: admin and planning. Drivers Alioun & Performance Evaluation of TLMP – Senegal 39 Mamadou Departure of two teams (one North, one SE) “ Evening AS&CB: Arrival in Saint-Louis, check in Hotel Sindone, calls by Alhousseynou to Regional Inspector and other officials to plan Mon. visits. Mamadou Evening EA&BD: Check into the Hotel de Paris in Kaolack. Calls by Babacar to Regional Inspector and other officials to plan Mon. visits. Alioun 4 Feb, Mon Eric ALLEMANO and Babacar DIOUF in Kaffrine 8:00 – 9:00 10:00 – 12:30 Interview at the Inspection d’Académie of Kaffrine. Discussion of issues related to insufficient quantities of TLMs delivered to the IA. Comments from principals suggest that:  the manuels d’appoint are not linked to the curriculum;  the TLMs for the primary grades are “too difficult” and should have been validated with children before being printed. The Academy was not involved in training on the TLMs. Visit to the Lycée de Boulel, a rural secondary school. TLMs available for science, history& geography, physics & chemistry for some grades but not others. Number of TLMs insufficient for some classes. Problem with combining two grades in one TLM: parts of the curricula for each grade left out. 7&8th grade math exercises “too easy” EA observes a physics & chemistry class. Mr. Samba BAKHOUM, IEVS Mr. Abdoulaye FAYE, BEMSG Mme Rokhaya SARR, Pedagogical counselor for English Mr. Boubacar TIMERA accountant & supplies manager Mme Neifatou N’DIAYE, Science teacher; Mr. Malal DIOP, History & Geography teacher Mme Bernadette MENDY, History& Geography teacher Mr. Abdou CISSOKO, Physics & Chemistry teacher. Lunch 14:00 – 15:30 Collège d’Enseignement Moyen (Middle School) in Kaffirine city. EA observes a science lesson. Mr. Samba THIAM, Math, Physics & Chemistry teacher Mr. Abdoulaye KA, Science teacher 16:00 – 17:00 17:30 – 18:00 Inspection Départementale de l’Education Nationale (IDEN) Debriefing at the Inspection d’Académie Mr. Ousmane BA, Inspector Mr. Fodé SARR, Accountant & supplies manager 5 Feb, Tues EA and BD in Kaolack Performance Evaluation of TLMP – Senegal 40 8:00 – 9:30 10:30 – 12:30 Lycée de Sibassor (rural secondary school) Focus group discussion with teachers about the TLMs. Primary school in Ngothie (rural). Focus group discussion with teachers and other stakeholders about the TLMs.  Mr. Samba NOBUR, Principal and 11 teachers.  N’DIAYE, President of the School Management Committee  Mme Néné DIOUF, committee member  Mme Fatou FAYE, President of the Women’s Group  Alioun, Representative of the Sports Committee  Mr. Mame Kor THIARE, Representative of the Village Chief. Lunch 3:00 – 4:00 Meeting with the head of the Inspection Départementale de l’Education Nationale (rural areas) Issues of TLM use in schools. 6 Feb, Wed EA and BD in Fatick 8:00 – 9:00 9:30 – 10:00 11:00 – 12:00 Briefing with the Departmental Inspector N’diongolor Middle School (rural)  Focus group discussion with teachers.  Visit to library Collectif des Chefs d’Etablissements de Fatick (Fatick Principals Association)  Focus group discussion  Training needs Mr. Saïdou BA, Inspector Mr. Ardo N’DIAYE, Principal Mr. Donga Carrera, Science teacher Mr. Papa Mamour DIOP, Physics teacher Mme Ndeye Magatte GUEYE, Science teacher Mme Aïda Ndoye N’DIAYE Science teacher Mr. Cheikh DIONE, Deputy Inspector Fatick IA Mr. Saïdou BA, Departmental Inspector, Fatick Mr. Assane FAYE, Accountant & supplies manager, Fattick IA Mr. Mame Kor FAYE, Librarian Mr. Saly SARR, IT technician Lunch 14:00 – 16:00 Attended, with IA staff, a JICA workshop in Fatik city on optimizing the use of textbooks and other learning aids. Invited to attend by the Deputy Inspector of the Fattick IA. 16:00 EA & BD drive back to the Hôtel de Paris in Kaolack and pack, check out and return to Dakar. Performance Evaluation of TLMP – Senegal 41 4 Feb, Mon Alhousseynou SY and Carol BENSON in Saint-Louis 8:00-8:15 Team meeting to finish planning the day 8:30-9:30 Meeting with l’inspecteur départemental (IDEN) de Saint-Louis (departmental director) to discuss project and locate schools and trainers; he accompanied us to the schools. Departmental Inspector (IDEN) Mr Samba Laobé DIOP 10:00-10:30 Arrival at (rural) Mpal Primary School; discussion with school principal, tour of library, checked presence of materials. School principal Insab BADJI Teachers Mme DIABO, Mme Bintu (bilingual) 10:40-11:15 Arrival at (rural) Rawane Ngom Middle School; found materials in his office. Asked 2 classes (4ème and 3ème) about SVT book. School principal Ngange GUÈYE Inspector Mr. Samba L. DIOP 11:45-12:35 Went to Académie, met Inspector General of IA in hallway (planned meeting later) and stayed to talk with Mr NDIAYE, who organized discussions with Specialist Inspectors. Inspector Gen/ Coord. of Middle and Secondary Ed. Amadou NDIAYE 12:35-12:45 Meeting with Mathematics Specialist Inspector. Math Inspector (IA) Alioune DIOP 12:55-13:05 Followed the Storage Chief (Comptable Matière) to new IA storage locale (10 minutes away) to see books in storage. Storage Chief Mr. TALL 13:10-13:25 Went to Centre de Formation (teacher training college) to meet another Specialist Inspector, this one for SVT (Science). Science Inspector (IA) Mr. Bactar FALL 13:35-14:05 Back to Académie, waited, then were able to meet with Inspector General. Inspector General (IA) Mr Ndar FALL 14:10-14:50 Lunch 15:00-15:50 (Waited at teacher training college and talked to people participating in Malaria workshop) 15:50-16:15 Meeting with SVT (Science) Inspector Science Inspector (IA) Mr Déthie Chiendella FALL 16:30 (Back to hotel to write notes and organize meetings for Tuesday in Louga). Meeting to synthesize findings of day 5 Feb, Tues AS and CB in Louga 7:00 (Check out of Hotel Sindone) Depart Saint-Louis for Louga. 8:10-8:45 Meeting with Inspector General of Louga IA, interviewed and he helped arrange visit to nearby middle school Mr. Lamine SARR, Inspector Genl. of the Louga IA 8:55-9:05 Meeting with principal of Middle School, interview Principal Middle School Mr. Abdullai DIALLO Mr. Amady KEBE from IA 9:05-9:35 Past and present librarians showed us books and explained how loan system works Mr Babacar N’DIAYE Mr Alias Ismaïl SYLLA 9:40-10:00 Interview in teachers’ room with two maths/science teachers Mr Mamadou THIAM Mr Abdulwahab SESAY 10:20-10:55 Meeting with Inspector General of IDEN Louga (whom we met in Saint-Louis yesterday). Inspector Genl. Of the IDEN Mr Hamedy KEBE 11:00-11:20 Meeting with school principal at Khady Mdiaye School director Performance Evaluation of TLMP – Senegal 42 Gning primary school; visit to class to see that they are using project TLMs (no lesson but they showed us their books—see photos). Ms Kourseline BARA Teacher Mr. Moussa DIAW 11:20-11:30 Went to the Centre de Formation looking for the CPIs (Conseilleurs pédagogiques itinérants) but they were not available. 11:30-11:50 Walked to the practice preschool beside the Centre de Formation to meet a parent (father of a middle school student) who is school principal. Preschool Principal Mr. SY 11:50-12:15 Back to the IA to witness people from the IDEN in Kebemer picking up TLM materials and others (see photos). Mr Alioune THIAM 12:15 Left for Thiès, lunch at Kebemer on the way. Evening Discussed day, Alhousseynou made calls to facilitate the next day’s visits. 6 Feb, Wed AS and CB in Thiès 8:00-8:30 Meeting with Inspector Gen at IDEN. Inspector General IDEN Thiès city Mr Talla FAYE 8:30-8:35 Viewed storage area and discussed distribution. Storage supervisor Mr Khatry DIAGNE 8:35-8:50 Discussed outside Mr. Sahid NDOY, chef Math Mr. Mamadou DJAW, collège Math/SVT 9:00-9:30 Meeting with Inspector Genl. Adjoint de l’IA and (during this meeting) quick interview of President of Parent-Teacher Association. Insp. GenL. Adjoint de l’IA Mr. Ibrahima BARR President of parent assoc. Mr. Moussa KANTE 9:30-9:40 Viewed storage area (locked, see photos taken from outside). 9:45-10:00 Meeting with principal of college. Principal of Mamadou Diaw middle school Mr Amarie MBAYE 10:00-10:20 Visited library and spoke with librarian and assistant. We recommended distributing the books that were just sitting on shelves and in boxes. School librarian Ms Pall Diop N’DIAYE Assistant Ms Dide Mbodji COULIBALY 10:30-10:45 Went to Training Centre, met with trainer. Mr Papa Birame THIANDOUNE (Return to Dakar, EA&CB check into Hotel Ibis) 7 Feb, Thurs Morning EA and CB prepare presentation for USAID 15:00-17:15 Debrief at USAID Mr Pape SOW Ms Michelle CHEN 8 Feb, Fri Morning Team meeting, sharing of findings in the North and the South East field trips. Eric ALLEMANO Carol BENSON Alhousseynou SY Babacar DIOUF Performance Evaluation of TLMP – Senegal 43 Afternoon Meeting with Abdou Maty Sene, Former Chief of Party TLMP Eric ALLEMANO Carol BENSON Alhousseynou SY Babacar DIOUF 9 Feb, Sat Morning Work on the preliminary draft of the IBTCI mission report Eric ALLEMANO Carol BENSON Evening Dinner for the team at Babacar Diouf’s home 10 Feb, Sun Evening Eric Allemano departs for Paris Performance Evaluation of TLMP – Senegal 44 ANNEX E. SUMMARY OF FINDINGS Summary of Findings about the TLMs Topic Findings Comments 1. Responsibility for writing the TLMs  2004-2006: 80% of the work done at ECSU with American authors  2007-2008: 50% of the work done at ECSU  2009-2011: 80% of the work done in Senegal  2012: 100% of the work done in Senegal by local authors. According to ECSU’s TLMP Program Manager. The lists of authors in the TLMs produced over the life of the project support this information. They are all Senegalese for the last books produced. 2. Variety of titles  28 titles produced for Grades 1 to 11. Subjects include French, math, science & environmental studies, history & geography, physics & chemistry, health and guidance.  Three of the 28 titles (elementary school readers) are reprints of existing, non-TLMP books as were three workbooks.  Only two teacher’s guides were produced, for environmental studies in Grades 1-2 and Grade 3. The post-primary books are “supplementary” books. The lack of training and teacher’s guides may explain why many teachers interviewed found the TLMs “too difficult” (esp. 7th-8th grade math, which some teachers in the Southeast refuse to use) or not “aligned” with the curriculum. The teachers were apparently not familiar with the concept of supplementary curriculum materials. 3. Validation of the TLMs  Little is documented. An expert committee was supposed to evaluate the drafts and they were piloted for 15 days in two regions.  Student learning was apparently not tested. The TLMs were developed in haste to meet deadlines. It is claimed that the TLMs are all aligned with the official syllabi, although this was disputed by some of the teachers met during the field work. 4. Copyright  The copyright holders of the early books Performance Evaluation of TLMP – Senegal 45 issues are the Project Director at ECSU and USAID Senegal. Later books add the MOE.  TLMs that are translated and edited versions of American textbooks do not mention the original authors or copyright holders.  Only a few TLMs document the sources of illustrations, graphs and charts, many of which were taken from the Internet or other sources. 5. Cultural sensitivity and special themes  Many photographs and illustrations do not reflect Senegalese or African people, objects and environments.  Males are represented far more frequently than females.  Mention of HIV/AIDS is restricted to the TLMs on health. The TLMs were supposed to be sensitive to culture, male-female equality, youth leadership, HIV/AIDS, etc. 6. Printing, distribution and storage of the TLMs  Overall, the printing process went well despite the fact that the printers had to do “rush” orders and deliver the books quickly to the IAs.  The MOE seems to have been inadequately involved in the distribution process in terms of determining the number of titles to distribute to different IAs. In addition, the IAs and IDENs were not clearly informed about the procedures to follow to get the books to the schools and the students.  Storage has been a problem although some schools keep the TLMs in their libraries or in specially-built rooms. Others are kept in the principal’s office or even home.  In a few schools, the TLMs are exposed to damage from dust, wind, rain and termites.  According to the printers, they had to bid separately for the printing of each TLM title, which was a cumbersome process.  Some schools did not receive enough TLMs to reach a ratio of one booke per student.  The team saw evidence of theft of the TLMs in at least one IA storage room.  Some schools charge parents fees for letting students take books home. Performance Evaluation of TLMP – Senegal 46 7. Teacher training Despite claims by the TLMP Project Manager and statements in newsletters that teachers had been trained, no evidence of this was found during the field visits. Training was apparently not budgeted. 8. Perceptions and uses of the TLMs  Whether or not the MOE informed the principals of the official status of the TLMs, many teachers expressed doubts about the issue.  The absence of teacher’s guides for all but two subjects made it challenging for poorly-trained teachers to use the TLMs.  Principals find that the TLMs help teachers do their lesson planning better than before. Some say that they find that the TLMs have boosted student learning.  Before the arrival of the TLMs, teachers in at least one region (Fatick) relied on the Internet for finding materials for their lessons.  When there are not enough TLMs for individuals, students often use available copies for group work. The TLMs were more favorably viewed by:  Educators in the South-East than in the North.  Academy, Departmental and subject-matter inspectors than the teachers.  Better-trained teachers than poorly-trained teachers. Parents had little to say but were generally pleased that the schools and the students had more books than before. A positive aspect of group use of a TLM is that it can promote collaborative learning. ANNEX F. EVALUATION OF SELECTED TLMS Title (year) Type Level Copyright holder(s) Editors Representation of men/boys Representation of women/girls Representation of local culture Integration of transversals (In)appropriate content SVT (2010) Suppl. 4 ème -3 ème (9th -10th) AM Sene, USAID & MINED Ed-in-Chief AM Sene Asst Ed C Seck 5 Men (pp 5, 58, 60, 79, 158) 3 Boys (15, 16, 29) 2 Women (photo 5, pregnant Euro 71) 2 Girls (cartoon 51, holding baby 78) Very little (kwashiorkor and goiter 5, cartoon leg 59, runners 58) No HIV or environmental issues Non-Africans (5,6,16,35,48,51,53, 59,63,65, 66,71- 73,77,80,97,99,159) This manual d’appoint is not intended to replace the textbook but “respects the objectives and competencies” of the curriculum (vii). Authors: 3 Senegalese (1 woman), 1 ECSU (J Houston). More males than females represented; of 4 females total in book, 1 woman pregnant and 1 girl holding baby (traditional roles). Overabundance of non-African cartoons and photos, e.g. drawings of naked white child/youth/adult of each sex (65) deemed highly inappropriate by informants. Highly unnecessary use of Western/non-African elements like white hands, eyes and ears, European family photos for genetics and Perrier water for experiment. Performance Evaluation of TLMP – Senegal 47 Math (2008) Suppl. 4 ème -3 ème (9th -10th) J Houston & USAID Ed-in-Chief J Houston Asst Ed A Sene 0 0 0 0 0 This manual d’appoint is “rich with the American approach” of situations/problems and is meant to complement study (viii). Authors: 2 ECSU (Houston with Lawrence), 2 Senegalese “author/adapters.” Intro by Laura Bush. Contains only French words and formulas. SVT (2010) Suppl. 5 ème (8 th) AM Sene, USAID & MINED Ed-in-Chief AM Sene, Asst Ed C Seck 1 Men (group 9) 3 Boys (12,21, cartoon 25) 1 Women (white 163) 1 Girl (cartoon 54) Dakar (Hwy constructn 3, 10,39) Urban foods (45, 47) Deals with “Environmental problems” Non-Africans (12,13,25,50,54,163) Non-African animals (77,81,94) “Support” book corresponding to “all of the points in the official curriculum of May 2008” (iii). Contains exercises to promote “mastery” or “methods.” Authors: 3 Senegalese (men). Few people represented, majority are boys. Some Senegalese photos illustrations, mostly urban. Phys Chem (2010) Suppl. 2 ème Sec (11th) AM Sene, USAID & MINED Ed-in-Chief AM Sene, Asst Ed J Houston 16+ Men (13,15,27, 28,54,105,107,108, 116, 137,145, 154,157, 176, 201, 213) 0 Boys 1 Women (cartoon family 15) 0 Girls Dakar (crowds 98,99) 0 Non-Africans (13,15,27,28,54,105, 107,108, 116,137,145, 154,157,176,201, 213, white hands 261-262) French cheeses (312) This “study instrument” conforms to the curriculum of 2008-2009. Each chapter has content, summaries and exercises (explained pp xiii-ix). Authors: 5 Senegalese (men). Almost exclusively men represented, almost all non-African. No transversals, little/no Senegalese culture. Phys Chem (2nd ed 2010) Suppl. 4 ème -3 ème (9th -10th) AM Sene, USAID & MINED Ed-in-Chief AM Sene, Asst Ed J Houston 9 Men 1 Boys 1 Women (grp 13) 1 Girls (cartoon 36) 0 0 Non-Africans (13,36,84,91,157,158, 163- 165,169, cartoon men 92,96,97) This book takes a “practical and simple American approach” combined with a “rigorous and precise Senegalese approach” (viii). Authors: 3 Senegalese (men), 1 ECSU (Houston). Intro by Laura Bush. Mostly non-African men in illustrations. SVT (2010- 2011) Suppl. 6 ème (7th) AM Sene, USAID & MINED Ed-in-Chief AM Sene, Asst Ed J Houston 3 Men (43,44,46) 0 Boys 0 Senegal (3,9,44,65, 66,74) ? Non-Africans (134, 149) This “support” book “respects the objectives and competencies” of the curriculum of Oct 2008 (vii). Authors: 5 Senegalese (2 women, 3 men). No females represented. Many photos of Senegal. Hist & Geog (2011) Work￾book Prim gr 3 USAID & MINED Ed-in-Chief AM Sene, Asst Ed C Seck (All men) 0 0 0 Old, low-quality photos, cartoons (9,29,49, 58) Performance Evaluation of TLMP – Senegal 48 This “livret” (workbook) is not meant to be written in, but it provides exercises for students. Old, often distorted illustrations/photos/caricatures. Collaboration with USAID basic ed (EDB) project. No authors listed, only “editing team” of 6 Senegalese (men). Envir Sci Tech (2006) Suppl. 3 ème (10th) J Houston & USAID Ed-in-Chief J Houston, Asst Ed M Coulson￾Clark Men (non-Af 28,32, 35,41,53,77, 80,81,83- 85,88,90, 94,100- 101,112…) Boys (59) Women (non-Af 41,47,53,77,112, Muslim woman 116) Senegalese farmers (77) 0 Non-Senegalese (28, 41, white hands 42, 47, urban 53,59,100- 101,112,117) This manual d’appoint depends on U.S. photos. (Exceptions are one Muslim woman and some Senegalese male and female farmers.) Contributing authors appear to be 5 Americans and 2 Senegalese (including A Sene). Performance Evaluation of TLMP – Senegal 49 ANNEX G. LIST OF CLARIFICATIONS ON THE TLMS FOR POSSIBLE MOE DISTRIBUTION The following is a list of clarifications for possible inclusion in an MOE letter to IA directors (copied to department directors) regarding TLMP materials. If possible, it could be attached to the proposed curriculum linking chart or brochure. 1. All regions have received TLMP materials (list them by level). These books financed by USAID are to be distributed free of charge to the schools. They should be considered school property but loaned to students and teachers for the appropriate level. It is up to the school to determine if this distribution is to be short-term (example: day of the lesson, 1-2 weeks from the library, or for the entire school year) but by the end of the school year the books should be returned to the school to be used the next year. School directors are instructed to do everything in their power to ensure return and re-use of books, however, it is understood that there will be occasional small losses due to wear and tear, misplacement, etc. 2. If you have not received these materials, please check with your IA or IDEN. If your school does not have the resources to pick up the materials at your IA or IDEN, we direct you to contact the Regional Governors and/or your city mayor to request transportation help (as for more remote schools Governors can use vehicles from the gendarmeries). 3. These materials were destined for enrichment of our national curricula even if they do not cover all points. Please review them within your subject department to see what is included and how content and activities could best be used. 4. They were also meant to be distributed to all teachers and pupils. They should not be kept in storage unless teachers and students have access to this storage and can sign out the books for their own use. 5. If you currently have cartons of TLMP materials in storage, please distribute them as soon as possible. We understand that the quantities may not allow for 1 student : 1 book ratio, but we recommend that school leaders (directors, principals) ensure that students have contact with these materials through one or more of these methods:  Students borrowing from school libraries or storage rooms, or from their teachers.  Students using them for in-class study or independent study.  Shared after-school activities (where 4-5 students living in the same neighborhood share one book, with 1 student responsible).  Teachers borrowing from libraries or storage rooms to use the TLMs in classrooms.  Teachers borrowing from libraries or storage rooms and signing them out to students.  Teaching students how to cover and care for books and return them at the end of each week/month/term/school year. Performance Evaluation of TLMP – Senegal 50 ANNEX H. CONFLICT OF INTEREST STATEMENTS Performance Evaluation of TLMP – Senegal 51 Performance Evaluation of TLMP – Senegal 52 Name Babacar DIOUF Title Consultant national Organization IBTCI Evaluation Position Team Leader Team member X Evaluation Award Number (contract or otherinstrument) AFR-12 -00000 I USAID Project(s) Evaluated (Include project name(s). implementer name(s) and award number(s) if applicable ) Textbooks & Learning Materials Project (Chicago State University , Alabama A&M , South Carolina State Univ., Elizabeth City State University, University of Texas San Antonio) I have real or potential conflicts of interest to disclose. Yes No XX If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include .but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated.including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the Implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 26 Fevrier 2 013 Performance Evaluation of TLMP – Senegal 53