November 2013 This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Thomas Tilson (Team Leader), Eric Allemano, Kofi Mereku, and Kofi Marfo under Task Order AFR-12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION REPORT Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania TLMP in Ghana Cover Photo Credit: Chicago State University EVALUATION OF THE USAID-FUNDED TEXTBOOKS AND LEARNING MATERIALS PROGRAM (TLMP) IN ETHIOPIA, GHANA, MALAWI, SENEGAL, SOUTH AFRICA AND TANZANIA TLMP IN GHANA Prepared by: Thomas Tilson (Team Leader) Eric Allemano Kofi Mereku Kofi Marfo November 25, 2013 The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of TLMP – TLMP in Ghana i TABLE OF CONTENTS Map of Southern Ghana............................................................................................................... iii ACRONYMS................................................................................................................................. iv EXECUTIVE SUMMARY.............................................................................................................. v A Snapshot of TLMP in Ghana .................................................................................................................................. v Implementation of TLMP ........................................................................................................................................... vi Evaluation Methodology............................................................................................................................................ vii Key Findings: Impact of TLMP on CSU ................................................................................................................. vii Key Findings: TLMP’s Legacy in Ghana.................................................................................................................. vii Key Recommendations............................................................................................................................................ viii 1. EVALUATION PURPOSE, QUESTION AND OBJECTIVES .............................................. 1 1.1 The Background to TLMP .................................................................................................................................. 1 1.2 Evaluation Objectives.......................................................................................................................................... 1 2. TLMP PROGRAM BACKGROUND....................................................................................... 2 2.1 CSU Responsibilities............................................................................................................................................. 2 2.2 MOE Context and the Educational System..................................................................................................... 3 3. EVALUATION METHODS & LIMITATIONS ....................................................................... 4 3.1 Research Conducted at CSU............................................................................................................................. 5 3.2 Research Activities in Ghana ............................................................................................................................. 5 3.3 Limitations of the Study ...................................................................................................................................... 5 4. FINDINGS AND CONCLUSIONS ........................................................................................ 6 4.1 CSU Achievements and Challenges................................................................................................................. 6 4.1.1 Achievements................................................................................................................................................. 6 4.1.2 Challenges........................................................................................................................................................ 7 4.2 TLMP Output.......................................................................................................................................................... 8 4.3 Program Management and Partnerships......................................................................................................... 9 4.3.1 Program Management at CSU and in Ghana........................................................................................... 9 4.3.2 Partnerships with MOE.............................................................................................................................. 10 4.3.3 Partnerships and Linkages with Colleges of Education and Universities........................................ 12 4.3.3 PAC and PTA................................................................................................................................................ 13 4.4 Program Implementation.................................................................................................................................. 13 4.4.1 Materials Development.............................................................................................................................. 13 4.4.2 Materials Assessment by Evaluation Team........................................................................................... 14 4.4.5 Teacher Training......................................................................................................................................... 15 4.4.6 The Current and Anticipated Future Status of CSU and TLMP ..................................................... 16 4.5 Outputs and Outcomes.................................................................................................................................... 17 4.5.1 School/Teacher/Classroom Observations............................................................................................. 17 4.5.2 Learner Reading Assessments and Outcomes...................................................................................... 18 4.5.3 Other Evidence of Learning..................................................................................................................... 21 4.6 Educator Comments on TLMs........................................................................................................................ 21 4.7 Stakeholder Interest In and Use of Materials/ Sustainability.................................................................... 21 4.7 Conclusions.......................................................................................................................................................... 22 5. LESSONS LEARNED............................................................................................................. 24 6. RECOMMENDATIONS......................................................................................................... 24 6.1 Materials Improvement..................................................................................................................................... 25 6.2 Systems Improvement....................................................................................................................................... 25 Evaluation of TLMP – TLMP in Ghana ii 6.3 Linking TLMP to USAID Priorities in Early Grade Reading....................................................................... 25 ANNEX A. SCOPE OF WORK ................................................................................................ 27 ANNEX B. QUESTIONS POSED IN VARIOUS INTERVIEWS CONDUCTED AT CSU . 30 ANNEX C. DATA COLLECTION INSTRUMENTS, GHANA STAKEHOLDERS.............. 34 ANNEX D. SCHOOL OF INTERVIEWS AND MEETINGS ................................................. 39 ANNEX E. READING ASSESSMENT FORM AND EVALUATION OF LEARNER READING COMPETENCY........................................................................................................ 43 ANNEX F. PRINTING AND DISTRIBUTION – A CASE STUDY....................................... 45 ANNEX G. REVIEW OF TLMP PUPIL’S WORKBOOKS AND TEACHER’S GUIDES..... 47 ANNEX H – THE ORIGINAL 14 TLMP DISTRICTS.............................................................. 67 ANNEX I. WORKBOOKS AND INSTRUCTIONAL MATERIALS PREPARED BY TLMP 2009-2012 ..................................................................................................................................... 68 ANNEX J. CONFLICT OF INTEREST STATMENTS ........................................................... 69 Evaluation of TLMP – TLMP in Ghana iii MAP OF SOUTHERN GHANA With locations visited marked Evaluation of TLMP – TLMP in Ghana iv ACRONYMS AEI President’s African Education Initiative CA Cooperative Agreement CRDD Curriculum Research and Development Division CSU Chicago State University EGRA Early Grade Reading Assessment GER Gross Enrollment Rate GES Ghana Education Service IBTCI International Business & Technical Consultants, Inc. KG Kindergarten LT Lead Teacher MOE Ministry of Education MSI Minority Serving Institution MTPDS Malawi Teacher Professional Development Support program NALAP National Acceleration Literacy Program NER Net Enrollment Rate P1-P3 Primary 1 - Primary 3 PAC Program Advisory Committee PTL Program Team Leader PMP Performance Monitoring Plan PTA Program Technical Assistance RTI Research Triangle Institute TL Team Leader TLM Textbooks and Learning Materials TLMP Textbook and Learning Materials Program TTC Teacher Training College UEW University of Education, Winneba UNICEF United Nations International Children’s Fund USAID United States Agency for International Development USAID/G United States Agency for International Development/Ghana USAID/W United States Agency for International Development/Washington Evaluation of TLMP – TLMP in Ghana v EXECUTIVE SUMMARY As part of the President’s African Education Initiative (AEI), Chicago State University (CSU) was awarded a Cooperative Agreement (CA) in 2005 to meet the needs of Ghana’s Ministry of Education (MOE) in creating curriculum, generating textbooks and teacher’s guides, and teacher training to support its recent decision to include kindergarten (KG) in its curriculum. CSU focused its attention on producing KG materials in English in literacy, numeracy, and environmental science. In addition, CSU was tasked with developing literacy, mathematics, and science materials for primary grades 1-3 (P1-P3). In 2008 CSU received another CA to allow it to continue the work it had started for KG and P1-P3, but owing to budget cuts, the materials developed for P1-P3 were not printed or distributed. The final level of funding of US $8 million in 2012 was to enable CSU to expand implementation of TLMP in Ghana; the total funding was comparable to the funds received by the other MSIs. CSU has continued work on materials development with support from USAID/Ghana. Of the six countries in which TLMP operated, Ghana and Ethiopia were the only ones with a focus on primary level throughout the total life of the activity. Further, in Ghana there were major changes in policy and practice with respect to languages of instruction and to incorporation of kindergarten (KG) into basic education during the overall life of TLMP plus the addition of the major NALAP (National Accelerated Literacy Program) associated with the restructuring, so this country report on the expansion phase of TLMP provides more information on the broader contexts surrounding TLMP than do the other country reports. The program evaluation of the expansion phase of TLMP had the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions (ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. Beginning in October 2012, the Chicago-based project team leader, Dr. Nancy Horn, had a number of in-person and telephone/e-mail interactions with CSU TLMP staff. The in-country component of the evaluation was originally scheduled for late November 2012, but at the request of the Mission (e.g., to accommodate national elections), it took place in June 2013. A Snapshot of TLMP in Ghana • TLMP was highly successful in producing teacher’s guides and workbooks for KG1 and KG2 in literacy, numeracy and environmental science and distributing 6.1 million of these materials throughout Ghana. In addition, it produced 220,000 wall posters that were distributed as well. The success of printing and distributing the materials was due to a capable local printer, Buck Press. TLMP also produced and field tested P1-P3 teacher’s guides and workbooks in English, mathematics and natural science, but for lack of funds these were not printed. • TLMP developed and implemented an in-service teacher training program for 2,000 teachers, but did not become engaged with pre-service teacher education. • Major factors affecting implementation were the decision of the MOE to change the language of instruction for lower primary from English to local languages and its decision to have expendable workbooks, which could not be reused and which costs kept from being reprinted, as a core part of the TLMP materials. Evaluation of TLMP – TLMP in Ghana vi Implementation of TLMP TLMP in Ghana began with a needs assessment in 2005, which was about the time the MOE decided to incorporate KG into the education structure, a move that took place in 2007. As a result, the MOE requested that TLMP focus on developing materials for KG, as Ghana didn’t have any materials for KG. Also, there was recognition that the teaching and learning of English remained an ongoing challenge. As a result of perspectives at that time, the MOE requested that TLMP develop materials for KG1 to P3 in English literacy, numeracy and science and that the materials should be in English. TLMP in Phase I (2005-2008) reprinted existing materials and, then in Phase II (2009-2012), developed new materials for KG1 to P3 in literacy, numeracy and science, and then printed and distributed 5.6 million copies of the KG1 and KG2 materials only – first to an initial 14 deprived districts, but then to all 170 districts throughout the country. The materials were available to all public schools and registered private schools; the latter were particularly appreciative because they give special attention to teaching English. The materials were highly valued by teachers. They were especially appreciative that the teacher’s guides were closely aligned with the official KG curriculum. In addition, teachers felt that the workbooks were especially valuable in helping children learn basic pre-reading skills. But the decision to use disposable workbooks was ill-advised, as in the majority of schools they were consumed in one year and, thus, the teachers were unable to continue using TLMP as designed. Only in about one-third of the classes visited by the evaluation team did the teachers make special efforts to preserve the books for re￾use, e.g., by having the children write answers to questions in the workbooks in a separate notebook. However, the major challenge facing the relevance and sustainability of TLMP was an important shift in focus of the Ministry. Shortly after the start of TLMP, the Ministry, with USAID support, decided to emphasize teaching in mother tongue (L1) in the early grades as the best way of engaging children in school, and for the transition to English by P4. The result of this initiative was NALAP (National Accelerated Literacy Program), a program for KG1 to P4 in which mother tongue dominated instruction in the early grades and there was only a limited role for English; there were only 10 minutes of ESL out of 90 minutes total for language arts in KG1, gradually moving to a 50-50 split between local language and English (L2) by P3 and P4. With this shift in policy, there was no longer a need for an intensive English program such as TLMP. As a result, one of the last major activities of TLMP was to merge the best parts of its program into the NALAP teacher’s guides in order to strengthen the limited, although important, English component. This merger or fusion process is nearly complete and TLMP, which is now primarily a mission-driven project, will now print the revised NALAP guides and support a major teacher training program for over 100,000 teachers starting in late 2013. Currently, USAID/Ghana has taken TLMP under its own wing, with a scheduled end of early 2014 (The Mission-based activities fall outside the Scope of Work for this evaluation, and are discussed to provide broader context). The Ministry would like to draw on the best of TLMP to strengthen the limited English component of the new bilingual early childhood education program. This has resulted in the current merger or fusion process, which is taking the best and most relevant activities and content of TLMP and strengthening the English component of the NALAP teacher’s guides. This will be an important contribution, but the Ministry is no longer interested in the reprinting or separate identification of TLMP. The new merged materials are to be referred to as the Child Literacy Program. TLMP will also print the new merged teacher’s guides and support the in-service training of over 100,000 teachers on the use of these new guides. CSU has worked most closely with the MOE’s Curriculum Research and Development Division (CRDD) to create the materials and implement the program. This approach helped to create a commitment and ownership of TLMP by GES, but it also limited technical assistance that might have been useful in areas Evaluation of TLMP – TLMP in Ghana vii such as early childhood education and M&E. In addition, the project engaged some faculty of the University of Cape Coast and the University of Education, Winneba and some of the colleges of education in creating the instructional materials and implementing the in-service teacher training program, as well as some members of the PAC and PTA. However, TLMP failed to engage with these universities and colleges in terms of trying to influence pre-service teacher education. In addition, TLMP failed to engage the Teacher Education Division (TED) in teacher training activities. Evaluation Methodology The evaluation used a mixed-methods methodology, primarily qualitative, comprised of key informant interviews, review of materials, and site visits with classroom observations and discussions with educators, both in the settings of their schools and offices and in the context of a writing workshop at the Ghanaian National Association of Teachers (GNAT). To these activities was added assessment of the literacy skills of some students in the Volta Region. The major limitation, apart from time, was that relatively few schools are actually using the TLMP materials as planned. The evaluation team visited 16 schools, 11 public and five private, in four regions, where four different major local languages are spoken. The evaluation team had intended to observe a number of TLMP classes, but in the end could not do so because so few schools – only five of the 16 schools visited, all but one of them in Volta Region, the last region where school visits took place – were currently using TLMP. The team also met with officials in seven district offices, as well as individuals at the University of Cape Coast and University of Education, Winneba and officials in three colleges of education. Key Findings: Impact of TLMP on CSU An important objective of the TLMP projects for all the MSIs was to develop interest and skills in supporting education projects in developing countries. After an uneven start, this objective was clearly met with CSU, which has learned how to manage a large international project funded by USAID. Ten CSU College of Education faculty members and doctoral candidates worked with the MOE and GES is designing, writing, and producing the instructional materials and in assessment activities. In addition, 20 CSU students served as interns in Ghana on the project, and there were two Ph.D. dissertations based on work in Ghana. CSU also established the Center for International Education on its campus, stemming from the work in Ghana. Key Findings: TLMP’s Legacy in Ghana TLMP succeeded in producing and distributing very large numbers of materials quickly, but otherwise the legacy of TLMP, per se, in Ghana is limited, largely in the form of posters and wall charts. As noted above, TLMP in the Ghanaian locales visited realistically is no longer in use as intended, due to combinations of change in Ministry approaches towards literacy, being subsumed by more recent projects, and the lack of TLMP workbooks. All but one of the private schools received the TLMP materials, but only 30% of the schools still had a supply of workbooks and were using them. The teachers in the other schools continued to use the teacher’s guides and, in some cases, the workbooks as reference materials. Many of the schools had started to use the NALAP materials. None of the teachers interviewed had received any TLMP training. At two (Cape Coast and Effutu) of the seven district offices, the education officials had not heard of TLMP; this highlighted a problem with communications between TLMP and the field. For example, the team was given to understand that TLMP did not orient the District Directors to the project when the materials were distributed nationally, thereby losing an opportunity for getting District Director support for implementing TLMP. Regarding the universities, TLMP did engage several lecturers and even had a MoU with the University Evaluation of TLMP – TLMP in Ghana viii of Cape Coast and the University of Education, Winneba, but this linkage did not result in incorporating any TLMP activities or materials in the universities’ academic programs. The lack of information and TLMP materials was also evident at the three colleges of education. The M&E activities of TLMP were weak. There was no M&E officer in the TLMP office and, thus, very limited ability to track implementation activities or to assess the impact of TLMP in the classroom and on student learning. CSU and the Ministry conducted just one learning assessment with KG2 tests in all three areas (literacy, numeracy and environmental science) in a limited number of experimental and control schools. The results showed a modest improvement in learning in the TLMP classes in literacy, but basically no difference in numeracy and science. The evaluation team administered a basic literacy test to a very small sample of KG children in three schools in Jasikan and Ho, in the Volta Region. (A wider sample wasn’t possible because the majority of schools were no longer using the TLMP program.) The children assessed had success in certain language skills (word recognition and naming pictures of familiar objects) and great difficulty in other areas (describing a moderately complex picture in the workbook). In doing word-picture assessment the researcher found that in some cases students seem to have memorized the word-picture sets in the order they appeared on the page. If the researcher chose the word-picture sets at random, some students had difficulty giving the right answer. Key Recommendations The recommendations are divided into improving materials and strengthening the education system. Materials Improvement • One fundamental recommendation is that TLMP’s work could have been more effective if TLMP could have been able to shift to first language instruction earlier workbooks. The workbooks and teacher’s guides could continue to serve a limited market including private schools that emphasize English language instruction and cosmopolitan public schools where it is not possible to teach in a common mother tongue. To help meet this need, the workbooks should be made available in the commercial market so that schools or parents could purchase them. Although it was not advisable to create disposable workbooks because of the unaffordable recurrent costs, theoretically, the KG1workbook might now be used over two years (KG1 and KG2) because of the more limited English program implemented in the context of NALAP focus on L1. However, there are no plans to reprint the workbooks. Because workbooks are highly valued by the Ministry and teachers, other forms of workbooks should be explored such as plasticized pages that children could write on, but then be erased so the books could be used the next year. As an alternative to workbooks (note that the NALAP has no workbooks), it would be desirable to create “activity books.” These would be non-disposable books that would have many of the advantages of workbooks, but where children would write in their exercise books rather than in the books themselves. • The approach to reading would be much improved by having a greater emphasis on phonics. In fact, the teacher’s guides and, especially, the TLMP workbooks are better suited to teaching in mother tongue rather than English because they assume the children have a greater level of language ability than is the case in English. In this sense, the TLMP materials, if translated into L1, might be better suited to NALAP than even the TLMP program. At least based on the very limited sample of students assessed, there is little actual learning to read taking place. Evaluation of TLMP – TLMP in Ghana ix Systems Improvement • Although TLMP in Ghana was more successful in this regard than various other TLMP implementations, a major design flaw was the failure to pay adequate attention to the need to train teachers in how to use textbooks and learning materials. • Although TLMP has done limited in-service teacher training and plans a massive teacher training program beginning near the end of 2013, the program gave no attention to the important pre￾service programs. The new NALAP program is working with pre-service institutions, and efforts should continue and be strengthened to ensure that the newly trained teachers are competent in bilingual education and teaching basic reading skills in L1 and L2. • The Early Childhood Education (ECE) Coordinators should be trained so that they better understand the new Child Literacy program and teaching L1, and to then be able to develop or support district-level or school-based training. • The book distribution process went well, thanks in large part to Buck Press, but the system can be strengthened to ensure that books are distributed everywhere where needed and in the correct numbers. Evaluation of TLMP – TLMP in Ghana 1 1. EVALUATION PURPOSE, QUESTION AND OBJECTIVES 1.1 The Background to TLMP The Textbook and Learning Materials Program (TLMP) was launched by the United States Agency for International Development in 2005 as part of the President’s African Education Initiative (AEI). It contributed directly to USAID’s effort to develop and distribute learning materials to improve sub￾Saharan African host country partners’ management capacity in the education sector. For the expansion phase of TLMP, which began in 2008/9, each of the five Minority-Serving Institutions (MSI) was responsible for managing and implementing the TLMP in a specific country and with achieving specific output results. Each MSI was also responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in primary schools in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MOE) and other local specialists. The main objectives of the TLMP were to: 1) Produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities in primary schools within SSA, 2) Strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) Ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). TLMP’s original focus was to develop TLMs for primary schools. In Ghana the Ministry of Education (MoE) determined that because it had very recently included kindergarten (KG) in its purview and there were no official texts or teacher’s guides, Chicago State University (CSU) should focus its attention on producing KG materials in literacy, numeracy, and environmental science. CSU was also tasked with the responsibility of developing literacy materials for primary grades 1-3 (P1-P3). TLMP was launched in Ghana with these responsibilities in 2005; in 2008 CSU’s CA was extended to allow the CSU project to expand on the work it had started for KG and P1-P3. However, owing to cuts in the TLMP budget, the materials developed for P1-P3 were not printed or distributed. The final level of funding for CSU to expand implementation of TLMP in Ghana was approximately US$8 million, comparable to the final levels of funding received by the other MSI. 1.2 Evaluation Objectives This performance evaluation covered work led by CSU during the expansion phase, 2008-2012, and was intended to satisfy the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions (ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. Evaluation of TLMP – TLMP in Ghana 2 2. TLMP PROGRAM BACKGROUND Of the six cooperative agreements for the extension of TLMP, TLMP in Ghana and in Ethiopia were the only ones to reflect a continuing focus on primary level literacy throughout both phases of the TLMP activity and, unlike TLMP in Ethiopia, it was affected by the very major change in education policy vis-à￾vis the primary language of instruction. Further, an additional set of major education interventions, the National Acceleration Literacy Program (NALAP), began in the course of the expansion phase. Therefore, to provide a broader context, this Ghana TLMP report provides more information as to predecessor and concurrent activities, which were not the subjects of this evaluation, than do the other country reports. 2.1 CSU Responsibilities For the period 2005-2008, TLMP was a component of the centrally-funded President’s African Education Initiative (AEI). CSU launched its project in 2006 with a rapid needs assessment, which was followed by a more comprehensive assessment. The CSU leadership team identified subject areas, grade levels, potential partners, teacher training institutes, and issues related to the educational system. After holding a number of workshops with the Curriculum Research and Development Division (CRDD) of the Ghana Education Service (GES), the CSU leadership team identified development of materials to meet the new needs of the MOE in their efforts to incorporate KG as the highest priority. CSU identified a Ghanaian representative and opened a small office from which to manage the activity. Cape Coast University, the Education University, Winneba (UEW), and CRDD partners worked in concert with CSU to identify a writing team and to implement the program. Unfortunately, in the AEI phase of the work, no early childhood specialist or bilingual experts were made a part of the team. Although university partners in Ghana included these specialists in their teams, CSU’s failure to provide support in this area made for difficulties and misunderstandings in the partnership. Moreover, all materials were developed in English initially and then were to be translated into Twi, one of 11 languages of instruction, but funding cuts prevented the translation from occurring. As many KG teachers do not have a good command of English, many had a difficult time understanding the teacher’s guides. Consequently, as part of its own education portfolio, USAID/Ghana has been supporting the translation of TLMP teacher’s guides into Twi. The difficulties and misunderstandings between the partners were exacerbated when the new materials were edited. CSU did not believe materials had been properly edited by Ghanaian partners and so preceded to edit them again. Eventually, issues were resolved and CRDD had the final say. It should be noted that some of the illustrations in the children’s workbooks were inappropriate, especially regarding gender stereotypes, but were retained, e.g., males depicted as doctors, policemen, footballers, and soldiers, while women are depicted as teachers and nurses – a stereotypical set of gender-based professions; depiction of community leaders as almost all men, etc. CSU had books that were exemplary in structure, content, and illustrations identified for use as models and CSU reprinted or edited and printed over 2.5 million colored teaching and learning materials, which included numeracy, literacy and environmental studies workbooks and teachers’ guides, as well as over 300,000 wall charts for use in the Kindergarten through Primary 3 classrooms. USAID/Ghana paid for the first print run, produced by All Good Books. Subsequently, Buck Press was identified as a printer who could take on the work, which ultimately included printing, storage, and transport for the distribution of books nationally. Prior to the MOE’s assuming responsibility for KG, the private (for￾profit, faith-based, other NGO, etc.) KG all had their separately developed curricula and materials. Because the MOE wanted for there to be standardized KG curricula and materials, TLMP distributed its materials to both public and private KGs. Starting with the AEI phase of TLMP, there were a number of miscues with respect to printing and distribution. CSU purchased printing equipment for GES, believing that GES was responsible for printing Evaluation of TLMP – TLMP in Ghana 3 educational materials. After the equipment arrived in Ghana, CSU learned that GES does not do its printing in-house. Storage and distribution was to have been in the hands of GES; however, Buck Press ended up taking care of this as an in-kind contribution to TLMP, and school districts picked up the TLMs themselves. In fact, there was never any formalized method for distribution, contributing to a lack of accountability. CSU envisioned providing training to school lead teachers on the use of the materials. GES was to have selected these teachers in each district (books were eventually delivered to all districts), but then did not complete the selection process. Hence, the training consisted of two week-long workshops for the teachers of the 14 districts that had been selected initially. CSU staff informed the team that it was not satisfied with the extent of the training it provided, but further training was restricted due to budget considerations. Among the many goals included in the CA that CSU sought to achieve in the 2008-12 period were to: • Sign a new MoU with the MOE, specifying roles of each partner • Conduct a new needs assessment • Continue to write and field test materials • Evaluate and monitor the effectiveness of achieving specified results – with focus on deprived Ghanaian primary schools that have not reached the gender parity index. • Continue to establish partnerships with the Curriculum Research & Development (CRDD) division of the Ghana Education Service (GES), University of Cape Coast, University of Education at Winneba, Centre for Education Development Evaluation and Management (CEDEM) and others guided by a Program Management Committee (PAC) • Branding deliverables • Disseminate results to other MSIs • Print instructional materials based on deliverables • Expand into 16 new districts, in addition to the existing 14 districts in ten regions • Train 5,000 teachers in use of materials and in interactive methods • Introduce teachers to cooperative learning • Support sustainability in the GES • Continue to form new partnerships CSU experienced difficulty in trying to meet these goals, as discussed in 4, Findings and Conclusions, below. 2.2 MOE Context and the Educational System The Ministry of Education (MOE) has overall responsibility for education policy in Ghana; within it the Ghana Education Service (GES), which was established in 1974, is responsible for the implementation of approved national policies and programs related to pre-tertiary education. GES has several divisions relevant to TLMP including basic education, teacher training, and CRDD. The MOE has been undergoing decentralization since the 1980s as one step to address several critical issues, including poor quality of teaching and learning, attrition and low achievement especially by girls; poor/no infrastructure; lack of well-trained and motivated teachers, especially in the rural and deprived urban areas; and poor quality teaching and learning materials. In conjunction with this, the MOE recognized the importance of early childhood education on the ability of children to succeed in school in subsequent years, and decided to incorporate a two-year KG program in 2004, rather than leaving it to social service agencies and private individuals, which historically had been the case. In 2007 the Ministry formalized an 11-year primary school system: 2 years KG, 6 years primary and 3 years junior high school. This is followed by a 4-year high school program. Evaluation of TLMP – TLMP in Ghana 4 Thus, when the TLMP needs assessment took place in 2005, the MOE was beginning to incorporate kindergartens into its structure, yet there were few qualified KG teachers, virtually no learning materials for kindergartens, and a continuing interest in trying to strengthen English language in schools. As a result, the MOE requested that TLMP focus on developing English language materials for kindergartens and the early primary grades. TLMP was committed to developing programs for KG1, KG2 and P1-P3 in language, numeracy, and science. The aim was to get good quality English language materials into the classroom as soon as possible. The inclusion of numeracy and science was also a response to MOE priorities. In 2007, the MOE added to lower primary (P1-P3) environmental studies and math that emphasized problem solving. The MOE sought to make teaching more practical and child￾centered using more interactive teaching methods that encourage enquiry and problem solving skills through the use of project work using local materials. The MOE also took note of research that suggests that teaching in the mother tongue in the first years of schooling has many benefits, including a smoother transition to a second language (English, in the case of Ghana). As a result, just as TLMP was starting, the context quickly changed with the development of the National Accelerated Literacy Program (NALAP). Although NALAP includes some English language instruction (L2), the emphasis is on mother tongue instruction (L1). In fact, out of a 90-minute class period for language and culture, only 10 minutes is allocated to English in K1, 20 minutes in K2, and slowly increasing to nearly 45 minutes (50%) of the time by P3 with a transition to English as the medium of instruction in P4. By 2010 there was considerable progress in the expansion of KG reaching 97% Gross Enrollment Rate (GER) and 59% Net Enrollment Rate (NER). Yet 70% of the KG teachers lacked formal qualifications (information from appraisal of Ghana’s Education Sector Plan 2010-2020, p. 19). Student progress in English proficiency by P3 still lagged with only 40% mean score on a national test, minimum skills at 57% and proficiency at only 20% (NEA, National Education Assessment). These results suggested the need for an even stronger English program in the early grades, to which TLMP was responding. Yet, the Ministry was committing itself to the importance of L1. In 2012 it published the document “Scaling up National Quality KG Education in Ghana.” It is worth noting that Ghana’s history of the use of L1 as the language of early grade instruction long predates Independence although with a number of detours along the way.1 Thus, the current policy is not new, even though the renewed emphasis on L1 is new. While the MOE is committed to L1 in the early grades, and is even considering delaying the transition to English as the medium of instruction by one year, English remains important and is the language of the leaving exam at the end of P6. English is perceived by the public as critical to being successful in Ghana and, thus, much of the public is not supportive of the new emphasis on L1. The current language policy is controversial and came up in virtually all of the interviews conducted by the evaluation team. Finally, one unanticipated variable within the MOE was that the Minister changed three times in two years, so the program had to go through all the approvals several times. Also, each Minister had a different outlook and perspective on TLMP, which required ongoing adjustments by CSU. 3. EVALUATION METHODS & LIMITATIONS The evaluation of Chicago State University’s implementation of TLMP in Ghana was primarily qualitative and made use of (a) Review of project documents, of the literature, and of materials prepared under the CA; (b) Interviews with CSU administrators and project staff in the U.S.; (c) Key informant interviews with USAID staff, implementers, educators at various levels, and other stakeholders in Ghana; (d) Classroom observations; and (e) Oral assessments of selected students. The overall Team Leader for 1 See, e.g., Charles Owu-Ewie, “The Language Policy of Education in Ghana: A Critical Look at the English-Only Language Policy of Education,” Selected Proceedings of the 35th Annual Conference on African Linguistics, 2006 Evaluation of TLMP – TLMP in Ghana 5 TLMP, Dr. Nancy Horn, conducted interviews at CSU in late 2012 and developed the research design. The field evaluation team consisted of two Americans (Dr. Thomas Tilson, Team Leader, and Dr. Eric Allemano) and two Ghanaian experts in education (Dr. Kofi Damian Mereku and Mr. Peter Kofi Marfo), who conducted their field work in Ghana between June 9 and June 20, 2013. The team conducted a number of interviews in Accra with a range of stakeholders: USAID/Ghana, MOE and GES officials, various stakeholders including Buck Press and UNICEF, and then visited district offices and schools in four regions (Greater Accra, Eastern, Central and Volta), where data were collected from directors, teachers and children. The research teams also interviewed representatives at the University of Cape Coast, the Education University at Winneba, Aburi College of Education, Jasikan College of Education, and the OLA (Our Lady of the Apostle) College of Education. (See APPENDIX C – SCHEDULE OF RESEARCH ACTIVITIES). 3.1 Research Conducted at CSU The overall evaluation research design called for research to be conducted at the MSIs to determine 1) the background to the program and how it was implemented, and 2) how participation in a USAID￾funded program built the capacity of MSIs to lead future projects. The interviews conducted at CSU were based on several sets of questions posed of leaders, program managers and participants (see APPENDIX B1 – QUESTIONS POSED IN VARIOUS INTERVIEWS AT CSU). 3.2 Research Activities in Ghana With the background information provided by CSU, the Program Team Leader (PTL) developed a background paper/research design that included different categories of questions to be posed to different stakeholders (see APPENDIX B2 – DATA COLLECTION INSTRUMENTS, GHANA STAKEHOLDERS). From the research design, the evaluation team developed a shortened version of key evaluation questions: 1. What has been the impact of TLMP in terms of: a. Materials produced by grade level and subject b. Materials distributed by region and district c. Use of the materials by teachers and children d. Student learning e. Local book printing and distribution 2. What have been the strengths and weaknesses of program management? 3. What are the major lessons learned? 4. Regarding sustainability, to what extent has the Ministry taken ownership of this program and prepared for its continuation? 5. What are the major recommendations for improving the program and to help ensure its continuation? 3.3 Limitations of the Study While, as intended, the evaluation did encompass schools in areas where different languages predominate, the major limitation was insufficient time to conduct an evaluation that included schools outside a radius of one day’s travel each way from Accra. This meant, particularly, that schools in northern Ghana, where opportunities for education have historically been limited, were necessarily left out. In addition, the time limited the number of officials and partners that could be interviewed. Another limitation to the study was the fact that the Ghana trip was postponed multiple times to accommodate USAID Mission scheduling requests, making it necessary to replace local consultants, whose schedules could not accommodate changed dates. The country visit to Ghana was to have been Evaluation of TLMP – TLMP in Ghana 6 the first to be conducted (in November 2012) but proved to be the last of the six country visits (June 2013). By the time the evaluation was actually undertaken, CSU was very involved in its transition and merger of TLMP materials in literacy, numeracy and science from KG1 to P3 with the NALAP teacher’s guides, a USAID/Ghana program launched in 2010 and adopted by the MOE as the standard curriculum for the early grades. So much attention was diverted to NALAP by CSU and the MOE/GES that by the time of the evaluation, many TLMP activities had almost become “invisible,” making it extremely difficult to collect data specifically on TLMP at any level. Another important limitation related to efforts to collect some achievement data from KG2 students and to observe TLMP classes. The majority of classes visited were no longer using the full TLMP program as the workbooks had been filled out by the children during the first year (as instructed) and, thus, were no longer available for use. Also, many teachers were now using the teacher’s guide from NALAP, as per MOE policy, and using TLMP only as resource materials. There were some exceptions to this, especially in the Volta Region, but the number of children tested was very limited and classes were observed in only a few schools were TLMP was still in use e.g., Living Star (private in Ga East, Denguanao and St. Mary’s (private) in Koforidua. In addition, a couple of classes visited in the Volta Region had been strongly influenced by the new Israeli-sponsored program MASHAV that emphasizes low-cost education materials, activity-based learning, and the elimination of separate subjects on the timetable in favor of a thematic or topical approach to organizing learning. GES recognized that many KG teachers had been confused about being presented both TLMP and NALAP programs. TLMP emphasized English and stuck closely to the official KG curriculum, whereas NALAP emphasized L1 and did not adhere as closely to the official curriculum. But, in the end, NALAP became the official KG curriculum and materials were distributed to all schools and KG1 – P3 teachers received initial training on NALAP. The evaluation team visited some districts (Ga East, Effutu, and Cape Coast) and schools (Philip Quaque Girls’ School, Cape Coast and Effutu Municipality KG and Primary School, Winneba/Effutu) where the District Directors and head teachers didn’t even recognize the name TLMP, even though the TLMP materials had, in fact, been delivered to the districts and schools. In some cases, the individuals arrived after the materials had been distributed, but in other cases TLMP no longer had name recognition. Nevertheless, given the combination of NALAP officially replacing TLMP and the fact that in many schools the TLMP workbooks had been used up, it was not surprising to find that TLMP was no longer being used in the majority of classes visited by the evaluation team. The exception was in Volta Region. 4. FINDINGS AND CONCLUSIONS We begin this section of the report with the data collected at CSU. We then move on to the output of the program so that the reader is aware of the number of items produced and to whom they were distributed. Findings and conclusions on the implementation of the program are then presented, as are the results of teacher comments and student reading assessments. 4.1 CSU Achievements and Challenges 4.1.1 Achievements CSU counts the following among its achievements associated with TLMP: • 10 CSU College of Education faculty members and doctoral candidates were engaged with the MOE and the GES in design, writing, production and assessment activities. • 20 CSU students were supported as TLMP interns in Ghana during the grant period. Students worked as researchers, observers and assistants in primary schools and returned to share their experiences with the CSU community. Two Ph.D. dissertations were produced by CSU students with data from TLMP. Evaluation of TLMP – TLMP in Ghana 7 • Initially, extra salary was provided to CSU faculty for participation, but now participation entails obtaining released time; this has produced better quality and reduced any conflict over extra salaries in departments. • The CSU College of Education created the Center for International Education (at the University). • Faculty development in the international field has taken place. • Papers were delivered at national, professional meetings including six papers presented at CIES • International experience is part of the tenure process. • The diverse experience of faculty prepared them better to address Chicago international education issues to help keep kids in school. They want to bring the lessons learned in Ghana back home – CSU developed international experience to teach students locally. • CSU developed skills and confidence in working with USAID. • CSU strengthened its administrative systems, so that it could wait for reimbursement. This helped to increase CSU’s credit rating. The board of trustees wanted formal reports, so greater accountability was established. • TLMP established national and international partnerships with private sector and commercial entities and raised funds to support the TLMP goals and objectives in the US. For example, United Parcel Services (UPS) donated $6,000 toward shipping. Hinsdale Medical Supply donated 3,000 emergency kits, and UBS Promotional donated 5,000 mathematical sets and school supplies. Buck Press, the Ghanaian printer, donated distribution and warehousing services valued at over $150,000. 4.1.2 Challenges Overall Although CSU had some important successes, it also faced many challenges, some of which were a result of early design decisions. Others evolved over the course of implementation. • There was a major design issue with TLMP as a whole from the outset in that the focus was almost entirely on the development and production of materials, with very little attention being paid to the training of teachers in how to use the materials that were produced. TLMP in Ghana was somewhat atypical in that there were plans to train 5,000 teachers, although ultimately only about 2,000 teachers were trained under the project. In Ghana as in other countries, the teachers themselves generally had relatively little experience in using textbooks or learning materials in their own learning, and training in the use of textbooks is by no means a systematic part of in-service training and is not included in pre-service training. • CSU’s financial management was an issue from the beginning as senior TLMP leadership at CSU was not as effective as was expected during the AEI phase, largely because of lack of internal capacity and skills. • Cutbacks in USAID funding prevented the translation of the materials into Twi, which – to make the materials useful – necessitated a USAID/G response to merge TLMP into NALAP to translate at least the teacher’s guides into Twi. • In addition, lack of funding precluded printing of the P1-P3 materials and the reprinting of the KG workbooks. • CSU had difficulty being taken seriously by GES as the Ghanaian partners considered the results of the initial needs assessments as inadequate. • CSU was beset with lack of cooperation by GES in identifying teachers for and participating in the TOT. Challenges presented by GES and MOE created a number of difficulties in project implementation that led the CSU leadership team to believe that a full commitment by to TLMP was never made and that it will not be sustainable. • The findings of the assessment report in 2010 (an edited version appearing in 2012) were not addressed in developing the framework of operations, leading to a number of ambiguities and Evaluation of TLMP – TLMP in Ghana 8 misunderstandings. • While USAID/G provided only limited support during the AEI phase, during the expansion phase it became totally involved and, in fact, took over financing and management of TLMP in 2012. • GES and CSU jointly decided that children should be given disposable workbooks with no firm plans to reprint the workbooks. Thus, in about two-thirds of the classes visited by the evaluation team, the children had written in the workbooks (as directed by the project), and, thus, the workbooks were not available in subsequent years. As a result, TLMP stalled in many classes. • KG teachers had limited English language ability and teachers had difficulty in understanding the directions provided in the teacher’s guides. • The evolving change in the direction of language policy in the MOE created confusion between TLMP and the USAID/G program, NALAP (see discussion below). As discussed above, although the MOE was interested in strengthening English when TLMP began, soon thereafter the MOE shifted its emphasis to the use of mother tongue or L1. In early 2010 NALAP (a USAID/Ghana-supported project) took the place of TLMP as the major curriculum in the early grades, and this has had a major impact on TLMP. NALAP quickly became the focus of the Ministry and is now the official language and culture program for KG1 – P3. NALAP has three major components: 1) the development of teaching and learning materials in eleven Ghanaian languages; 2) the publishing and distribution of over 5 million textbooks, teacher’s guide and other materials such as “big books”; and 3) the training of 80,000 teachers. These components have been completed and NALAP has been implemented nationwide. It focuses on L1, although it also includes a limited amount of English instruction. Thus, the initial emphasis of TLMP on English turned out to be a challenge under NALAP. 4.2 TLMP Output TLMP was initially launched in 14 pilot districts that were considered by the MOE as among the most deprived in terms of resource allocation for effective teaching and learning and were also facing serious challenges with regard to gender equity. Criteria used in choosing these districts includes the lack of: access road, accommodation, transport, electricity, communication, water, health facility, school infrastructure, and teachers. Schools were chosen in the following regions/districts. PILOT DISTRICTS CHOSEN Region District District Capital Ashanti 1. Amansie West Manso Nkwanta Brong Ahafo 2. Pru Yeji Central 3. Assin North Assin Fosu Eastern 4. Birim North New Abirem Greater Accra 5. Accra Metro Accra 6. Ga East Abokobi Northern Region 7. Sawla-Tuna-Kalba Sawla 8. Saboba 9. Chereponi Saboba Upper East 10. Garu Tempane Garu Upper West 11. Nadowli Nadowli Volta 12. Krachi West Kete-Krachi 13. Ho Ho Western 14. Bia Bia The major deliverables of TLMP were the development, printing and distribution of instructional materials – teacher’s guide, workbooks, and wall charts or posters. • 8,170,172 teacher’s guides and workbooks were printed and distributed by Buck Press under Phase 2 for kindergarten classes throughout the country. This number includes 4,149,353 million books that were printed for national distribution in 2010/2011; this number exceeded the CA by Evaluation of TLMP – TLMP in Ghana 9 more than 100 per cent. The approximate cost of each TLM was $1.28. • 220,000 laminated, colored wall-charts were developed and printed by Chicago Laminating. Inc. based in Arlington Heights, Illinois at a cost of $136,710. They were shipped to Ghana and distributed for use as supplemental teaching materials. Some information obtained from interviews and our document review identified a number of anomalies in distribution that could not be clarified. • Communication between TLMP leadership and District Education Directorates was not effective enough since 20 - 60% of the Directors and Store Keepers did not receive the information that materials were coming (Monitoring and Distribution Report 2012. GES. P. vii). This assessment was verified by the visits of the evaluation team to district offices that were part of the national distribution where the team was told that neither the District Director nor the Early Childhood Coordinator were informed about TLMP when the materials were sent to the District Logistics Office. • The quantities of materials delivered at the various regions and districts matched the enrolment of KG1 and KG2 pupils in the majority of the Districts. The mismatches increase as one moves from the District level to the school and classroom levels. 4.3 Program Management and Partnerships 4.3.1 Program Management at CSU and in Ghana Lack of Adequate or Appropriate Staffing. CSU maintained a small administrative staff in Accra and did not have its own full-time technical staff. It relied almost totally on MOE personnel, especially in CRDD, to develop the materials, provide teacher training and to conduct some M&E. The small staff had an important positive aspect in that the direction of the project and the activities were firmly entrenched within GES, which, in turn, helped to ensure ownership by the Ministry. Nevertheless, the evaluation team questions whether it was a good decision for TLMP not to have any of its own long-term technical staff – no one in early childhood education, English and bilingual education, or monitoring and evaluation, either in Ghana or in the U.S. An early childhood specialist was added only after a review and change in program leadership in 2010. Although CSU provided some technical assistance in Ghana and training for MOE staff in the United States, TLMP seems to have been handicapped by not having professional technical staff of its own based in Ghana. This severely limited in-country technical assistance and training and, as described below, greatly limited important M&E activities. Insufficient Communications. The program did not have a strong advocacy component and communication was limited. For example, when the materials were distributed nationwide, there was communication with only some District Directors or the ECE Coordinators. As a result, although the materials did, in fact, get distributed to schools through the District offices, most senior officials were not in a position to offer support or guidance to the teachers. “… the two District Directors from each of the four of the Regions, namely Ashanti, Northern, Upper West and Western said they received information before the arrival of the materials at the District Education Office. As many as six districts from six Regions reported that they did not receive any information prior to the arrival of the materials” (Monitoring and Distribution report, p. 7). “The process of communication between TLMP Officials and District Education Directorates was not effective enough since 20 - 60% of the Directors and Store Keepers did not receive the information” (Monitoring and Distribution report, p. 26). Partly as a result of the lack of communication at the district level, follow-up teacher training did not take place either at the district or school levels. The evaluation team visits to district offices also confirm the lack of communication. As part of the national distribution of materials, the District Directors and EC Coordinators that the team met did not receive any information from TLMP about the materials being Evaluation of TLMP – TLMP in Ghana 10 distributed. Thus, they were not in position to support the teachers or to help implement TLMP. Inadequate M&E. While a monitoring and evaluation structure had been developed for the initial phase of TLMP in Ghana, for the expansion phase, there was no Performance Monitoring Plan (PMP), no annual needs assessment, and no regular data collection from school visits including interactions with the head teachers, teachers, parents, and children. In addition, there was only one small study that sought to measure the impact on learning of TLMP. Each of the quarterly reports provided by CSU to USAID/W documented certain activities, but the in-depth monitoring of the production, distribution and use of the materials was not undertaken consistently. However, the following assessments were carried out once CSU’s TLMP senior leadership changed in 2010. The information generated by these assessments was valuable, but still represented a limited M&E effort: • TLMP Program Assessment Report (March 2011) • TLMP Impact on Academic Achievement (March 2012) • TLMP Field Test Results (Basic 1 thru Basic 3s) (June 2012) • NALAP Formative Evaluation Report, Ghana (April 2011) 4.3.2 Partnerships with MOE CSU’s primary interaction with the MOE was with the CRDD within GES. From the beginning, CRDD took a leadership role in TLMP, most importantly, determining in the beginning that the program should support the newly incorporated KG classes, and that the materials should be in English and cover literacy, numeracy and science for KG1 – P3. In addition, CRDD took the lead role in developing the new curriculum and materials, with CSU providing important training and technical assistance. From the perspective of CSU, however, it appeared that CRDD initially only wanted the materials and did not express an interest in support for teacher training, training in the US, or in evaluation. CSU stated that working with the GES was problematic. GES would not provide a list of all those who needed to be paid with the topping up funds. Also GES did not want a “project” run by outsiders. To overcome any ambiguity, the project director established contracts with all people and put money in the GES account designated for teacher training food and lodging and required bills to support the use of funds. These administrative issues aside, the senior GES officials including the Director of CDRR, the EC Coordinator, the Deputy Directors General, and the Director of Basic Education all spoke positively about TLMP and its role in helping to meet the important needs for KG. The Director of CRDD noted the following: • Children can't speak adequate English in P4. • There are different schools of thought regarding English and L1, yet the high-stakes exam at the end of junior high school is in English and there is much parental pressure to emphasize English language instruction. • CRDD doesn't have language specialists and is understaffed, with only 18 of some 45 positions filled. • There is only one semester on local languages in the colleges of education, which is not enough to develop necessary L1 skills for the teachers. • There is need to change attitude towards L1 as much of the public remains supportive of emphasizing English language instruction. • The TLMP program created and distributed a large number of books, which helped to close the gap in pupil to book ratio. • Capacity was strengthened at CRDD, including in computer graphics. • CSU has provided some technical expertise, especially language specialists. They provided five days of training at CRDD, sent people to the CIES conference, and provided one month training in the Evaluation of TLMP – TLMP in Ghana 11 United States for some CRDD staff. Although CRDD was the key partner for TLMP and made major contributions in terms of materials development, the project would have liked for CRDD to have had greater ownership, not just assigning persons to work on the project, but to have been more pro-active and to have provided additional financial commitment, not just in-kind contributions. The GES Teacher Education Division (TED) is responsible for quality assurance in education. However, it appears that TED has been completely sidelined with respect to TLMP. For example, its Director informed the team that he was never consulted about TLMP, was not aware on the initial TOT and has no report of any trainings by TLMP, and had never seen any of the materials prior to the interview. The evaluation team met with district officials in the following seven districts – Accra Municipality, Ga East, Koforidua (New Juben Municipality), Cape Coast, Effutu Municipality, Jasikan, and Ho. In every location, the team also visited schools. The following is a brief summary of the highlights relating to TLMP of the district and school visits. This summary excludes visits to universities and colleges of education, which are described in the next section. Koforidua (New Juaben Municipality) – On June 12, the team split with some members going to Koforidua where they visited the Denguano M/A Basic School and St. Mary’s Day Care and Preparatory School (private). Accra – On June 13, the team visited the Accra Metro Education Director Mr. Alfred Kofi Osie and Mrs. Grace Agyepong Dankyria, the ECE Coordinator, and then visited the following schools: • Osu Presbyterian KG School (actually three schools). This was a well-supported public school. The TLMP materials had arrived and were used for two years, but then the workbooks were used up and, thus, they are not using TLMP any more, although the teachers still use the teacher’s guide as a resource material. Of particular interest was the new Israeli MASHAV program, which has a strong emphasis on locally available materials, active student involvement, and school days organized by topics rather than subject matter. • Osu Presbyterian KG School Annex. In contrast to the above school, this one was small and impoverished with no parental support. TLMP was welcomed, but then discontinued when the workbooks were used up. The teacher’s guides were still being used as a reference. Ga East – On June 14, the team visited Ga East Municipal Education Office and met with the Director Ms. Florence Damali. She was not familiar with TLMP. • The team met with staff at Pantang Presbyterian Basic School. They used the TLMP materials for one year only until the workbooks were used up; the teachers still refer to the teacher’s guides. • The team then visited the private Living Star school. The school received TLMP materials for two years. The workbooks were used up, but the parents would buy them if available on the commercial market. The team visited four KG classes. Cape Coast – One sub-team visited Cape Coast on June 17. Neither the Deputy Director for Supervision (previously a Circuit Supervisor) nor the ECE Coordinator had any knowledge about TLMP. There had been no contact with the District office when the materials were sent to the Stores. • Philip Quaque Girls’ School. The school used TLMP for one year and now uses NALAP. Sometime TLMP materials are used supplementary activities or for homework. • Abakam CRAN Presbyterian School. The head teacher was not there, but the KG2 teacher had not heard of TLMP. • Golden Treasure International School (private). This school did have some TLMP workbooks for KG2, although most were at least partially filled out. Only with effort did the KG1 teacher find the teacher’s guide (no workbooks). Evaluation of TLMP – TLMP in Ghana 12 Effutu – The Central Region team visited Winneba, Effutu Municipality on June 18 and met with the Director Ms. Hilde Eghan and Ms. Thelma Essuman, ECE Coordinator. The Director has heard of TLMP, but only based on her previous work at GES in Accra. She has not heard of TLMP in her current position (only three months). The ECE Coordinator had heard of TLMP about two years ago at a meeting, but was not informed when the TLMP materials were delivered to the District Stores. The ECE Coordinator said that two teachers had come up with ways to extend the life of the workbooks, including one teacher who had the children write with chalk. The teachers valued the workbooks. No training was provided to the teachers, although the ECE Coordinator thought that use of the materials was straightforward. Teachers are now using NALAP, but use TLMP as a resource. • Effutu Municipality KG and Primary School where the head teacher was not familiar with TLMP. However, the TLMP had arrived and there were many workbooks still available, some fresh and some partially used. The teachers were mending the books and also trying to erase what the student had marked. • ACM KG and Primary School. Only a couple of posters and teacher’s guides in sight. The workbooks were used for one year only. • Ebenezer Memorial Education Centre (private). Again, they used the TLMP books for one year and, currently, the teacher’s guides are just a resource material. Jasikan. The Volta Region sub-team visited Jasikan on June 17, where the Deputy Director Mr. Dompreh and J. K. Daniel verified that they had received TLMP materials. • Jasikan College of Education Demonstration School. The team administered a basic literacy test to eight students. Ho. On June 18, the Volta Region team visited the Ho Municipal Education Directorate and met with Mrs. Dora Agorsor, Director, and Mrs. Beauty Agbeve, ECE Coordinator. This was a TLMP pilot district and they were supplied twice with materials. • Mawuli E.P. Primary School. The second set of books arrived in 2012 and the children were writing in their exercise books in order to preserve the workbooks. • Kpenoe E. P. Primary School. The teachers use the books, although instruction is largely in L1. The team tested 8 students. Accra. On June 19, a team member visited the North Legon Little Campus (private), but found that they have had no experience with TLMP. 4.3.3 Partnerships and Linkages with Colleges of Education and Universities Regarding partnership with the universities and colleges of education, the efforts were mixed. Staff from some of the universities and colleges were involved in helping to create the new materials and to support teacher training initiatives, but when the evaluation team visited the University of Cape Coast, the University of Education, Winneba (UEW), and four colleges of education, they found only limited familiarity with TLMP. Almost none of the staff met by the evaluation team knew about TLMP. In addition, none of the individuals met recalled receiving any of the TLMP materials. There is no evidence that any of the pre-service teacher training programs at the universities or colleges included any TLMP materials or methodology. TLMP signed MoU with University of Cape Coast and University of Education, Winneba to collaborate in the production KG materials. At Cape Coast, three senior lecturers were selected for the implementation committee. Various workshops were held during the first year, but the University withdrew because CRDD took over and sidelined the University. The University did not receive TLMP materials. At the University of Education, Winneba, the team met with a senior lecturer in the Department of Evaluation of TLMP – TLMP in Ghana 13 Early Childhood Education. Although they teach a course on the preparation of TLMs using local materials, they do not have the TLMP or NALAP materials and no library of their own. The lecturer says that TLMP never involved UEW or those lecturers who were training teachers. He has never seen the TLMP materials. The Principal at OLA College of Education was aware of TLMP only because she attended a workshop on the merger process in her capacity as Secretary for the Association of Principals of Colleges of Education. Otherwise she has not been aware of TLMP and has never received any of their materials. This is unfortunate because they teach the preparation and usage of TLMs in schools. At Aburi College of Education, the senior officials said that they had not heard about TLMP and have never used any of the TLMP materials. The Jasikan College of Education began its Early Childhood Education program in 2008 following a selection process by TED. The Coordinator of ECE said that he had never heard of TLMP and that the college does not use the TLMP materials. One of the reasons for visiting this College is that they are active participants in the new Israeli MASHAV program. From the visits to the universities and colleges, it appears that TLMP did not reach out to the pre￾service teacher training institutions and did not send them TLMP materials. In the initial stages, according to TLMP documents and the report of one senior UCC lecturer, TLMP did invite some individuals to help create the new materials, to assist with the teacher training activities, and for membership on the PAC. Yet, these initial important steps seem not to have carried over into the training programs at these institutions. The training efforts of TLMP focused only on in-service teacher training. 4.3.3 PAC and PTA The Program Advisory Committee (PAC) was established in 2006 with senior level personnel from CSU and the Ministry – Ministry, GES, the Health Book Development Council, the Faculty of Education of the University of Cape Coast and the University of Education, Winneba, among others. The following February PAC held a workshop that established three teams – Design, Writing, and Assessment. Shortly after this meeting, the Assessment Team selected already existing materials for printing. PAC conducted a materials assessment and then identified content area experts to develop the literacy and environmental science materials at both KG1 and 2 levels. They reviewed the national standards to conform the books to them, and identified the topics to be covered in the content. PAC used to meet 2-3 times a year and played an important role as described above. However, PAC had not met for many months before the arrival of the evaluation team. In practice, PAC has evolved into PTA (Program Technical Advisors) that are attuned to activities associated with Modification 8 to the CA. It brings in relevant parties and adds individuals as may be needed. The PTA committee is made up of four CSU experts and four Ghanaians. Typically the PTA invites principals of the colleges of education, teachers, lecturers and other stakeholders to meetings. The last planning meeting took place in June 2013, after the evaluation team left Ghana. 4.4 Program Implementation 4.4.1 Materials Development After the initial printing of existing and revised teachers’ guides and student workbooks, the next materials were developed at CRDD using some of its own staff, plus many individuals from the outside, especially from the Universities and Colleges of Education. CSU provided initial input into some of the design factors, conducted a workshop at CRDD, and brought some CRDD staff to Chicago for training. Nevertheless, this was principally a CRDD activity. Evaluation of TLMP – TLMP in Ghana 14 CRDD then sent the draft materials to CSU for review and feedback. This was a valuable step. CRDD gave very useful feedback, which was then reviewed by CRDD. Most of the feedback was incorporated into the final documents, but some suggestions, especially in the illustrations, were felt to be culturally inappropriate. Once CRDD revised the materials, it field tested them in a range of schools including different regions and both urban and rural locations. Final revisions were then made before the printing and distribution. A significant design flaw was the decision to use disposable workbooks at the KG1 and KG2 levels. The use of workbooks was strongly valued by the MOE, but neither the MOE nor TLMP had projected funding to continually reprint and distribute them. As a result, in the majority of classes that the evaluation team visited, the use of TLMP materials largely collapsed because the workbooks had been consumed in the first year. Just five of the sixteen public and private schools visited were still using the workbooks. In the Central Region, all but one class visited by the team had used up the workbooks and, as a result, had largely stopped using TLMP. However, in the Volta Region, the team found most classes had figured out a way to continue to use the workbooks: teachers copied the exercises on the chalkboard and then children copied them in their regular exercise books. “The majority of the pupils confirmed that they were not allowed to take the materials home ... Head Teachers and Teachers gave realistic reasons for not allowing the children to take the books home. Parental commitment and involvement are required in order to allow children to take home the books” (Monitoring and Distribution Report, p. 29). However, many schools reported that, at the end of the year, the children were given the workbooks that they had filled out and took them home for good. 4.4.2 Materials Assessment by Evaluation Team The in-country evaluation team had the following comments on the TLMP materials: • The TLMs are colorful, attractive and well laid out. • There are a few pictures that are gender-stereotyped or inappropriate in Ghana (One exception was a picture of a girl and a boy. She holds a doll, he a car). • The TLM teacher’s guides and student books seem to be designed to teach reading to students who already have some understanding of English. • The workbooks and teacher’s guides do not have glossaries with translations of vocabulary in a Ghanaian language. • There is no guide to pronunciation or exercises in sound recognition. This would have been useful since sometimes L and R sounds are reversed by Ghanaian speakers, “TH” sounds are not pronounced clearly and the stress pattern of English sentences during oral work seems to follow that of Ghanaian languages. • There is no presentation of grammar. Tenses are limited to the simple present and present progressive when describing pictures in the student workbook. It would be difficult for students to “talk about” some of the more complex pictures in their workbooks with much spontaneity unless there are some exercises with verbs, prepositions and conjunctions. • Overall, the keen language-learning ability of young children is not put to adequate use. • In general, the materials seem not to take into adequate account the fact that the children are not English-speaking. • There are lists of vocabulary words, but no definitions or translations into any Ghanaian languages. • There are two-word verbs such as “sit down,” “get up,” and “lie down,” but no guidance about how to teach these verbs other than via illustrations. The difference in meaning between, e.g., “sit down” and “sit,” “lie down” and “lie” needs to be explained. • The simple present, past and past progressive tenses appear in the student books and need explanation. • More relevant materials to northern areas of Ghana. Evaluation of TLMP – TLMP in Ghana 15 • The materials need a pacing chart. • There should be more 3-4 letter words to match pictures. • Pictures need labels and better identification. • Some drawings are not in natural colors. • Some drawings are too difficult. Previously, the evaluation’s Team Leader, Dr. Horn, conducted a detailed review of several workbooks and teacher’s guides, reflecting (a) pedagogical quality and (b) physical and design quality. Annex G begins with a delineation of the major evaluation criteria for the materials and continues with detailed analyses for KG1 in literacy and numeracy, and KG1 and KG2 environmental science. In general, TLMP can be commended on the quality following the workbooks and teacher’s guides. In Literacy for KG1, some negative comments include – children looking sad, inappropriate illustrations, emphasis on sight reading rather than phonics, lack of assessment activities, and marginal durability. In numeracy, overall very good, but it may have been better to start with numbers rather than classification. The KG 1 Environmental Science has many strong characteristics, but may draw on vocabulary and writing exercises that are beyond what has been taught in the literacy lessons. The only materials that were printed and used in classrooms were for KG1 and KG2. However, TLMP did complete the development and field testing of P1-P3 teacher’s guides and student workbooks in literacy, mathematics, and natural science, a total to 18 books. However, there are no funds in TLMP to print these materials. In addition, the national curriculum has changed, so there is need to review these materials. It seems likely that the mathematics and science materials would be useful in supporting the existing curricula in these subjects. And, as part of the merger process, these materials were reviewed and some content and activities have now been incorporated into the NALAP teacher’s guides. 4.4.5 Teacher Training Teacher training was included in this CA from the outset. CSU held two training-of-trainers (TOT) workshops for educators in 2008, one in Sunyani in Brong-Ahafo and the other in Koforidua in the Eastern Region. Trainers were equipped to facilitate workshops with 300 KG teachers in the 14 districts on the use of the textbooks and learning materials. Based on this experience, CSU determined that during the 2009-12 expansion period, other TOTs would be held, this time targeting 2,000 additional teachers. The objectives of this latter training were to: • Enhance teachers’ expertise in using the TLMs • Equip them with a deeper understanding of the best practices for use of the materials • Prepare participants to use the TOT model to disseminate best practices to teachers The first tier of the TOT model was implemented in May 2011 to 24 trainers and was evaluated to make decisions about the effectiveness of the TOT model in providing professional development workshops. Five training modules were created focusing on the following areas: how children learn, learning styles, teaching methods, teaching strategies, and songs, poems. Districts that participated in this training included: Accra Metro; Tema Metro; Ga East and West Municipal; Kokuku Krowor Municipal; Akwapen North; Birim North; Kwahu North; New Juaben, and Suhum Kraboa Coaltar from the Greater Accra and Eastern regions. Training was seen as a two-step process: the TOT provided to district level facilitators, and the delivery of workshops by these facilitators in their respective districts. It is unfortunate that only the first part of this training was evaluated and there were no efforts to follow up with the facilitators in the field to determine how any subsequent workshops were implemented with teachers and how frequently this was done. However, CSU compiled and published a book on the topic, CSU Impact on Literacy in Ghana, West Africa (ISBN 978-0-9837851-3-2), which includes the needs assessment, mid-term assessment, teacher training handbook and modules, two doctoral dissertations, Evaluation of TLMP – TLMP in Ghana 16 guidelines for writing, editing and formatting TLMs, guidelines for researchers, and the sustainability report. Portions of this book were provided to the team, but an outcomes/learning assessment was not included. This was to have been completed by CRDD and other Ghanaian partners. Under the initial terms of the CA, CSU envisioned training 5,000 teachers. Several issues prevented the reaching of this target: 1) the lack of cooperation by the GES in identifying teachers, especially in light of the adoption of the MOE of the NALAP curriculum; 2) USAID/W’s budget cuts that prevented payment of accommodation, fees, and payment of facilitators; and 3) failure of GES and printers to agree in a timely manner on the distribution of books. The final number of teachers to be trained was 2,000. Facilitators were to have trained lead teachers at schools in each district, who then were supposed to train their KG colleagues. GES started but then did not complete the selection process throughout the country. Hence, the training covered only those schools in the 14 districts that had been part of the pilot (although TLM distribution, ultimately, was national). Providing an honorarium for lead teachers was not recommended. Moreover, it was determined that KG teachers, especially in the deprived areas, lacked the competence to use the TLMs due to the fact that most of them were untrained, the materials were new in the GES curriculum, and many could not communicate well in English. This made using the teacher’s guides extremely challenging for teachers. 200 LTs were to have been chosen; many GES staff attended TOT along with the 150 teachers. Trainees were district trainers who were to train the teachers in their districts, but only one did this – others did not as there was no incentive for cascading. When the TOT was presented, it was supposed to be first in a series, but then the budget was cut and the TOT had to stop. Unfortunately, a report on the training was never written or the documentation is missing. The GES did one TOT but never shared their materials. In spite of the TLMP training of 2,000 and the subsequent TOT, the evaluation team did not come across a single teacher who said that he or she had received training on TLMP. 4.4.6 The Current and Anticipated Future Status of CSU and TLMP In some ways TLMP seems to have stalled during the past year. All the KG materials had been developed, printed and distributed and many teachers trained. And the P1-P3 materials were developed, but there are no funds for printing. Also, even in the schools, although some teachers had used “work-arounds” such as having learners write in regular exercise books, the majority of teachers met by the evaluation team had stopped using TLMP as the student workbooks have been used up. In addition, NALAP materials had been distributed and instructions given to teachers that NALAP was now the official program. Nevertheless, TLMP was developing and implementing an important exit strategy consisting of the following: 1. The merger of key concepts and activities from TLMP into the NALAP teacher’s guides. The goal of the merger is to capture and combine the best practices of TLMP into the NALAP program in terms of methodology, content, and activities in order to strengthen the English component. In addition, some of TLMP practices would also strengthen the dominant L1 program within NALAP 2. To reach consensus concerning the proper balance and usage of L1 to facilitate a smooth and successful transition to English instruction by P4. 3. To print, and distribute the revised NALAP teacher’s guides. 4. To train all KG1-P3 teachers in the new teacher’s guides 5. To use effectively technology to reduce costs and to ensure a high quality teacher training program. The evaluation team attended one of the workshop activities where the merger or fusion process was taking place, a process that began in October 2012. Although there still wasn’t total clarity regarding the possible future existence of TLMP materials, the subject matter teams had make much progress and Evaluation of TLMP – TLMP in Ghana 17 expected to complete the merger of the materials by the end of July 2013. Shortly after the evaluation team departed Ghana, TLMP held an important workshop in June including the CSU project director that brought together the key players in Ghana to review the merge process including some of the new materials, and to finalize the strategy to train over 100,000 KG-P3 teachers at the end of 2013 and the beginning of 2014. A part of the strategy is to create and distribute DVDs with model lessons, presentations on methods on teaching basic reading skills, etc. that can be used to support the training process and even reviewed by individual teachers at a later date. The DVDs should help to ensure that the quality of the training is maintained throughout the cascade process. TLMP plans to train a 54-person National Resource Team (5 days). These people, in turn will train 1,100 master teachers (3 days) at the district level, who will then train approximately 100,000 KG1-P3 teachers. 4.5 Outputs and Outcomes 4.5.1 School/Teacher/Classroom Observations The evaluation team visited seven district offices and sixteen schools (eleven public and five private) in four regions. The predominant local languages in these regions are the Fante and Twi dialects of Akan, Ga, and Ewe. School Location PUBLIC Osu Presby. Accra (Greater Accra) Osu annex Accra Abakam Cape Coast (Central Region) Philip Quaque Cape Coast ACM Effutu (Central Region) Effutu Effutu Presby. Basic School Ga East (Greater Accra) Kpenoe Ho (Volta Region) Mawuli Ho Jasikan Demo Jasikan (Volta Region) Denguano Koforidua (Eastern Region PRIVATE North Legon Accra Golden Treasures Cape Coast Ebenezer Effutu Living Star Ga East St. Mary’s Koforidua (*) Schools marked in italics were still using the TLMP workbooks. (**) Schools in bold were in the original 14 districts, which were identify as some of the deprived districts. The other schools were added as part of the national distribution of materials. The major findings from visits to the sixteen schools are as follows: • All but one of the private schools and all of the public schools had received the TLMP materials. The Ministry approved the distribution of the materials to properly registered private schools, but Evaluation of TLMP – TLMP in Ghana 18 there is no information on the number of private schools that received the materials. • All of the private schools emphasized the teaching of English; one school had a sign in the KG classroom stating “English is the medium of instruction.” Given the emphasis on English, it would appear that the TLMP materials would continue to be very useful. • Of the sixteen schools visited, only five schools or 30% still had a supply of workbooks and were using them. This included only one of the private schools as well. • All of the schools spoke highly of the materials, especially the workbooks, which they said were important learning tools for the children. • The majority of the schools (10) said that they had used up the workbooks, with most schools having the children take home the books at the end of the school year. Except for two schools that had received a surplus of books, these schools had no more books for the following year and, thus, effectively stopped using TLMP. • The five schools that still had books did so as the result of teachers who realized that more workbooks would not be forthcoming, and they figured out ways to preserve the books. Some teachers arranged to erase most of what the children had written. Other teachers had the children write in their exercise books. A few teachers even tried to photocopy some pages, but this proved too expensive. In short, the decision to use disposable workbooks was ill-advised, and even when the materials were sent to all districts, there was no advice from TLMP about strategies to preserve the books. • Some teachers said that if the workbooks were available on the open market, they would encourage parents to buy them. • None of the teachers met by the evaluation team had received any training on TLMP. • The posters were in evidence in most schools and are likely to be the longest lasting of the TLMP materials, although there was little evidence that they were actually being used. The TLMP materials could continue to be of special value in both private schools and public schools in cosmopolitan areas where there are so many local languages that the school needs to use English as the medium of instruction. 4.5.2 Learner Reading Assessments and Outcomes There are two sources of data on learning achievement. The first, described below, is an English test administered by the evaluation team to 24 students in three schools in the Volta Region. The second is a test administered by TLMP. The Assessment Instruments and Process The purpose of the materials prepared by TLMP is to help Ghanaian children in kindergarten to prepare for listening, speaking, reading and writing in English. A learning assessment instrument was developed by team member Dr. Eric Allemano to measure student performance in English – speaking, listening, reading and writing (included in Appendix C) and was administered to children in three schools in and around Jasikan and Ho in the Volta Region. The sample was comprised of a total of 24 students – two boys and two girls in both KG1 and KG2 in each school. The sample is small because so few schools visited by the team – five of the sixteen public and private schools – were still using the TLMP program. The remaining schools never used the materials or had stopped using TLMP once the workbooks had been filled out by the children and, thus, were no longer available for subsequent years. Because of the limited sample, it is not possible to generalize from these results to the impact of TLMP on student learning. Nevertheless, the results give some indication of the ability in English of at least these KG1 and KG2 students. Certain factors made the assessment challenging. First of all, the children were very young (aged 4 in KG1 and 5 in KG2) and had little or no contact with English outside of the classroom. Another factor was the reduced number of hours devoted to English since the introduction of instruction in national Evaluation of TLMP – TLMP in Ghana 19 languages under NALAP. Although the research was done close to the end of the 2012-2013 school year, the students in the Volta Region had covered only half of the lessons in the TLMP materials for their grade level. As a result, the children had had only half of the instruction that they were supposed to have received and their vocabularies were very limited. A further factor made it difficult to assess letter recognition in English. In Ghana, children generally learn the sound of a letter rather than its name. For instance, they learn that the letter A is ah rather than “Ay”, the letter B is bh rather than “Bee” and the letter C is kh instead of “See”. This made it challenging for the American researcher to test letter recognition among the students as he was not certain of how “C” and “K” were differentiated or whether vowels were long or short in the Ghanaian system. Literacy and speaking skills were assessed by five methods. 1. Listening and speaking were tested by asking the child to answer a series of questions about himself/herself in English (e.g., What is your name? Are you a boy or a girl? How old are you? What grade are you in?). 2. Letter identification was conducted by asking children to name 10 letters in English (both upper and lower case). 3. Oral (speaking) fluency (word accuracy and phrasing) was assessed by asking KG2 students to describe a picture they were familiar with in the TLMP workbook (for example, a boy sitting at a table, reading a book near a bookcase, a woman in a shop, choosing fruit). The exercise was too difficult for KG1 students so they were asked to name items that the researcher pointed to on the TLMP wall charts (which did not have word labels). 4. The KG1 students were also asked to identify pictures with words (ex. fish, in the workbook) and to read the word under the picture. Word recognition for KG2 students was conducted by having the children read 10 words drawn from the KG2 workbook. 5. Writing competence was measured by asking the child to write three letters, write his/her name and/or one word of her/his choice. Results The children assessed had success in certain language skills (word recognition and naming pictures of familiar objects) and great difficulty in other areas (describing a moderately complex picture in the workbook). In doing word-picture assessment the researcher found that in some cases students seem to have memorized the word-picture sets in the order they appeared on the page. If the researcher chose the word-picture sets at random, some students had difficulty giving the right answer. Characteristic KG 1 KG 2 Comment 1. Listening & Responding 60% answered at least 1 question 100% answered at least 1 question 1 correct answer: usually the child’s name. 2. Letter Identification 70% identified 80% 90% identified at least 80% 3. Oral Fluency 0% able to describe a picture 25% able to describe a familiar picture in the workbook. The researcher had to ask probing questions. 4. Word Recognition 20% recognized 70% of pictures in the workbook 35% recognized 80% of the pictures chosen on a wall chart 35% identified at least 70% from workbook 65% recognized at least 90% of the pictures chosen on a wall chart Evaluation of TLMP – TLMP in Ghana 20 5. Writing 65% able to write 3 letters 65% able to write their names 15% able to write a word in English 90% able to write 3 letters 100% able to write their names 65% able to write a word in English Conclusion It is evident that students do improve their skills in English by the end of KG2 and that the TLMs seem to be correctly used. There is evidence that the TLMs help the students using them to outperform students in schools that do not use them, as shown by test results collected by CSU in 2012. However, it is difficult to compare the results of the testing done in the three schools visited in the Volta Region with those of testing done a year earlier by CSU. Literacy Content Domains TLMP Sch. Control Sch. Mean S.D Mean S.D Conversation 1.83 0.346 1.79 0.367 Visual Discrimination 2.70 0.519 2.56 0.622 Phonological Awareness 0.28 0.395 0.39 0.413 Auditory/Visual Comprehension 1.49 0.613 1.35 0.603 Awareness of Positions 1.01 0.840 0.74 0.772 Source: Assessment of the Impact of TLMP on Pupil Achievement in Some Selected Districts The results of the testing done by CSU indicate that with the exception of Phonological Awareness Content Domain, TLMP school students outperformed their control-school counterparts on content Domains in language and literacy. • Children do seem to make progress in English literacy skills as they move from KG1 to KG2 • Children perform better at tasks that are more behaviorist than cognitive (memorization over thinking) and may reflect a preference of teaching style and pedagogy • There were no obvious correlations between letter naming and word reading accuracy, or between letter naming and comprehension • Children in both KG1 and 2 have minimal expressive language facility (e.g., none of the children spoke in complete English sentences). • Given that most of the test items were drawn from the student’s grade level TLMP workbook or the wall charts, it is apparent that both the linguistic level and the instructional objectives of the TLMs are above most children’s language and reasoning abilities. The TLM reading method is quite reminiscent of “Dick and Jane”, the main characters in popular basal readers that were used for decades to teach American children to read. The books relied on the whole word or sight word reading method, and repetition, with some limited attention to phonetic analysis. This approach has been essentially replaced in English-speaking countries by various adaptations of phonics. However, the TLMP materials that were developed for Ghana do not include phonics or any exploration of sentence structure and grammar. . One difficulty of using the sight word reading approach with KG1 and 2 students in Ghana is that for the most part, they do not know English. While teachers may translate words and sentences, the TLMs appear to assume that the learner already knows how to speak English reasonably well but needs to learn how to read it. The absence of phonics/pronunciation exercises and of practice with sentence structure and basic conjugations is a further limitation to the TLMs in Ghana. An ESL/EFL approach would have been more appropriate for the vast majority of learners. Evaluation of TLMP – TLMP in Ghana 21 4.5.3 Other Evidence of Learning TLMP did conduct one study that measured learning achievement. It consisted of a limited number of classes and children.2 A total of about 474 students were tested in the three domains – literacy, numeracy and environmental science. Half of the children were in TLMP schools and the other half in control schools. The results showed only a small difference in favor of TLMP classes in literacy, and almost no difference in numeracy and science, as shown in the table below. Literacy Numerac y Scienc e TLMP 7.30 8.84 7.96 Control 6.65 8.78 7.96 It should also be noted that the TLMP Assessment Report, Final (2011) included a finding that, of the teachers who returned the questionnaire “91% had observed positive changes in the children’s achievement since the TLMP workbooks and materials have been used.” P. 33. But as clearly stated in the Report, this finding was based only on teacher perceptions and not any test data. 4.6 Educator Comments on TLMs Staff at all but one of the sixteen schools visited by the evaluation team were familiar with TLMP, but only five schools were continuing to use the workbooks on a regular basis. This notwithstanding, staff at most of the other schools indicated that the teacher’s guides and workbooks were good reference materials. All the teachers who had used the materials highly praised the teacher’s guides and workbooks. They especially appreciated that the teacher’s guides closely followed the curriculum and, thus, continue to be a good reference source. In addition, all the teachers spoke highly of the workbooks, saying that they were an important learning tool for the children. In schools where the workbooks were no longer available, and where parents were buying other workbooks in the commercial market, the teachers said that they would highly recommend to the parents that they buy the TLMP books, if they were available. 4.7 Stakeholder Interest In and Use of Materials/ Sustainability CSU worked with the Mission to try to ensure sustainability of TLMP, and PAC also sought ways to ensure sustainability. But the evolution and dominant role of NALAP means that the legacy of TLMP will likely be a stronger English component within the NALAP teacher’s guides resulting from the merger or fusion process. The GES senior personnel met by the evaluation team were very supportive of TLMP and valued the contribution that the materials and training made in support of the newly incorporated KG classes into the educational system. They would still like to see some of the TLMP content used to support the English language component of NALAP, which has been the major focus of the recent merger or fusion process. They would not like, however, to see a continuing separate identity for TLMP. Nevertheless, the TLMP materials are perceived as good, especially for teaching L1, and should be translated into local languages to help children learn to read. NALAP, with its focus on L1, but also with an English component, has become the new official language program of the Ministry and will subsequently be called just Child Literacy. 2 TLMP Evaluation Final Report, 2013 Evaluation of TLMP – TLMP in Ghana 22 The TLMP teacher’s guides that are currently in the schools can continue to be used as reference materials, and the workbooks that still remain in some schools can continue to be used until they are completely worn out. From discussion with the two GES Deputy Director Generals, the MOE is not interested in having these workbooks reprinted and, in any case, TLMP does not have the funds to do so. It should be noted, however, that NALAP does not include workbooks for the children, although the MOE recognizes the value of having them. Based on the TLMP experience, it would not be a good decision to develop and print workbooks to be filled out by children because the Ministry cannot afford to replace them every year. CSU and its partners have struggled with alternatives to disposable workbooks, which might be applied to other reading activities. Some ideas are as follows: • Use “Activity Books,” which may be similar in many respects to workbooks, but children would not write in them. The written exercises would be done in the children's exercise books. One variation of this Activity Book would be an occasional page that the teachers could photocopy for the children. Also, the teacher might copy some exercises on the blackboard. • Alternatively, should there be funds for some repeated printings, perhaps with plasticized pages of the workbook so that children could write on them, perhaps with crayons, which could later be erased. Costs could be reduced by using black-and-white illustrations for the workbook instead of color (Buck Press estimated a 20% savings.) The Director of CRDD noted that in any event, materials need to be replaced every three years. The Scope of Work for this evaluation did not call for review of current approaches for teaching literacy in Ghana. However, the team noted that the TLMP materials assumes a greater ability in English among the KG students than the team observed. The methodology, however, could be applicable for enhancing learning in L1. 4.7 Conclusions The overall conclusions of the field study are the following: Positive: • Support of KG. TLMP served a useful purpose, as the KGs had just been incorporated into the Ministry and there were not instructional materials. TLMP developed good teacher’s guides that were closely aligned with the curriculum, pus attractive workbooks and posters. The focus was on English language instruction, which met a felt need of the Ministry at that time. It helped to give important support to early childhood education and to developing the foundations for reading skills in English. • Materials. TLMP produced good teacher’s guides, attractive workbooks, and posters for KG1 and KG2 that were highly valued by the teachers. Virtually all of the persons whom the evaluation team met appreciated that the teacher’s guides were aligned with the curriculum and that the workbooks were effective in helping to children to develop preliminary skills for reading. • Printing and distribution. TLMP is to be highly commended for printing and distributing such a large number of good instructional materials – over 6 million. This is a major accomplishment. • Shift to focus on L1. Although the changing circumstances emphasizing L1 were outside of TLMP’s control, TLMP was still able to make a contribution by strengthening the limited English component of the new NALAP teacher’s guides through the “merger” process. • Local leadership. From the beginning, CSU sought to place responsibility and the leadership of TLMP in the hands of GES. CRDD was, in fact, the lead organization for developing the new materials, guiding the teacher training program, and for all other aspects of the program. In addition, both the PAC and PTA included a range of senior Ghanaian professionals in education. • Capacity development. CSU had several activities designed to strengthen local capacity, especially in Evaluation of TLMP – TLMP in Ghana 23 CRDD. They held a one-week workshop in Accra, hosted CRDD staff for training at CSU, sponsored educators to attend a CIES conference and, perhaps, most importantly, provided valuable ongoing feedback on the new instructional materials being developed. Negative: • Workbooks The decision to create disposable workbooks was unfortunate as most of the schools, as instructed by TLMP, had the children faithfully fill out the workbooks during the first year and then, at the end of year, sent the workbooks home with the children. Thus, there were no workbooks for the subsequent year and, in a majority of the schools, the teachers were unable to continue with TLMP, only using the teacher’s guide and, perhaps a remaining workbook as reference materials. It was unrealistic to think that either the donors or the Ministry would be able to reprint the guides in subsequent years. And even though this fact should have been obvious when it was decided to distribute the materials nationwide, there was no guidance for the teachers on how they might preserve the life of the workbooks. • Language policy. The impact of TLMP will be limited because of a change in Ministry policy in support of L1 and the NALAP program. Although this change is not the fault of TLMP, perhaps TLMP could have responded earlier to the changing environment. While appreciative of the role that TLMP played and the materials developed, the Ministry no longer supports TLMP as a separate entity or the printing and distribution of the materials. NALAP is the new official program, and TLMP’s last useful initiative is to merge the useful concepts and activities into the NALAP teacher’s guides in order to strengthen the small English component. In addition, TLMP will provide additional support by printing the new teacher’s guides and supporting a large-scale teacher training program at the end of 2013 and beginning of 2014. • P1-P3 materials. Both teacher’s guides and workbooks were developed and piloted, but they were never printed because of a shortage of funds. Now, with the new NALAP program, there are no plans to print these materials, which may waste potentially useful resources, especially in support of the existing mathematics and natural science curricula. • Pedagogy. For all the positive aspects of the TLMP materials, the approach to teaching basic reading skills seem a bit off in in two regards. First, both the teacher’s guides and the workbooks assume an excessively high level of English skills among the children. For example, the pace of vocabulary development is far too fast and the language required to discuss the drawings in the workbook or posters is beyond the ability of the children beginning in KG1. This approach would be much better suited for instruction in L1. Perhaps the program would be useful if translated and adapted to teach L1. The second concern is a clear strategy for teaching basic reading skills. The materials over￾emphasize sight recognition or a “whole-word” approach, and do not give enough emphasis to phonics. • Communication. TLMP fell short in its communication. For example, when the materials were distributed nationally, there did not seem to be any communication with the District Directors or the ECE Coordinators about the materials and how these key officials could help to support the program. The evaluation team met many individuals who had not heard of TLMP itself. A few individuals, such as head teachers, had, in fact, been aware in the past when materials were delivered, but had simply forgotten about TLMP for lack of any further communication. • Universities and Colleges of Education. Although TLMP involved some individuals in creating the materials and carrying out teacher training from the universities and colleges (including MoUs with UCC and UEW), there was no follow-up to help ensure that their pre-service programs incorporated the TLMP program and materials. • In-service teacher training. TLMP limited its teacher training activities to in-service, yet even this was too limited. TLMP trained about 2,000 teachers and also had a planned TOT program whereby district officials would carry out local teachers, but this did not happen. In fact, every one of the teachers interviewed by the evaluation team said that they had not received training in TLMP. Evaluation of TLMP – TLMP in Ghana 24 • M&E. The inadequacy of M&E activities was a major shortcoming. TLMP had no ongoing ability to follow-up on their training initiative, the distribution of materials, any issues related to program implementation, and measurement of the impact of the program on learning. CSU did manage to carry out about four studies, but this could not replace the need for regular monitoring and evaluation activities. In fact, CSU did not even have a PMP, which seems unusual for a USAID cooperative agreement. • Lack of adequate or appropriate staffing. Part of the deficiencies stem from lack of professional expertise on the TLMP team. The program staff was largely administrative and there were no full￾time professionals in ECE, bilingual education, or M&E. 5. LESSONS LEARNED • Need for experienced implementing institutions. A major objective of the USAID/W initiative was to strengthen the capacity of MSIs by giving them exposure to international development and working with USAID. Yet, at least in the initial stages, CSU had difficulties in administration and finance that stemmed, in part, from lack of experience in this area. Even on the programmatic side, a more experienced institution, or with appropriate project staff, might not have decided to have disposable workbooks, or to use an approach to reading that was better suited to L1 than L2, or such a limited M&E capacity. • Full-time technical staff in-country. A project of this magnitude needs at least minimal professional staff, either local or international, in areas such as ECE, bilingual education, reading, and M&E. • M&E support. There is always a need for good M&E support, beginning with a PMP that identifies major indictors of success and guides data collection and analyses, plus a strong monitoring program to assess implementation progress. In addition, especially with a program such as TLMP whose objective is to improve reading skills, there is need for a strong summative evaluation program.. • Sustainability. Consideration of sustainability is important from the very beginning of a project. In the case of TLMP, there was a major error in not recognizing that future funding to reprint workbooks was unlikely. In addition, as soon as NALAP was identified as the new language program focusing on L1, TLMP might have been able to initiate collaborative efforts much sooner for the benefit of both programs. • Need for good communication. A program such as TLMP needs a good communication strategy, especially one that reaches out to key partners. For example, when the materials were distributed nationally, there was need to information District Directors and ECE Coordinators on TLMP – the objectives, materials and the potential role of the districts to support the program. Periodic communication including the solicitation of feedback would be a great help • Include key partners. TLMP focused on in-service teacher training, but neglected the importance pre-service training. It is important to include adequately all relevant institutions, in this case, the universities, colleges of education, and TED. 6. RECOMMENDATIONS We have divided the recommendations into two categories. The first is on improving the materials. The workbooks in their current form are not likely to have any further use in Ghana given the new NALAP program. An exception to this could be in private schools where English language is emphasized or public schools in cosmopolitan areas where English is used because there are too many mother tongues represented in the classes. We also recommend that the TLMP workbooks be made available in the commercial market. In any of these situations, the following recommendations to improve the materials may be useful. The second category is on recommendations to improve the relevant education systems. Evaluation of TLMP – TLMP in Ghana 25 6.1 Materials Improvement • Create new activity books for NALAP based on the TLMP workbooks. The activity books could cover both the mother tongue and English components. The major difference between an activity book and a workbook is that the children would not write in the activity books, which avoids the major problem of annually having to replace workbooks at a cost that is unaffordable to the Ministry. • As mentioned elsewhere in this report, the methodology of the TLMP materials is a better match for teaching introductory reading skills in L1 rather than L2 because the exercises, vocabulary, and grammatical structure in the KG1 materials assume a level of English that is beyond the level of young Ghanaians. The materials and methodology, however, could be applied to the L1 NALAP materials. • The instructional approach to reading should rely much more on phonics rather than word sight recognition. • Before abandoning completely the idea of a workbook, examine some options to preserve the workbooks, while still giving children the opportunity to write in them. One such option might be to use plasticized pages on which the children might write with crayons or chalk, which could be removed at a later time. • There is need for a glossary in English and L1 languages. 6.2 Systems Improvement There are a number of ways in the educational system might be strengthened to support TLMP: • Once it was known that NALAP was to become the official language curriculum for KG and the early primary grades, perhaps USAID, CSU and GES could have begun a process much earlier than the current “merger” initiative to determine the best options for TLMP to support the change. • The MOE could have had greater accountability – greater commitment than just assigning a person to work on the project, be active and provide more financial commitment up front. They should make a financial commitment, not just in kind. • It would have been helpful if the MOU with the MOE had specified sustainable policies relating to ECE and language teaching, the use of L1 and L2, the printing of materials and teacher training. • There was need for a more clear structure for payment of MOE personnel who helped to develop materials or conducted teacher training. • There is need to ensure that all key partners are fully and properly involved such as the universities, colleges of education, and the GES’ Teacher Education Division. • KG teacher training at colleges and universities need to be synchronized and made relevant to the curriculum and materials in schools. • There is need for a full training program at the pre-service as well as in-service level. • The ECE Coordinators need special training and support so that they, in turn, could guide the training and support of the KG teachers in their districts. • The book distribution system, although basically sound with the support of Buck Press, needs fine tuning because some locations in the country were not well served. • There is need for a good M&E system for both formative and summative evaluation tasks. 6.3 Linking TLMP to USAID Priorities in Early Grade Reading Early grade reading is a high priority of USAID as stated in the USAID Education Strategy, 2011 Goal one: “Improved reading skills for 100 million children in primary grades by 2015.” Improved literacy has a modest impact on national economic growth and helps to ensure the retention and success in subsequent grades. The strategy also recognizes the importance of early childhood education: “The Evaluation of TLMP – TLMP in Ghana 26 strategy recognizes the importance of educating children in their native languages where possible in the earliest grades” (USAID Education Strategy, 2011 Goal one, P. 9). TLMP fits well with these USAID priorities, especially early childhood education and the development of basic reading skills. In hindsight, the choice of focusing on English language instruction turned out not to be in line with both changing priorities in Ghana and within USAID – shift towards more emphasis on developing basic skills in the mother tongue. USAID/Ghana, in fact, supported two projects with a mother tongue emphasis, first EQUAL and then NALAP. By the end of TLMP, its legacy will be an improved English language component within NALAP. TLMP also missed out on two other relevant areas regarding reading. While it developed good teacher’s guides, workbooks and posters, it seemed not to have a clear strategy for teaching basic reading skills. Some other USAID programs such as MTPDS in Malawi developed excellent phonics￾based guides for teachers. Also, TLMP lacked a rigorous approach to measuring the impact on learning of its materials and teacher training initiatives. In part, this will now be corrected by the development and implementation of an EGRA program being implemented with Mission funds with the support of RTI. Evaluation of TLMP – TLMP in Ghana 27 ANNEX A. SCOPE OF WORK DESCRIPTION/ RESULTS-ORIENTED STATEMENT OF OBJECTIVES (SOO) Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania I. BACKGROUND TLMP contributed directly to USAID‟s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners‟ management capacity in the education sector. Each Minority-Serving Institution (MSI), based upon the provisions noted in their Cooperative Agreement (CA), was responsible for managing and implementing the TLMP in a specific country and with achieving specific output results. Each MSI was also responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in primary schools in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MoE) and other local specialists. The main objectives of the TLMP were to: 1) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities in primary schools within SSA, 2) strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). TLMP Cooperative Agreement History • TLMP Ethiopia: USAID Cooperative Agreement RLA-A-00-09-00035-00; In coordination with local entities, Alabama Agricultural and Mechanical University (AAMU), over 3 million English for Ethiopia textbooks were produced and disseminated for grades 1, 6, 7, and 8. Over 132 teachers were subsequently trained to use the materials in classroom settings. • TLMP Ghana: USAID Cooperative Agreement RLA-A-00-09-00036-00; In coordination with local entities, Chicago State University (CSU) has trained 260 teachers in using the developed materials. Over 6 million materials and textbooks have been created and distributed for students up to grade 3 in mathematics, environmental science, and English. • TLMP Malawi: USAID Cooperative Agreement RLA-A-00-09-00033-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over five million supplemental reading books, teachers guides and training materials and trained nearly four thousand teachers on methodological classroom usage. • TLMP Senegal: USAID Cooperative Agreement RLA A 00-09-00037-00; In coordination with local entities, Elizabeth City State University (ECSU) produced and distributed over 1.8 million materials in both French and English for grades 2-10 in Evaluation of TLMP – TLMP in Ghana 28 science, mathematics, and language arts. Over 160 teachers were trained on utilizing the materials as part of their curriculum. • TLMP South Africa: USAID Cooperative Agreement RLA-A-00-05-00079-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over 1.4 million materials in 11 languages for grades 4, 5, and 6, as well as trained over 6,000 teachers. The work was completed in 2009. • TLMP Tanzania: USAID Cooperative Agreement RLA-A-00-09-00034; In coordination with local entities, South Carolina State University (SCSU) created and disseminated over 1.1 million materials for secondary level usage in the fields of science and mathematics. Over 1,200 teachers were trained. The Contractor will be provided with each institution’s Cooperative Agreement by each individual institution, which will include the relevant scope of work. The Contractor will be required to obtain other pertinent documents as necessary. II. PURPOSE The purpose of this SOO is to support the Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania. III. SCOPE OR MISSION Task 1 – Data Collection. Task 2 – Data Review. Task 3 – Coordination and Management. Task 4 – Site Visit. Task 5 – Data Analysis. IV. PERFORMANCE OBJECTIVES / DESIRED OUTCOMES The Contractor shall provide all labor, equipment, supplies and materials, and travel necessary to conduct Textbooks and Learning Materials Program Evaluation (PE). The PE is intended to satisfy the following objectives: • validate stated program goals and impacts; • assess the results achieved for each host partner country in relation to intended program targets, as well as standardized and variable indicators by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions, Ministries of Education (MoEs), etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • determine if in-country institutions (with support from U.S.-based Minority-Serving Institutions [MSIs]) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • review allocated USAID funding in terms of usage and overall cost effectiveness; • highlight specific program accomplishments per MSI-host country partnership; and Evaluation of TLMP – TLMP in Ghana 29 • document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the New Agency Education Strategy http://www.usaid.gov/our_work/education_and_universities/documents/USAID_ED_Str ategy_feb2011.pdf 6 V. OPERATING CONSTRAINTS / LIMITATIONS We anticipate that Awardee would complete one site visit per country and that the site visits would take no longer than 10 days each. There is not a requirement for specific key personnel or a combination of key personnel to complete the site visits; however consistency in terms of personnel for the site visits is preferred. The Contractor shall perform the PE in accordance with USAID ADS 203 and the new USAID Evaluation Policy published in January 2011. The USAID ADS 203 Performance and Monitoring Guidance can be found here: http://www.usaid.gov/policy/ads/200/203.pdf. The new USAID Evaluation Policy can be found here: ttp://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf Monthly Status Reports. The Contractor shall provide written reports to the USAID COTR or his/her designee on the progress of the work, contacts made, and problems encountered on a monthly basis. They should be submitted by the last business day of every month. Comment Reponses. Comments will be provided to the Contractor electronically. The Contractor shall prepare comment responses that clearly state the actions taken to incorporate the comment or show the changes in a redline and strikeout version of the revised report. The Contractor may contact the reviewers for clarification. Unresolved technical issues shall be coordinated with the COTR. Evaluation of TLMP – TLMP in Ghana 30 ANNEX B. QUESTIONS POSED IN VARIOUS INTERVIEWS CONDUCTED AT CSU GENERIC QUESTIONS POSED OF ALL MSIs TLMP Program Administrators (MSIs and Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? • Tell me about the Lessons Learned from the implementation of Phase 1? How did this affect your approach and the substance of Phase 2? What changes were made in personnel? Why? • How is phase 2 different from phase 1? • What are the components of your TLMP agreement? How many TLMs in what subject area? How many volumes of each? Do you have any kind of results/outcomes report that identifies all the numbers? • In country, how did you go about assembling your team? Did you have any assistance doing this? From USAID? Other stakeholders? What skills/abilities did each person have? What were their responsibilities? • At your university, how did you go about assembling your team? What skills/abilities did each person have? What were their responsibilities? • How did you monitor your progress? Do you have a PMP? Materials Development Process • What process was used in the creation of these materials? What guidelines did you follow in creating the materials? How did you determine the appropriateness of vocabulary, readability, complexity of structure, etc.? When creating the materials in English, how did you factor in that the students were learning English as a second or third language? How did you insure that the materials conformed to national curriculum standards? • In implementing the project, what role did the MOE play (specify unit)? What roles did your university play? What guidelines did the MOE provide? What level of competency did the MOE/curriculum developers have? What level of expertise did you university provide? What challenges emerged in your work with the MOE? How were they resolved? • What process did you use to review and revise the materials? How were the materias, field tested? Who was involved? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? Are they now able to take on similar work for other projects or for the MOE? What would prevent them from being the designated printer for the continued printing and distribution of these materials? Was the printer just a printer or also a publisher? • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Could I have a copy of the teacher training curriculum with handouts? How were learner-centered teaching methods blended with training on the TLMs? Did you make a DVD of the process? • What other teacher training is provided by the MOU? In what format? How did the TT for TLMP differ from the TT for other areas? Evaluation of TLMP – TLMP in Ghana 31 • In conducting TOTs, were teacher salaries topped off? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? Program Management and Outcomes • How was the university strengthened as a result of Phase 1? As a result of Phase 2? What is the university now able to do that it was not before the program? How has it built the university’s capacity to develop ideas for, submit proposals for, and implement other USAID projects? • What other types of assistance did USAID provide to you, either in the mission or in DC? What was the substance of that assistance? • What kind of networks and/or public-private partnerships did you create in country and/or in the US? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, how have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials, i.e., how did the program improve the national publishing/printing industry? • What types of policy changes have you observed as a result of your work? Have any new policies been created in admission of children to school (Ghana – kindergarten)? About distributing books vs. keeping them locked in cupboards? About class size? About early literacy development? • What role will the MOE play in extending the whole idea of TLMP? Has anyone been appointed to conduct follow-up activities? • What other funding/projects have you leveraged to continue the work of TLMP or to expand it in other directions? • What accomplishments are you most proud of? What are the outstanding features of TLMP for you, your team, and your institution? Logistics • Which officials in country shall we interview? What are their contacts? • How shall we work with your in-country team? • From your point of view, which would be two of the “best” schools and two of the “worst” schools? What are your criteria? Where are these schools located? What are their contacts? SPECIFIC QUESTIONS POSED OF TLMP LEADERSHIP TEAM AT CSU 1. National Curriculum • What is the national curriculum for K1? • Get copy of K2 materials (only have K1) (K2 includes: auditory discrimination, auditory comprehension, conversation, storytelling, pre-reading activities such as: sorting, writing, left to right eye movement, visual discrimination/memory skills, visual comprehension, understanding positions, reading pictures, phonological awareness, introduction to the letters of the alphabet, associating lower and upper case letters, matching objects with letters, letter sounds, formation of two-three letter words/sounds and rewriting activities) • What changes are being made in the national curriculum for English, math and environmental studies (NALAP addendum, p. 9) • What is the current language policy in education in Ghana? • What happened to the P1-3 materials? • What impact has the project had on the national curriculum? On other policies related to KG? Evaluation of TLMP – TLMP in Ghana 32 2. Research Reports • Get March 2012 study of three schools on achievement • Have you videotaped any of the teachers delivering lessons in each of the three areas using the TLMs? • Get field test reports for P1-3 materials 3. Book Printing/Distribution • What happened to the printing equipment that was purchased originally and given to the GES? • How was the distribution of books monitored? Who was ultimately responsible for distribution? • From your studies, how many workbooks were delivered to each district/school? Does every KG child have a workbook? How do teachers monitor the work done by the children? 4. Merger with NALAP • Get original proposal; transfer of responsibility from USAID/W to USAID/Ghana – who will be your COTR? • How different is the TOT model to be implemented in the merger vs. in TLMP? What lessons were learned? • Pages missing from 8/22/12 document • How will the bilingual/L1-L2 transition affect the TLMP materials in all three areas? What did you learn in how these materials need to be adapted to a bilingual environment rather than just English? • What will the review/editing/field testing process look like? How will it be different from the TLMP process? What lessons have been learned? • What lessons were learned in the earlier phases of TLMP that will inform how you do things under the merger? • What team will you need to assemble for the merged activities? 5. Teacher Training • In choosing the master teachers, how did they cope with English? • How was this monitored after the TOT? • How many teachers did each TOT trainee train? • Was the whole country ever covered? What is the plan for including all districts? • How did TLMP TT plan fit in MOE/GES overall teacher training activities, both pre-service and in-service? • Are TLMP materials now included in pre-service TT? • What were the evaluation results of the cascaded model? • How did teachers measure improvements in their students after teachers were trained and students used the workbooks and materials? 6. Book Production • In the earliest phase of the TLMP project, certain already-existing titles were reproduced. How many of these books/titles were reproduced vs. actual number of materials produced, printed and distributed? • How were the books field tested? • The original intention of the project was to create materials in the mother tongue. What challenges did you face when the budget was cut and the materials were not translated? Evaluation of TLMP – TLMP in Ghana 33 7. Team • How did you become involved in TLMP? What were your roles and responsibilities? How have you seen the project change over time? What were its strengths? Its weaknesses? What were the lessons learned? • Who are the CSU team members now? What are their skills/abilities/subject matter specializations? Who can comment further on TLMP up to now? • Who are the Ghanaian team members now? What are their skills/abilities/subject matter specializations? Who can comment further on the TLMP up to now? • Names and contact information in Ghana of all partners. Evaluation of TLMP – TLMP in Ghana 34 ANNEX C. DATA COLLECTION INSTRUMENTS, GHANA STAKEHOLDERS In Accra • What has been the mission’s role in implementing TLMP? What types of support activities have you provided to the project? • How does TLMP fit in with other USAID education program/priorities in this country? How do you see the experience of TLMP leveraging or leading to Chicago State Univ. working on other educational development activities in Ghana or elsewhere in the future? • Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbook and learning material development? • How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable? • What specific challenges has TLMP faced in this country? How were they addressed? • How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outputs and outcomes of TLMP? What would you change? • Would you say that the model for TLM creation developed by Chicago State Univ. (working with teachers to write the materials) can be utilized in other areas? Or was this a process you feel can be used for only supplementary or complementary materials? How would it need to be improved to be useful in other contexts? • How satisfied were you with your relationships with Chicago State Univ. and the work they did? What suggestions do you have for overall improvement? • What were the lessons learned for the mission in overseeing the TLMP? Would the mission support a similar program in the future? 2) MOE Administrators • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? During what period? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How did you decide which staff members/departments were to work on the TLMP? Were they seconded to the program or were program responsibilities added to their normal tasks? Were any incentives provided for participation? What? • How was it decided which schools would receive the TLMs? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training? • What types of policy change has the MOEST/GES instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training? • What other types of teacher training does the MOEST/GES provide? How frequently? • How do PEAs assess teachers? How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLMs included in teacher training (either pre-service or INSET)? • What will the MOEST/GES do to continue the production of TLMs now that the program has ended? What new textbook policies have been developed as a result of TLMP? • Is the MOE ready to use its own funding in the creation of TLMs in the future? What would prevent this from happening? • How has the material presented in the TLMs been included in national exams? Has performance on exams improved with the use of TLMs? • How was TLMP monitored by the MOEST/GES? What indicators did you use? How often did Evaluation of TLMP – TLMP in Ghana 35 you go to schools to observe the use of TLMs? How was TLM production managed and monitored? • How has the MOEST/GES benefited from TLMP? How has it been challenged? Were there any negative effects of TLMP? If so, what were they and how were they addressed? • How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? (Why?) If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different? • How satisfied are you with the collaborative relationships established with Chicago State Univ.? What could be improved? 4) TLMP Program Administrators Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? Did you have any assistance doing this? From USAID? Other stakeholders? Materials Development and Distribution Process • What process was used in the creation of TLMs? What did the MOEST/GES do (specify unit)? What did the program do? What challenges emerged in your work with the MOEST/GES? How were they resolved? How did you liaise with all stakeholders? • In implementing the project, what role did the MOEST/GES play (specify unit)? What roles did your office play? What guidelines did the MOEST/GES provide? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? What work are they now able to take on with other clients? What other services did the printer provide? • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? • What other teacher training is provided by the MOEST/GES? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries supplemented? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? • How successful was the cascade training model? How many teachers did those who participated in the TOT actually train on the use of the TLMs? Program Management and Outcomes • What was the TLMP management structure in Ghana? What types of services did you provide to the MSI and other stakeholders? • What was the composition of the Ghana PAC? How often did they meet? What decisions did they make? How did these decisions affect the project? • How often did you visit program implementation sites? What types of monitoring did you perform? How frequently? • What types of assistance did USAID provide to you? • What kind of networks and/or public-private partnerships did you create? How are you collaborating with other stakeholders? Evaluation of TLMP – TLMP in Ghana 36 • From your point of view, as a result of the TLMP project, what has changed either positively or negatively? How have teachers changed? Headteachers? District/Provincial administrators? PEAs? The MOEST/GES itself? The printers and distributors of the materials? • What types of policy changes, if any, have you observed as a result of your work? • What accomplishments are you most proud of? If a TLMP-type program were to be undertaken again, what would you do differently? 5) Printers/Publishers • Tell me about your operations before you were granted the TLMP contract and how they changed (either positively or negatively) as a result of TLMP participation? • How did the contracting occur with Chicago State Univ.? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these? • What is your opinion of the management training that was provided to you by Chicago State Univ.? What key lessons did you learn that you applied to the management of your operations? • What instructions were you given on how to distribute the TLMs? From whom? What kind of difficulties did you encounter in keeping to the distribution schedule? • When/how did you distribute the TLMs after they were produced? To whom did you distribute them? How many TLMs were delivered to each receiver? What kind of tracking/delivery system did you establish? What kind of challenges did you have in distributing the materials? How were these overcome? • How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have your hired? What new equipment have you purchased? What other inputs would you require to take on more textbook production projects? In Field 6) District Directors • What is your current position? How long have you been in this position? • Have you been involved in TLMP and, if so, what was your particular involvement? • How many schools with KG classes, total KG classes, and KG teachers do you have? • What percent of your KG teachers are trained? • When did you receive the TLMP materials? Do you know how many materials arrived by grade level and subject? What percent of your KG classes receive these materials? • What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received? What recommendations do you have to improve the delivery service? • How many of your KG teachers received the initial TOT training on TLMP? • How many of these teachers subsequently taught other teachers? How many? • Has the district given any additional support or training to the KG teachers on using TLMP? • How many of our KG classes initially participated in TLMP? • How many of your KG classes are currently using the TLMP materials on a regular basis. If the number of teachers actively using TLMP has decreased, why was this so? • From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank TLMP in terms of helping KG children learn English including basic reading skills? What do you recommend to improve the program? In Schools: 7) Head Teachers • What is your current position? How long have you been a principal at this school? Evaluation of TLMP – TLMP in Ghana 37 • What are the grades in your school? How many boys and girls? • How many KG classes and teachers do you have? What is the enrollment in each KG class? • Are you familiar with TLMP and, if so, is it being used in your school on a regular basis? • How many set of TLMP materials did your school receive? Were these sufficient? • What has happened to the workbooks given that children are supposed to write in them and they have not been replaced? • Have the teachers tried to extend the life of these workbooks and, if so, what have they done? • Are some teachers no longer using TLMP because there are no workbooks? • Is TLMP being used more, less, or the same as compared to three years ago? If there has been a change in the use of TLMP, why is this so? • How many of your KG teachers were trained in TLMP? Were they trained in a TOT or did other teachers provide the training? Are they still at the school, or have some departed or shifted to another grade level? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? 8) Classroom Teachers • How long have you been a teacher? How long at this school teaching KG? • How would you classify your ability to interact in and teach in English? (excellent, very good, good, somewhat deficient, poor) • How many students do you have in your classes? Specify class and number of students. • Have you been trained in TLMP? If so, who did the training and how long was the training? How would you rate the teaching? • If you attended the TLMP TOT training, how many other teachers did you teach afterward? What worked well? What difficulties did you encounter in doing this? • If you did not attend any training related to the materials, how did you learn how to use them? • Do you have the TLMP KG Teacher’s Guide? • How many TLMP workbooks do you have? How many children are in your class? • If you don’t have enough workbooks for all the children, what do you do? • Are these workbooks already filled in by children in previous years? If so, how do the children use these workbooks? • Have you stopped teaching TLMP because you don’t have workbooks? • For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLMs materials? • What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not? • How “ready” were your students to use the materials distributed? Was the vocabulary at a level that could be understood by students? What type of difficulties do the students have in using the materials? How should the program overcome these difficulties? • What changes have you observed and recorded in children’s achievement in English since the TLMP workbooks/materials were introduced? • In using the TLMP workbooks/materials, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice. • What is the greatest challenge your students experience in using the TLMs? • What do you think is the overall impact of the program on your students? What kind of Evaluation of TLMP – TLMP in Ghana 38 difference does it make in learning for a child to have textbooks/workbooks? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the delivery of the program? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the delivery of the program? Evaluation of TLMP – TLMP in Ghana 39 ANNEX D. SCHOOL OF INTERVIEWS AND MEETINGS DATE TIME ORGANIZATION PERSON TITLE ACCRA 10 June 8:00 TLMP Mrs. Ewura-Abena Ahwoi In-Country Coordinator 10:00 CRDD/GES Mr. Jacob Kor Director, CRDD 12:00 GES Mrs. M. Okae National ECD Coordinator 3:00 PM GES Mr. C. Dowuona￾Hammond Liaison Officer June 11 9:30 USAID M. Perez, A. Jehando, J. Petro Education Office Chief, Education Management Specialist, Education Advisor 12:00 TLMP Mrs. F. Boakye-Yiadom Head, Merger Implementation Team June 12 Aburi College of Education Mrs. Beatrice Boateng￾Sampong, Vice Principal Aburi College of Education Mr. Baffoe Bonmie, Teaching Practice Coordinator New Juaben Municipality Mrs. Docia Abbah Director Mrs. Victoria Obeng Deputy Director for Basic Schools Mr. Peter A Odei Deputy Director for Supervision and Monitoring Densuano M/A Basic School Mr. Bosoompem George Bismark Head teacher St. Mary’s Day Care and Preparatory School, Koforidua Ms. Mirium Ofosu Appiah Proprietress RTI Mr. Paa Nii Bossman M&E Officer?? DRCC/GES Mr. Isaac Asiegbor Director, Assessment Services Unit UNICEF Mr. Madeez Adamu Issah Education Specialist June 13 8:30 Accra Metro Education Directorate Ms. Grace Dankyira ECE Coordinator 9:30 Osu Presbyterian KG, Ga East Lydia Offei Kwapong Head teacher 11:30 Osu Presbyterian Annex Gladys Ayiku Head teacher Evaluation of TLMP – TLMP in Ghana 40 KG 3:00 PM CRDD (former) Mrs. Sarah Agyeman￾Duah Retired former Director 4:00 PM GES Mr. Charles Aheto Tsegah Deputy General Director June 14 9:00 Ga East Municipal Education Directorate Ms. Florence Damahli Director 9:30 Pantang Presbyterian Basic School Mr. Samuel Nkotia￾Dardom Head teacher 11:00 Living Star School (private) Mrs. Mary Ameyaw Proprietress Mr. Joseph Obeng￾Yeboah Assistant Head Teacher 2:30 PM Buck Press Mr. Kofi Buckman Managing Director Mr. Tony Boamah Director of General Operations Mr. James Ottumani Marketing Director June 17 9:30 Jasikan College of Education, Volta Mr. Ansi Samuel Vice Principal Mr. Torkonyo Moses Martey ECE Coordinator Mr. Agbenor Akpeteasi Head of Languages Mr. Nyavor Kunche Nyavor Sports and Health Tutor 11:30 Jasikan College of Education Demo School Mr. Feda Robert Kwasi Head teacher 11:00 Jasikan District Education Office Mr. Dompreh J.K. Daniel Deputy Director (HRMD) Ms. Kikiri Victoria ECE Coordinator Mr. Yao Augustine Wuabu Storekeeper 9:30 Metro Cape Coast Education Office Mr. Felix Ansah Director following Supervision 10:00 Philip Quaque Girls’ School, Cape Coast Ms. Olivia Appian Head teacher 11:00 Abakam CRAN school, Cape Coast Teachers Evaluation of TLMP – TLMP in Ghana 41 11:30 Golden Treasure International School (private) Ms. Rita Dodoo ECE Coordinator 12:00 OLA College of Education Rev. Elizabeth Amoako Principal 1:00 PM Metro Cape Coast Education Office Ms. Vivian Etroo Director 2:30 PM University of Cape Coast Dr. Kafui Etsey (Mrs) Senior Lecturer Dr. Clement Agezo Head, Department of Basic Education June 18 8:30 Ho District Education Office Mrs. Dora Agorsor Director Mrs. Beauty Agbeve Coordinator ECE 10:00 Mawuli E.P. Primary School, Ho Mrs. Ocloo Mawuse Head teacher 12:00 Kpenoe E.P. Primary School, Ho Mr. Appiah Samuel Head teacher 9:00 Effutu Municipality Education Directorate (Winneba) Mrs. Hilda Eghan Director Mrs. Thelma Essuman ECE Coordinator 10:30 Effutu Municipal KG and Primary School Ms. Ruth Duncan￾William Head teacher 11:00 ACM KG and Primary School Teacher 12:30 Ebenezer Memorial Education Centre (private) Mrs. Grace Anakyewa Ehintoh Proprietress Mrs. Agnes Appeadu Head teacher 2:30 PM University of Education, Winneba Mr. W. K. Agbeke Senior Lecturer, Department of ECE June 19 9:30 GES Mr. Stephan Adu Deputy Director General, Basic Education 11:00 TLMP Mrs. Ewura-Abena Ahwoi In-Country Coordinator 1:00 PM North Legon Little Campus (private) Ms. Nicholina Bannerman Director, Administration Mr. Douglas Djarbeng Managing Director Mr. Simon Peter Attah￾Cats Director, Academic June 20 10:30 Chicago State University Ms. Jamille Watkins￾Barnes Assistant Director, TLMP Evaluation of TLMP – TLMP in Ghana 42 4:00 PM USAID Ms. Cheryl Anderson Ms Marisol Perez Ms Adama Jehanfo, J. Petro Mission Director, Education Office Chief, Education Management Specialist, Education Advisor Evaluation of TLMP – TLMP in Ghana 43 ANNEX E. READING ASSESSMENT FORM AND EVALUATION OF LEARNER READING COMPETENCY Following is the protocol that was to be used for schools where assessments of students were to be made. 1. Class selection - K2 only 1.1. Interview all teachers together 1.2. For class observation, ensure teacher has been trained and materials available 1.3. Then select section A, or A and B. Try to get 1 male and 1 female teacher 2. Child selection 2.1. 2 boys and 2 girls 2.2. Select at random. Do NOT let teachers select the students. Perhaps count children to 15 and then 30 for adjacent boy and girl 3. Test administration 3.1. By our team only. (Probably without the teacher) 3.2. Set and adhere to time limit 3.3. Fill out data sheet for each child 3.4. Describe to student what is to be done 3.4.1. Learn about English skills 3.4.2. Not interested in any individual child, but want to find out in general abilities in English 4. Conduct a modified EGRA test, assessing: 4.1. Letter recognition 4.2. Word recognition 4.3. Ability to read a simple sentence In practice, it proved necessary to adjust this protocol. After the teachers were interviewed, the consultant explained the learner literacy skills test to be administered to KG1 and KG2 students. The learner reading competency test was revised to accommodate conditions found in the three schools visited in the Volta Region, as described in the text. After an interview with the teachers and an explanation of the literacy skills test, the researcher selected four KG1 and four KG2 students to be tested. In each grade-level group there were four boys and four girls. Because someone had to explain to the children in a Ghanaian language what the testing procedure was going to be, and to explain steps as the test progressed (especially in KG1), a teacher had to be present during the testing. The testing was done in the corner of the classroom so that the wall charts would be visible. The skills tested were the following: 1. Listening and responding The researcher asked each child (KG1 and KG2) to answer these questions: “What is your name?”; “How old are you?”; “What is this?” (pointing to the table where the testing took place). 2. Letter identification Each child was asked to identify these letters: M, b, r, P, A, g, F, T, Z s. 3. Oral fluency in describing a picture A) A picture was chosen from the workbook and the child was asked to describe it (ex. The boy is sitting on a chair). Evaluation of TLMP – TLMP in Ghana 44 B) Pictures were pointed to on the wall charts and the student was asked to name them. 4. Word recognition Pictures and the associated words were chosen by the researcher for the student to read from the workbook. Some were chosen in the sequence in which they appeared on the page, others at random. 5. Writing The researcher asked the student (with the help of a teacher who explained the task) to do the following: 1. Write three letters (either capital or lower case, as the child preferred). 2. Write their first name. 3. Write a word of personal choice in English. Evaluation of TLMP – TLMP in Ghana 45 ANNEX F. PRINTING AND DISTRIBUTION – A CASE STUDY Per IBTCI’s January 2013 consultations with the Africa Bureau’s Education Team, review of the physical production of TLMP materials was to be only of secondary concern for the evaluation and, therefore, IBTCI has developed and presented only the most relevant information on this aspect of TLMP for the other CAs and MSIs. However, since each of the MSIs was required to produce a minimum of 600,000 copies of materials and procurement of this amount of work presented challenges novel both to the MSIs and often to the printers and since CSU was, to the best of our knowledge, the only MSI actually to procure significant equipment, we feel that the example of production of TLMP materials in Ghana would provide a useful case study of potential value both to Missions and to implementers who may find themselves with similar needs. According to Buck Press Ltd, they responded to invitation to Bid (Bid No. 08211S), published Friday March 28 and Saturday March 29, 2008 in the Daily Graphic and Ghanaian Times. It competitively won the Tender as per Notification of Award of Contract dated April 24, 2008 and signed by the CSU Director of Purchases. Buck Press was identified as the primary printing contractor as they were willing to produce, store and transport the books nationally for distribution to the schools (storage was BP’s in-kind contribution). Transporting them was to have been covered by GES, but due to lack of funds/misunderstandings, school districts picked up the TLMs themselves. According to Mr. Buckman, the following issues emerged during the contract period: • Variations (increases) in pages of some of the titles. • Delays in submitting artwork due to corrections in the scripts. As such, Buck Press, Ltd had to work under pressure to deliver within limited time frame. • Storage of finished titles for very long time. • Breaking the materials by Districts involved extra packaging. • Buck Press advanced the cost for distributing TLMs to the Districts before being reimbursed. The evaluation team in its visit to Buck Press noted that it has grown to become a full-service printing (not publishing) company that provides good quality printing services. The TLMP contract made a significant difference to Buck Press, which was able to purchase new equipment and upgrade its storage capacities. The project did not provide technical or managerial capacity building services for Buck Press. During the visit, the team observed the production of a mathematics textbook for 7th grade. The glossy paper, the print, the binding and the illustrations were of high quality. Buck Press is currently producing eight other textbooks for a different client. In addition, the company has also undertaken large employment schemes to facilitate the provision of secure identification card services in Ghana, Liberia and beyond. It also ensures that its waste paper is recycled. For TLMP, it was able to print 1,458,006 items in three weeks. The value of the contract was $930,525.50 USD. The project paid the Buck Press invoice two months after delivering TLMs on environmental studies, numeracy and literacy to the regional education offices’ store houses. The number of TLMs dispatched to each regional education office was determined by the number of students and teachers listed in the MOE’s statistical yearbook for KG1 and KG2. In 2010 and 2011 Buck Press printed 3,881,353 teachers’ guides and student workbooks at a cost of $4,915,000, including distribution costs. Mr. Buckman explained that printing in black and white would have been about 20 per cent cheaper than full color printing (the option chosen by the project). Buck Press explained that the Chicago State/USAID contract had been challenging. The finished products were to have been delivered to the MOE warehouse in Tema and the Ministry was to have ensured forwarding to its district offices. However, when the production of the first run of TLMs was completed in September 2008, the Tema warehouse was full. Buck Press then delivered the books to the Central Region and Accra at no extra charge. This was a gesture of “corporate social responsibility” valued at Evaluation of TLMP – TLMP in Ghana 46 $150,000. Buck Press distributed the TLMs to District Stores (depots) serving 170 districts in the quantities determined by the Ministry and the CSU project, and the district store managers signed bills of lading. Buck Press confirmed that it had not delivered any TLMs to teacher training colleges or universities with teacher training departments. It should be noted that early in TLMP CSU made the mistake of buying its own print equipment. There are two offset printers in the TLMP office, neither of which have worked for some time. However, with new additional funding, TLMP plans to repair the printers and to use them to print the newly revised NALAP teacher’s guides for national distribution. The background of the printers is not totally clear. It appears that they were purchased with the expectation, initially, that GES would produce all the TLMs. However, the equipment was never used by GES as the contract was then developed with Buck Press. And the former Director of CRDD says that the printers “did not satisfy the specifications of CRDD.” The printing equipment was initially placed in CRDD, but then moved to TLMP with the idea that it could be used to print materials for piloting of materials. A comment from one senior officer was that there has been substantial maintenance costs and that it might have been more cost effective to have the printing done commercially. For the more limited runs, perhaps it would have been more cost-effective to have purchased a copy-printer or just a high capacity photocopying machine. Evaluation of TLMP – TLMP in Ghana 47 ANNEX G. REVIEW OF TLMP PUPIL’S WORKBOOKS AND TEACHER’S GUIDES SUMMARY OF MATERIALS EVALUATION CRITERIA A. Pedagogical Quality 1. Conformity to the syllabus 2. Organization and presentation of content 3. Correctness of information 4. Conformity to socio-cultural values in Ghana 5. Integration of skills and cross cutting skills 6. Adaptability and applicability to local and global settings 7. Relevance to the learner’s age and interests 8. Use of quality illustration 9. Appropriateness for learner and teacher 10. Content appropriate for learners of diverse abilities 11. Presence of enjoyment index 12. Assessment of student learning 13. Language and communication aspects B. Physical and Design Quality 1. Type and quality of paper 2. Type and quality of binding 3. Size and dimension 4. Quality of print 5. Type face and size of sentence length 6. Structure Each of the characteristics above is further divided into success stories-categories. PUPIL’S WORKBOOK – KG1 LITERACY A. Pedagogical Quality No. Criteria Remarks 1 Conformity to the syllabus • Extent to which the text covers the entire syllabus • Depth and breadth of concepts • Arrangement of the material in respect to the pedagogical needs of the learner • Conformity to the general and specific goals of the subject area 2 Organization and presentation of content • Sequencing of content in relation to sound pedagogical approach (simple to complex, known to unknown) • Use of a variety of presentation techniques • Use of illustrations, italics, bolds, subheads, paragraphing Each lesson includes a selection of the following: talk about the picture, sorting, coloring/drawing/ matching, tracing letters, visual discrimination, same/different, pattern making, what is missing,. Emphasis is on vocabulary building, although there are no strategies to Evaluation of TLMP – TLMP in Ghana 48 use the vocabulary except once during each lesson when a child must describe a picture. In general, the children are depicted as looking sad – largely the result of drawing the mouths of people illustrated. 3 Correctness of information • Correctness of facts and concepts • Accuracy of figures • Free from spelling and grammatical errors Good 4 Conformity to socio-cultural values of Ghana • Content abides with the moral values of Ghana • Content shows tolerance to diverse beliefs • Content reflects love and respect for the country The back of a nude male is presented on p. 34 as he is washing. Boys do not jump rope as depicted on pp. 32 and 43. 5 Integration of skills and cross cutting skills • Content helps learners build positive life skills • Content informs learners on HIV/AIDS, gender, OVC, children’s rights, safety, environmental issues The 24 units (plus one for revision) cover the “normal” topics – myself, my family, body, food, colors, home, shops, school, friends, animals, clothes, safety, and health. The topics could lend themselves to further exploration, but at the KG level, this could be inappropriate. 6 Adaptability and applicability to local and global settings • Content informs learners on global technological, economic and political issues • Content informs learners of current issues • Extent to which knowledge in the text can be applied by learners in their locality Content is local and is designed to have children acquire the vocabulary of different locales and people’s activities. 7 Relevance to the learner’s age and interests • Activities in the text relate to the learner’s age, needs, and interests • Illustrations and pictures relate to the level of the learner’s development and interests All relevant. However, almost all lessons require coloring or drawing and a student must have access to crayons or colored pencils to complete each lesson. 8 Use of quality illustrations • Meaningfulness and relevance • Clarity • Location and balance in relation to text • Effective use of color • Variety of illustrations • Presence of captions • Correctness of illustrations All appropriate 9 Appropriateness for learner and teacher • Extent to which learner/teacher are likely to enjoy the material • Extent to which content guides the learner to the intended knowledge Pictures and activities can be expanded in any way the teacher decides. 10 Content appropriate for learners of diverse abilities Learners can proceed in accordance Evaluation of TLMP – TLMP in Ghana 49 • Content considers both slow and fast learners to the teacher’s or their own pace. 11 Presence of enjoyment index • Content integrates elements of enjoyment (humor, jokes, cartoons songs, puzzles) • Use of different techniques to stimulate learners’ interests There is a considerable variety of activities, although no songs/games are considered. 12 Assessment of student understanding • Use of feedback, summaries, diagnostics, practice and review activities • Adequacy of assessment activities to test different levels of cognition • Variety of assessment techniques employed None is present in the workbook, although there are three revision units in which the same type of activities as the other units are included/repeated. 13 Language and communication aspects • Extent to which the level of difficulty of language reflects the learners’ level • Extent to which concepts are exact • Correctness of spelling and grammar • Extent to which the material stimulates learners’ language usage (catchwords, proverbs, slogans) Emphasis is on vocabulary and learning how to sight read (rather than sound out letters/phonics) with names of items appearing under a picture of the item. Reading is only at the picture/word level and does not address any skill building in sound/ symbol correlation. Talking about each picture at the outset of each lesson requires a certain amount of vocabulary; it is not clear if this is in English or home language. B. Physical and Design Quality No. Criteria Remarks 1 Type and quality of paper • Water resistant cover • Opacity of paper & weight Size of book is A4 with a sturdy cover, although it may not be water resistant. 2 Type and quality of binding • Type of binding in relation to pages • Strength of binding wire, thread or gum Binding is just two staples and does not appear sturdy. 3 Size and dimension • Size and dimension in relation to learners’ age A4 – may be too big for KG. 4 Quality of print • Legibility of ink (text and illustrations) Very good 5 Type face and size and sentence length • Typeface and size in relation to learners’ level • Lengths of sentences in relation to learners’ level Size letters and illustrations are good. No sentences are used except to give directions (which would probably be given orally by the teacher). 6 Structure • Balance of chapters • Endmatter (glossaries, indices, appendices) Unit balance; all almost the same. No end matter. Workbook is 91 pages and it is not clear how the lessons progress; if the Evaluation of TLMP – TLMP in Ghana 50 workbook is to be used over a year, it may not withstand rough treatment. TEACHER’S GUIDE – KG1 LITERACY A. Pedagogical Quality No. Criteria Remarks 1 Objectives • Objectives stated in the guide match the syllabus • Clarity of objectives ( realistic & measurable) Emphasis on listening and speaking. Outcome objectives are clearly set forth at the outset of each lesson. 2 Organization and presentation • Sequencing of topics in relation to pupils’ book TG includes the materials presented in the children’s workbook, identifying and explaining different activities to be undertaken to teach the lesson. It is not clear what level of English teachers have achieved; understanding the TG requires a significant amount of English as well as an understanding of early childhood education, learner￾centered methods, and how children learn using different strategies, 3 Level of cognition • Activities measure the level of understanding (knowledge, comprehension, application, analysis, synthesis, and evaluation) The TG takes the teacher through a series of age-specific activities for children to introduce them to English and give them language experience. Discussion of the pictures requires children to use English language constructions that have not really been introduced – the teacher says something, and the children parrot: e.g., I am walking (and illustrates while walking). Use of the total physical response method is good, but more practice is needed in using this construction and others. 4 Teaching and learning strategies • Teaching and learning strategies enhance the teaching-learning process • Strategies help the teacher attain the stated objectives • Strategies give extra information compared to the pupils’ book Many different types of activity are provided, along with how to implement them to achieve learning objectives. A final part in each unit includes suggestions for parents that teachers can make in parent conferences. Writing is introduced through making scribbles and coloring so that children can become comfortable in holding writing implements. Teaching songs is a good way to help children learn vocabulary and structure. All of the songs and poems Evaluation of TLMP – TLMP in Ghana 51 are in the Appendix. Most are from the US. However, it takes time to teach and to learn these and it is not clear how much time each lesson should take. No time allocations are made to teach each unit and each element in each unit. A significant number of materials to manipulate are required to teach the units and it is not clear where a teach would get these implements, e.g., bottles, rattles, drums, etc. Assessment is largely “be mindful of benchmarks” without actually noting who has mastered what. Teachers are to gather books recommended in the appendix for “read aloud” activities, with two excerpts from the “Ananse” stories given; it is not clear how to systematically use these materials to enhance language development. 5 Teaching ideas • Guide provides new ideas for enhancing classroom instruction Lots of ideas are present, but no time frame is given in teaching them. 6 Referencing • Extent to which the guide helps identify other materials and resources Many in the appendix, but no idea how teachers can access these. 7 Extra questions/examples/Answers • Extent to which the guide gives extra questions and examples apart from those in the pupils’ book • Extent to which the guide provides correct answers to questions in pupils’ book The student workbook is a practice book in which students practice a limited set of skills, with the teacher providing a great deal more in the classroom. There are no questions in the students’ workbooks. B. Physical Design and Quality No. Criteria Remarks 1 Type and quality of cover/paper • Attractiveness of cover • Water resistant cover • Opacity of paper • Weight of paper Same cover and paper as children’s workbooks; not sure if cover is water resistant. 2 Type and quality of binding • Type of binding in relation to number of pages • Strength of wire, thread or gum Binding is more substantial gum, but in humid climates may not be durable. Evaluation of TLMP – TLMP in Ghana 52 3 Size and dimension • Size and dimension of the book A4 4 Quality of print • Legibility Very good 5 Typeface and size • Typeface and size are easy to read God use of headings and separation; dense, but separations make it easy to read. 6 Structure • Forematter (rationale, theoretical framework) • End matter (appendices, glossaries, indices) The TG is 243 pages long, includes the student workbook (divided so as to show where expanded lessons should be taught), but does not give instructions on letter recognition even where children are asked to “read” them based on their associated pictures. PUPIL’S WORKBOOK – KG1 NUMERACY A. Pedagogical Quality No. Criteria Remarks 1 Conformity to the syllabus • Extent to which the text covers the entire syllabus • Depth and breadth of concepts • Arrangement of the material in respect to the pedagogical needs of the learner • Conformity to the general and specific goals of the subject area 2 Organization and presentation of content • Sequencing of content in relation to sound pedagogical approach (simple to complex, known to unknown) • Use of a variety of presentation techniques • Use of illustrations, italics, bolds, subheads, paragraphing The workbook begins with classification/grouping rather than with actual numbers (which begins only on p. 19 – Unit 2). Groupings are by color, size, shape, matching and ordering, all of which may not be appropriate if the numeracy skills are to be supported by literacy skills. The illustrations of people are much better in the numeracy book than the literacy book; the people are smiling. Beginning with Unit 2, children practice tracing and writing the numbers after they count items in pictures. Pictures are of items common in the environment. Unit 3 introduces shapes and solids, although some of this was in Unit 1. Evaluation of TLMP – TLMP in Ghana 53 3 Correctness of information • Correctness of facts and concepts • Accuracy of figures • Free from spelling and grammatical errors All good 4 Conformity to socio-cultural values of Ghana • Content abides with the moral values of Ghana • Content shows tolerance to diverse beliefs • Content reflects love and respect for the country Yes, pictures are of things common in Ghanaian culture; the other items n/a 5 Integration of skills and cross cutting skills • Content helps learners build positive life skills • Content informs learners on HIV/AIDS, gender, OVC, children’s rights, safety, environmental issues N/A 6 Adaptability and applicability to local and global settings • Content informs learners on global technological, economic and political issues • Content informs learners of current issues • Extent to which knowledge in the text can be applied by learners in their locality N/A 7 Relevance to the learner’s age and interests • Activities in the text relate to the learner’s age, needs, and interests • Illustrations and pictures relate to the level of the learner’s development and interests Learners are asked to count items 1-10, to trace and write numbers, and to distinguish between shapes. Illustrations all support these activities. 8 Use of quality illustrations • Meaningfulness and relevance • Clarity • Location and balance in relation to text • Effective use of color • Variety of illustrations • Presence of captions • Correctness of illustrations Good. Milo can was at first indistinguishable. 9 Appropriateness for learner and teacher • Extent to which learner/teacher are likely to enjoy the material • Extent to which content guides the learner to the intended knowledge Good progression 10 Content appropriate for learners of diverse abilities • Content considers both slow and fast learners Workbook pages can be done as assigned or as the children pace themselves. 11 Presence of enjoyment index • Content integrates elements of enjoyment (humor, jokes, cartoons songs, puzzles) • Use of different techniques to stimulate learners’ interests Nothing presented in the workbook. 12 Assessment of student understanding • Use of feedback, summaries, diagnostics, practice and review activities Nothing presented in the workbook. Evaluation of TLMP – TLMP in Ghana 54 • Adequacy of assessment activities to test different levels of cognition • Variety of assessment techniques employed 13 Language and communication aspects • Extent to which the level of difficulty of language reflects the learners’ level • Extent to which concepts are exact • Correctness of spelling and grammar • Extent to which the material stimulates learners’ language usage (catchwords, proverbs, slogans) Starting with classification was inappropriate; should have started outright with numbers. B. Physical and Design Quality No. Criteria Remarks 1 Type and quality of paper • Water resistant cover • Opacity of paper & weight Cover paper is of a card stock that is not water resistant. Weight of paper of book similar to all books produced and may have a limited life. 2 Type and quality of binding • Type of binding in relation to pages • Strength of binding wire, thread or gum For 38 pages of A4 paper, only two staples bind the pages, so book may have limited life. 3 Size and dimension • Size and dimension in relation to learners’ age A4 may be cumbersome for KG 4 Quality of print • Legibility of ink (text and illustrations) Good 5 Type face and size and sentence length • Typeface and size in relation to learners’ level • Lengths of sentences in relation to learners’ level Only sentences are instructions, which are probably given orally by teacher. 6 Structure • Balance of chapters • Endmatter (glossaries, indices, appendices) Number of pages of each unit uneven. It is not clear what the classification section is all about as it is not linked to the other two units on counting and shapes and patterns. There is no end matter. TEACHER’S GUIDE - KG1 NUMERACY A. Pedagogical Quality No. Criteria Remarks 1 Objectives • Objectives stated in the guide match the syllabus • Clarity of objectives (realistic & measurable) Outcome objectives are clearly stated at the beginning of each unit. 2 Organization and presentation • Sequencing of topics in relation to pupils’ book Book includes children’s workbook split up into activities that are augmented by the teacher. Each unit lists materials required, but some may not be Evaluation of TLMP – TLMP in Ghana 55 available to teachers. 3 Level of cognition • Activities measure the level of understanding (knowledge, comprehension, application, analysis, synthesis, and evaluation) The first unit on grouping and sorting is misplaced and children may not have the appropriate vocabulary for the items identified. 4 Teaching and learning strategies • Teaching and learning strategies enhance the teaching-learning process • Strategies help the teacher attain the stated objectives • Strategies give extra information compared to the pupils’ book Practical activities are varied, but could be expanded on, i.e., number songs; does not have the same variety of activities as literacy book. Each lesson has suggested activities for parents. Unit 3 on shapes and patterns is introductory to geometry and could be addressed later, paying attention instead to numbers beyond 10, 5 Teaching ideas • Guide provides new ideas for enhancing classroom instruction Not very creative; follows workbook fairly closely without many added activities. 6 Referencing • Extent to which the guide helps identify other materials and resources None 7 Extra questions/examples/Answers • Extent to which the guide gives extra questions and examples apart from those in the pupils’ book • Extent to which the guide provides correct answers to questions in pupils’ book Not many in this subject at this level. No questions. B. Physical Design and Quality No. Criteria Remarks 1 Type and quality of cover/paper • Attractiveness of cover • Water resistant cover • Opacity of paper • Weight of paper Card stock used for cover; not water resistant. Paper same as other books and TGs 2 Type and quality of binding • Type of binding in relation to number of pages • Strength of wire, thread or gum For a 68-page book on A4 paper, just two staples hold the materials together, suggesting limited durability. 3 Size and dimension • Size and dimension of the book A4 – OK for teachers 4 Quality of print • Legibility Good 5 Typeface and size • Typeface and size are easy to read Good Evaluation of TLMP – TLMP in Ghana 56 6 Structure • Forematter (rationale, theoretical framework) • End matter (appendices, glossaries, indices) Limited explanation of how to teach numeracy to 4-year olds (the target audience). Children who start KG can be much older. No end matter. PUPIL’S WORKBOOK – KG1 ENVIRONMENTAL STUDIES A. Pedagogical Quality No. Criteria Remarks 1 Conformity to the syllabus • Extent to which the text covers the entire syllabus • Depth and breadth of concepts • Arrangement of the material in respect to the pedagogical needs of the learner • Conformity to the general and specific goals of the subject area 2 Organization and presentation of content • Sequencing of content in relation to sound pedagogical approach (simple to complex, known to unknown) • Use of a variety of presentation techniques • Use of illustrations, italics, bolds, subheads, paragraphing The structure of this book does not conform to that of the others. Materials are presented in sections and then subdivided into units, whereas the literacy and numeracy books are divided in units and then subdivided into lessons. There are three sections: building a healthy individual, the physical environment and the social environment. The first unit introduces the self, the body and care of the body. Body parts are not introduced in the literacy book until unit 5 (page 13), but begins the section in ES. There will most likely be a vocabulary shortfall in this book. 3 Correctness of information • Correctness of facts and concepts • Accuracy of figures • Free from spelling and grammatical errors 4 Conformity to socio-cultural values of Ghana • Content abides with the moral values of Ghana • Content shows tolerance to diverse beliefs • Content reflects love and respect for the country Girl dressed in traditional clothes named Fatima, a Muslim name, which may require different dress. 5 Integration of skills and cross cutting skills • Content helps learners build positive life skills • Content informs learners on HIV/AIDS, gender, OVC, children’s rights, safety, environmental Pictures and exercise promote eating fruits and vegetables and how to cover your mouth when sneezing. Also covered is how everyone in the family helps to keep the house and Evaluation of TLMP – TLMP in Ghana 57 issues the surroundings clean. No other items covered. 6 Adaptability and applicability to local and global settings • Content informs learners on global technological, economic and political issues • Content informs learners of current issues • Extent to which knowledge in the text can be applied by learners in their locality N/A for the most part, but point above can be applied anywhere. 7 Relevance to the learner’s age and interests • Activities in the text relate to the learner’s age, needs, and interests • Illustrations and pictures relate to the level of the learner’s development and interests Vocabulary may be a bit above children’s level – e.g., “quench” on p. 12, “bamboo” on p. 27, “dolphin” on p. 32, etc., etc. When naming animals, it cannot be assumed that just because a child lives in Africa that they will be familiar with the animals of the savannah and know their habitats. The summary of all sections call for a lot of writing which the children have not learned to do in literacy. 8 Use of quality illustrations • Meaningfulness and relevance • Clarity • Location and balance in relation to text • Effective use of color • Variety of illustrations • Presence of captions • Correctness of illustrations Illustration on p. 10 shows a “healthy” baby and another that looks emaciated to illustrate why fruits and vegetables are eaten. This is inappropriate. Illustration on p. 14 asks child to circle best way of washing hands; it is not clear what is being taught. The illustration on p. 16 of children brushing teeth is not clear as the “stick” that the boy is using is no sufficiently detailed to differentiate it from a toothbrush. The illustration of the back side of a nude boy taking a shower is inappropriate. 9 Appropriateness for learner and teacher • Extent to which learner/teacher are likely to enjoy the material • Extent to which content guides the learner to the intended knowledge Topics covered are not interrelated and do not cover items sequentially; rather, sections have their own vocabulary. 10 Content appropriate for learners of diverse abilities • Content considers both slow and fast learners As a stand alone, the book could be used according to the teacher’s or the student’s pace. However, children will have difficulties in vocabulary and in writing as these have not yet been introduced in literacy. Evaluation of TLMP – TLMP in Ghana 58 11 Presence of enjoyment index • Content integrates elements of enjoyment (humor, jokes, cartoons songs, puzzles) • Use of different techniques to stimulate learners’ interests None 12 Assessment of student understanding • Use of feedback, summaries, diagnostics, practice and review activities • Adequacy of assessment activities to test different levels of cognition • Variety of assessment techniques employed Summaries appear at the end of each section, but they require writing which the children may not yet be able to do. No assessments in workbook. 13 Language and communication aspects • Extent to which the level of difficulty of language reflects the learners’ level • Extent to which concepts are exact • Correctness of spelling and grammar • Extent to which the material stimulates learners’ language usage (catchwords, proverbs, slogans) Language level may be a problem as there is so much new vocabulary. No other materials are present. B. Physical and Design Quality No. Criteria Remarks 1 Type and quality of paper • Water resistant cover • Opacity of paper & weight It appears there is a water￾resistant cover; paper quality is the same as all other books. 2 Type and quality of binding • Type of binding in relation to pages • Strength of binding wire, thread or gum There are 52 A4 pages held together with two staples meaning they are not durable. 3 Size and dimension • Size and dimension in relation to learners’ age A4 may be too big for children 4 Quality of print • Legibility of ink (text and illustrations) Good 5 Type face and size and sentence length • Typeface and size in relation to learners’ level • Lengths of sentences in relation to learners’ level Good – but sentences introduced in summaries and this is the first time they appear, except for instructions. 6 Structure • Balance of chapters • Endmatter (glossaries, indices, appendices) Uneven; nothing at end. Evaluation of TLMP – TLMP in Ghana 59 TEACHER’S GUIDE – KG1 ENVIRONMENTAL STUDIES A. Pedagogical Quality No. Criteria Remarks 1 Objectives • Objectives stated in the guide match the syllabus • Clarity of objectives ( realistic & measurable) Outcome objectives are set at the beginning of each lesson – check to see if conforms to syllabus. 2 Organization and presentation • Sequencing of topics in relation to pupils’ book Two manners of presentation include discovery teaching (child-centered) and direct teaching (review, present, guide, provide feedback, and encourage independent practice). 3 Level of cognition • Activities measure the level of understanding (knowledge, comprehension, application, analysis, synthesis, and evaluation) Exercises appear to be age appropriate, but sight words require additional understanding and teaching. 4 Teaching and learning strategies • Teaching and learning strategies enhance the teaching-learning process • Strategies help the teacher attain the stated objectives • Strategies give extra information compared to the pupils’ book Key sight words are identified for each lesson, but phonics is not part of the lesson. Materials needed are specified, but these may not be available to the teacher (e.g., large sheets of brown paper, fruits). Writing is required for summary exercises, and children have not yet been taught to do this. Songs are provided in the appendix, but no time is set as to how long it should take for each part of the lesson or to teach the song. 5 Teaching ideas • Guide provides new ideas for enhancing classroom instruction A broad range of hands-on activities are presented for each lesson. 6 Referencing • Extent to which the guide helps identify other materials and resources The TG identifies five titles for further use and ends with notes the teacher can make on activities that work well and/or need adjustment. 7 Extra questions/examples/Answers • Extent to which the guide gives extra questions and examples apart from those in the pupils’ book • Extent to which the guide provides correct answers to questions in pupils’ book The TG has much more in it than the children’s workbook. The workbook give the child the chance to do certain exercises and become used to working with a book, but it is not a text per se. The TG provides many extension activities to learn a lesson using different approaches. Evaluation of TLMP – TLMP in Ghana 60 B. Physical Design and Quality No. Criteria Remarks 1 Type and quality of cover/paper • Attractiveness of cover • Water resistant cover • Opacity of paper • Weight of paper Cover paper appears to be water resistant; pictures on contents are on the cover; paper used is standard for all books. 2 Type and quality of binding • Type of binding in relation to number of pages • Strength of wire, thread or gum The 91-page TG is held together with only two staples, thus reducing the durability of the book. 3 Size and dimension • Size and dimension of the book A4 paper seems appropriate for teachers 4 Quality of print • Legibility Good 5 Typeface and size • Typeface and size are easy to read Good 6 Structure • Forematter (rationale, theoretical framework) • End matter (appendices, glossaries, indices) The same framework is used for each lesson; introductory materials spell out the approach to be used; end matter includes an appendix with all songs, poems and games to be used. Some are parodies from US-based songs and games. PUPIL’S WORKBOOK – KG2 ENVIRONMENTAL STUDIES A. Pedagogical Quality No. Criteria Remarks 1 Conformity to the syllabus • Extent to which the text covers the entire syllabus • Depth and breadth of concepts • Arrangement of the material in respect to the pedagogical needs of the learner • Conformity to the general and specific goals of the subject area The same three sections that were in KG1 are in KG2 – individual, physical and social environment. It is not clear if these are what the Ghana syllabus calls for. 2 Organization and presentation of content • Sequencing of content in relation to sound pedagogical approach (simple to complex, known to unknown) • Use of a variety of presentation techniques • Use of illustrations, italics, bolds, subheads, paragraphing Healthy Individual - The first unit starts out with my physical needs – food, clothing, shelter and water with children asked to differentiate different uses of the basic needs. Pictures are appropriate. Writing through tracing letters is introduced. A rhyme in English is given, although it is not clear if instructions in the math book should be followed, i.e., teach in any language as it is the related concepts that are important. The summary requires a significant level of reading ability in English. Evaluation of TLMP – TLMP in Ghana 61 Unit two talks about spiritual needs and attributes of God. While Ghanaian culture promotes these, it is inappropriate for a religious unit to be included in materials paid for by the USG. Unit 3 focuses on taking care of boy waste with a picture of a boy defecating into a pit latrine and urinating into a toilet. The instructions are to cross out the “bad” things as it includes a boy putting a handkerchief around his mouth/nose while sneezing, and a girl sneezing into the air. The questions to be answered require writing and it is not clear if children have this skill. Physical Environment – Unit 1 addresses sources of light, but does not depict a fire and the dangers of having one – which is more common than an LED light bulb. Writing and reading are included although children may not have these skills. Unit 2, weather and seasons, has very good illustrations and has the children drawing different weather patterns. Unit 3 on keeping the environment clean does not depict implements used in Ghana. Unit 4 on safety finally depicts fire, and an electric iron, which is not necessarily used in the rural areas so children may not know what this is. Social Environment – Under unit 1 my school, three boys are depicted jumping rope; this is highly unlikely in Ghana as this is seen as a girls’ activity. While the flag is in the picture of the school, the instruction is to color it (not stating which color goes where) and nothing more is done to the flag. This is an opportunity to bring in civics, but the activities in the workbook do not direct the children in this way. Because this lesson has a lot of “letter tracing,” it is clear that children still do not know how to write, yet the summary lessons at the end of each unit require children to write. Unit 2 focuses on my community; the depiction of a health clinic would have opened up many areas for teaching, but only police station, PO, chief’s palace, a mosque and a church are depicted. The market is depicted in the next lesson, and only women are shown as vendors. Women selling foodstuffs is very common, but men also sell. This is an excellent time to raise the question why Evaluation of TLMP – TLMP in Ghana 62 women sell certain things and men others. It is only at the end of this unit that a hospital/clinic in the community as an important place, but there is no depiction. Under the unit festivals and other celebrations, only men are depicted in killing animals, drumming and dancing, and at a durbar. It may be that the queen mother is depicted behind the chief, but the picture is not clear are depicted also, and this is not a regular custom. Independence day is included in the next unit, with the children learning the national anthem and the national pledge, but no discussion or activities as to why this is celebrated. 3 Correctness of information • Correctness of facts and concepts • Accuracy of figures • Free from spelling and grammatical errors Well done 4 Conformity to socio-cultural values of Ghana • Content abides with the moral values of Ghana • Content shows tolerance to diverse beliefs • Content reflects love and respect for the country Eating nutritious food, drinking clean water, etc., are all in accordance with moral values. Using the proper receptacle to pass personal waste is also important as many people do not use sanitary facilities. By reinforcing appropriate behaviors, children will begin to act appropriately. However, the downside of poor behavior also needs to be addressed. Wearing the right clothes in different types of weather is presented, but the types of clothing depicted may not be a part of a household priority (rain coat, rain hat). The illustrations of why we go to school (pp. 68-9) are sexist – males are depicted as doctors, policemen, footballers, and soldiers, while women are depicted as teachers and nurses – a stereotypical set of gender-based professions. Birthdays are depicted as they are in the west (cake with candles, pp. 82-3) and do not take local customs into account. Birthday cards are also included and this is not a part of custom for children, especially in families that are resource poor. Sexism continues with the depiction of community leaders, all of which are men (except the mother in the picture of the parents) (pp. 89-90). This lesson clearly demonstrates that the writers knew little about the leadership roles women play, or wanted to ignore them. 5 Integration of skills and cross cutting skills There are a lot of misrepresentations in this Evaluation of TLMP – TLMP in Ghana 63 • Content helps learners build positive life skills • Content informs learners on HIV/AIDS, gender, OVC, children’s rights, safety, environmental issues section. However, the issue of safety is fairly well covered on an introductory level. 6 Adaptability and applicability to local and global settings • Content informs learners on global technological, economic and political issues • Content informs learners of current issues • Extent to which knowledge in the text can be applied by learners in their locality Only the final point is well covered. 7 Relevance to the learner’s age and interests • Activities in the text relate to the learner’s age, needs, and interests • Illustrations and pictures relate to the level of the learner’s development and interests Good 8 Use of quality illustrations • Meaningfulness and relevance • Clarity • Location and balance in relation to text • Effective use of color • Variety of illustrations • Presence of captions • Correctness of illustrations In general, these are good, but some are not as clear as others. 9 Appropriateness for learner and teacher • Extent to which learner/teacher are likely to enjoy the material • Extent to which content guides the learner to the intended knowledge Good coverage of issues in each section – movement from the individual to the family and to the community is good, although some other topics might have been included about social relationships and getting along (i.e., no bullying). 10 Content appropriate for learners of diverse abilities • Content considers both slow and fast learners OK 11 Presence of enjoyment index • Content integrates elements of enjoyment (humor, jokes, cartoons songs, puzzles) • Use of different techniques to stimulate learners’ interests Yes, songs and games are integrated. 12 Assessment of student understanding • Use of feedback, summaries, diagnostics, practice and review activities • Adequacy of assessment activities to test different levels of cognition N/A Evaluation of TLMP – TLMP in Ghana 64 • Variety of assessment techniques employed 13 Language and communication aspects • Extent to which the level of difficulty of language reflects the learners’ level • Extent to which concepts are exact • Correctness of spelling and grammar • Extent to which the material stimulates learners’ language usage (catchwords, proverbs, slogans) Not clear when writing is undertaken full scale; seems to be a mix-up between letter tracing and the need to actually write things. B. Physical and Design Quality No. Criteria Remarks 1 Type and quality of paper • Water resistant cover • Opacity of paper & weight Cover seems to be water resistant; paper that used for all texts. 2 Type and quality of binding • Type of binding in relation to pages • Strength of binding wire, thread or gum 102-page workbook is bound with glue, which may not be very durable in humid climates. 3 Size and dimension • Size and dimension in relation to learners’ age A4 may be too big for students. To reduce wear and tear, separate books might be developed for each section. 4 Quality of print • Legibility of ink (text and illustrations) Good 5 Type face and size and sentence length • Typeface and size in relation to learners’ level • Lengths of sentences in relation to learners’ level Not clear what learner’s level should be in KG2, although sentences appear to be short and direct. It is also not clear how and when sentence structure is introduced in written form. 6 Structure • Balance of chapters • Endmatter (glossaries, indices, appendices) Sections seem to be equally balanced. No end material in Children’s workbook. TEACHER’S GUIDE – KG2 ENVIRONMENTAL STUDIES A. Pedagogical Quality No. Criteria Remarks 1 Objectives • Objectives stated in the guide match the syllabus • Clarity of objectives ( realistic & measurable) Outcome objectives are established on the basis of activities, but not overall behavior. Not clear if what is covered is actually guided by the syllabus. 2 Organization and presentation • Sequencing of topics in relation to pupils’ book The move from individual, to the physical and then to the social environment seems OK, although Evaluation of TLMP – TLMP in Ghana 65 it is not clear how these are integrated. 3 Level of cognition • Activities measure the level of understanding (knowledge, comprehension, application, analysis, synthesis, and evaluation) It appears that children are provided a sufficient number of activities to learn in different ways so that all of these are addressed in one activity or another. However, there is no real means of measuring these addressed in the lesson plan. 4 Teaching and learning strategies • Teaching and learning strategies enhance the teaching-learning process • Strategies help the teacher attain the stated objectives • Strategies give extra information compared to the pupils’ book Word walls are introduced for all sight words presented at the outset of each lesson and then reinforced/used throughout the lesson. Teachers are directed to producing charts, drawings or pictures of different scenes, etc., to expand the application of the lesson; teachers may not have the resources to produce these items. Children are directed in each lesson to write something, although it is not clear when they learned to write. A number of field trips in the community are suggested; the logistics of guiding many children (how many are in each KG class?) requires significant planning so as to have enough staff to oversee the children. 5 Teaching ideas • Guide provides new ideas for enhancing classroom instruction Many given and teacher can choose what to do or can create her/his own 6 Referencing • Extent to which the guide helps identify other materials and resources Other materials are noted in the appendices. 7 Extra questions/examples/Answers • Extent to which the guide gives extra questions and examples apart from those in the pupils’ book • Extent to which the guide provides correct answers to questions in pupils’ book Some are given, but value questions are somewhat sketchy as sexism is apparent in the illustrations. The only questions answered are the summary questions at the end of each section. B. Physical Design and Quality No. Criteria Remarks 1 Type and quality of cover/paper Appears to be water-resistant; Evaluation of TLMP – TLMP in Ghana 66 • Attractiveness of cover • Water resistant cover • Opacity of paper • Weight of paper paper same as all books. 2 Type and quality of binding • Type of binding in relation to number of pages • Strength of wire, thread or gum 139-page TG is bound with glue, which may not be durable in humid climates. 3 Size and dimension • Size and dimension of the book A4 seems appropriate for teachers 4 Quality of print • Legibility God 5 Typeface and size • Typeface and size are easy to read A shift has occurred with this TG in the size of the typeface in the introduction: it is much smaller and does not look like the children’s workbook, although all the topics are there. 6 Structure • Forematter (rationale, theoretical framework) • End matter (appendices, glossaries, indices) There is very little introduction that is relevant specifically to the book; it is the general introduction used in other books. There are four appendices: rhymes and action songs; stories; key words; parental support; and references. The book is structured in the same way as the others, with objectives, suggested teaching and learning procedures, key sight words, teaching and learning resources, several activities, and summaries. The systematic approach across all books is helpful in guiding teachers through the teaching and learning process. Evaluation of TLMP – TLMP in Ghana 67 ANNEX H. THE ORIGINAL 14 TLMP DISTRICTS CRITERIA FOR SELECTING THE DISTRICTS 1. Lack of Access Road 2. Lack of Accommodation 3. Lack of Transport 4. Lack of Electricity 5. Lack of Communication 6. Lack of Water 7. Lack of Health Facility 8. Lack of School Infrastructure 9. Lack of Teachers Districts REGION DISTRICT DISTRICT CAPITAL Ashanti 1. Amansie West Assin Manso Brong Ahafo 2. Pru Yeji Central 3. Assin North Assin Fosu Eastern 4. Birim North New Abirem Greater Accra 5. Accra Metro Accra 6. Ga East Abokobi Northern Region 7. Sawla-Tuna-Kalba Sawla 8. Saboba 9. Chereponi Saboba Chereponi Upper East 10. Garu Tempane Garu Upper West 11. Nadowli Nadowli Volta 12. Krachi West Kete-Krachi 13. Ho Ho Western 14. Bia Bia ANNEX I. WORKBOOKS AND INSTRUCTIONAL MATERIALS PREPARED BY TLMP 2009-2012 YEAR TITLES Environ. Env. Num. Num. Literacy Lit. Environ . Enviro n Num. Num. Literacy Litera cy TOTAL S Wk Bk 1 TG 1 Wk Bk 1 TG 1 Wk Bk 1 TG 1 Wk Bk 2 TG 2 Wk Bk 2 TG 2 Wk Bk 2 TG 2 2009 270,000 8,501 270,000 8,501 270,000 8,501 202,000 5,501 202,000 5,501 202,000 5,501 1,458,00 6 2010/1 1 1st print 471,489 12,918 471,489 12,918 471,489 12,918 359,443 11,712 359,443 11,71 2 362,150 11,712 2,569,39 3 2nd print 289,511 7,732 289,511 7,729 289,511 7,732 222,890 7,188 222,890 7,188 220,890 7,188 1,579,96 0 TOTA LS 1,031,00 0 29,151 1,031,00 0 29,14 8 1,031,00 0 29,151 784,333 24,401 784,333 24,40 1 785,040 24,401 5,607,35 9 ANNEX J. CONFLICT OF INTEREST STATMENTS I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 03, 2013 Name THOMAS TILSON Title CONSULTANT Organization IBTCI Evaluation Position Team Leader Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 19, 2013 Name MEREKU, DAMIAN KOFI Title PROF. Organization IBTCI Evaluation Position Team Leader Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 19. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 20. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 21. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 22. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 23. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 24. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. √ √