PACEEQ Participation of Communities in the Quality and Equity of Basic Education Internal Evaluation of the Performance of the Literacy Centers of the PACEEQ Project Prepared for World Education Guinea and PACEEQ partners by: Dr. Kristen Edgar Potter August-November 2004 Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 2 List of Abbreviations Academy for Educational Development AED Alphabetiseurs Villageois (Village Literacy Facilitators) AV Association Parents Eleves et Amis des Ecoles (Parent Associations) APEAE Bureau d’Association Parent Eleves et Amis des Ecoles (PA boards) B/APEAE Education Action Plan EAP Education Development Center EDC Education Pour Tous EPT Non-Governmental Organization NGO Participation Communitaire a l’Education PACEEQ de base pour l’Equite et la Qualite Projet Gestion des Resource Naturale PGRN Research Triangle Institute RTI Service Nationale d’Alphabetisation (National Literacy Service) SNA World Education Guinea WEG Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 3 Acknowledgements This evaluation documents the work in the domain of literacy that has been done to date under the PACEEQ project as well as under the Community Development Through Mobilizing Support for School Improvement program implemented by World Education from 1997 to 2001. In addition to a brief history and overview of the literacy program in Guinea, this evaluation includes a compilation of strategies employed by the project that worked well and recommendations for improved strategic implementation based on documentation, field experience and evaluation data. The intent of this evaluation is to: • document program work to • improve the functioning of literacy centers that are currently operational, • assist literacy centers that have moved into self-sustaining cycles of literacy programming, and • inform future strategic planning. This evaluation was a participatory effort involving a team including both literacy and monitoring and evaluation specialists. Team members included PACEEQ program partners (World Education, Research Triangle Institute, Save the Children, and the Academy for Education Development) with staff both central and regionally based, as well as Guinean and international external consultants. The Evaluation Team extends appreciation for all the assistance made available by the kind staff of the PACEEQ project. Staff members were helpful and insightful, as well as clearly proud of the good work they had done. Special thanks to M. Barky Diallo, chief of party for PACEEQ, M. Sene Diop, program manager, and to the literacy team led by M. Mamadou Saidou Barry, Literacy Coordinator and Mamadou Samba Sow, Literacy Training Specialist, and M. Boucoum. We would also like to thank M. Diang and M. Boubacar for their assistance with translations and data entry and M. Hassan for his amazing and prompt data base development. Abdoulaye (Dad) Diallo and Kadiatou Diallo were instrumental in supporting the logistics in Mamou and in Conakry which was a big help. The teams in each of the project prefectures worked diligently to provide a solid base of information upon which this evaluation could be based. A final thank-you is extended to the communities, facilitators and learners in literacy centers that participated in the evaluation. The evaluation teams were welcomed into the communities and into the centers, where the literacy work was discussed with pride and honesty. A warm thanks to all PACEEQ staff, and to the PACEEQ communities. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 4 Executive Summary Overview of the PACEEQ Literacy Program The PACEEQ program began in October 2001 as a four-year project designed to increase community participation in primary schools to improve education quality and gender and urban/rural equity. One of the program strategies was to support community based literacy centers, with project indicators including the establishment of 225 village based literacy centers, serving 6750 learners, of whom at least 70% should finish the course with 75% of those completing the course be assessed as literate. The PACEEQ literacy program has developed a comprehensive process, based on the literacy and numeracy booklets piloted by World Education in Mamou from 1998 to 2000 and expanded it into a complete literacy program process. The PACEEQ literacy program helps communities organize, initiate and run literacy centers. Booklets, developed initially in Pular were transcribed into the three other major local languages of Guinea (Kpelle, Malinke, and Sousou) and a strong program process was designed. Social negotiation is used by communities to asses their need, interest, and ability to support a literacy center. A strong facilitator training component was added to prepare village literacy facilitators to teach the PACEEQ materials and to maintain the administrative documentation needed for transparent management of a literacy center. And a support and monitoring strategy using literacy specialists from regional NGO partners was organized. In 2002 communities began to establish literacy centers and by the end of the first year a total of 132 community literacy centers were functioning. Currently 260 literacy centers have served or are serving 7,582 learners. This exceeds the project target number of literacy centers by 16% and other centers are poised to open before the end of the year. Of the 161 literacy centers that have completed the initial literacy course, approximately 70 % of learners completed the course and 70% of the course completers were evaluated to be literate. The purpose of this evaluation was to document the management, functioning, and value added to the PACEEQ project of literacy centers, to explore retention, attendance, and attainment issues, and to develop a set of recommendations that offer ideas and guidance to improve future expansion of the literacy program. The evaluation process outlined in the Terms of Reference that was a three-step strategy: 1. Review and analyze the instructional materials, monitoring as support tools, project documents, and statistics that are associated with the literacy component of PACEEQ. 2. Design and Implement a field collection strategy, including the development of interview tools, and the organization of a collection strategy for teams in each of the project regions. 3. Compile and interpret data at a national level, synthesizing field observations with conclusions and recommendations The above described evaluation strategy has allow the evaluation team to respond to the following three key questions from the Scope of Work: Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 5 1. How does the PACEEQ literacy component strengthen a community’s ability to improve the quality and equity of the schools and community? How does literacy add value to the achievement of the overall goal of the project? 2. What are the factors leading to a lack of retention in literacy classes? Are there factors that increase / decrease drop-out rates? 3. What are the barriers to learners achieving the target level of “literate?” Are there factors that could be better supported in order to augment the achievement of literacy learners? Highlights of the Evaluation The evaluation teams visited 75 literacy centers, 261 community members, 526 B/APEAE members, 150 learners and 75 facilitators across PACEEQ’s six operating regions. The evaluation documented a program that is active and well received in the communities; 95% of facilitators in the literacy centers expressed a high level of satisfaction with the program; among learners, 86% thought the program was “very interesting” and 97% felt they were succeeding well in their classes. The need for literacy centers and enthusiasm in many communities is demonstrated in the simple number of centers has already far exceeded project goals, and the fact that a full 25% of the centers have completed their first course of literacy and have “re-opened” with new sets of learners. Additional centers exclusively for women have been established as well. These new initiatives indicate a program that is well received and responsive to community needs. Reviewing documentation from the literacy program at central PACEEQ level indicates sound collaboration with the Guinean National Literacy Service (SNA), as well as PACEEQ literacy materials being used by other development agencies new prefectures. Partnerships with communities are also strong, since need for literacy centers is based on community needs identified in the Education Action Plans of the APEAE and then established through a process of social negotiation. This interesting component of the literacy program process is an example of efforts taken to ensure ownership of the centers on a community level. The literacy program process includes a well designed and comprehensive facilitator training program, which is another strength of the PACEEQ literacy program. The nine￾day training offers a great deal of information to absorb in a single workshop, and both facilitators and B/APEAE members requested additional training support. The facilitator training program is vital to the success of the literacy program; a set of refresher trainings would increase the impact of the initial training messages and skills. The literacy program is designed to support the wider PACEEQ goals of increased community involvement in the schools, and as such, the literacy booklets are based around the theme “a school of quality.” The booklets are well organized and logical in terms of the literacy learning, but the pictures, which form the basis for the thematic discussions from which key words and phrases drawn, are a weak point. Post-literacy booklets are currently being developed. However, a clear strategy of how such a post￾literacy program would be run has not yet been decided. The program strategy should be decided prior to the development of the materials so the program design impacts the booklet designs. Themes for the post literacy booklets will be drawn from project target areas, but the focus should be on new topics, not re-visiting the same topics and pictures Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 6 again. The demand for the post literacy from learner, facilitator, B/APEAE members and community members during this evaluation is, in itself, and indication of the success of the project. The PAACEEQ literacy program process also includes an extensive evaluation, monitoring and support component. Formal assessments of learners, formative evaluations of each class session, observation tools to document on-going monitoring visits by the NGO literacy specialists, and a range of administrative documentation, ranging from lesson plans and enrollment records to visitor logs and inventory are kept by the facilitators. Many respondents during interviews noted that centers were poorly managed, and part of the problem is the excess of administrative documentation. The literacy center observation form documented an attendance rate of 68% on the day of the evaluation team visit, although 84% of enrolled B/APEAE members were present. When asked about non-attendance, health issues and social obligations were the main reasons offered, followed by poverty. It was also noted that centers were poorly managed and located far from learners’ homes. Facilitators explained that people dropped out because there was a lack of conviction about the importance of literacy among the community at large so learners (and centers) were not well supported. When learners were interviewed during this evaluation, 94% felt they were doing well, all agreed that the sessions were interesting. Over a quarter of the respondents noted the good methods used by the facilitator during class. Learners’ reported that between 72 and 89 percent of their children were enrolled in school, in hopes of a better future for the child and for their families. The main reason that children were not enrolled was poverty, but 22% also reported that there were not enough classrooms or teachers available. Learners said that after attending literacy classes they now buy more schools supplies for their children (35%), follow student work at home (26%) are careful about punctuality and absenteeism among students (11%), and have reorganized household chores to allow more time for homework (9%). 86% of the learners reported being more active in APEAE activities, attending meetings and donating money for the schools. Members of the B/APEAE were interviewed and were highly motivated to become literate. Overall 70% of the board members are enrolled in literacy classes. They report better knowledge and skills for communicating including taking notes and organizing documents, attending meetings where they participate in debates more often, and they have become more active in community awareness campaigns advocating for the school. Interestingly, board members noted the need for better social negotiation to support the literacy centers and learners. Community member focus group interviews were held with 261 community members, including 59 women. They explained that literacy was important for their community economically, for cohesion, and to focus interest on the schools. Lack of a stable site for literacy was their most common concern, and they explained that poverty was an issue impacting smooth running of the centers. Community members reported donating a room for literacy classes, working on community awareness and visiting the literacy classes among their contributions in support of the literacy centers. Gender equity was a concern and they had observed improvements due to the literacy class topics, as well as improvement in school/community relations. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 7 Recommendations • To remain a leader in the domain of literacy in Guinea the strong collaboration with SNA should be enhanced and contact with other literacy programs should be initiated to create synergies, allow PACEEQ to benefit from work done by others, and to broaden the impact of the PACEEQ literacy program. • The social negotiation process that has been established by PACEEQ is one of the important facets of the literacy program and should be expanded to include a wider section of the community beyond those associated with the APEAE and the formal schools. Better social negotiation would deepen community conviction regarding the importance of local language literacy, and thus increase support for the learners and the centers and diminish absenteeism and drop-out. • The PACEEQ literacy program materials are interesting, well organized and adapted to the target communities. However, booklets could be improved by better visuals that describe a topic and are attractive to learners and secondary audiences in the communities. Better pictures could positively impact learning and classroom practice, as well as increase the visibility of the program. • Post-literacy program strategy should be developed prior to designing post-literacy materials. Post-literacy materials should be based on new program messages and innovative design formats that are different from the initial literacy booklets. • The facilitator training is a solid foundation, but the training should be reinforced with a set of “refresher trainings,” held monthly in regional sites to support facilitators’ understanding and mastery of active methodologies and other skills. • A short course for the learners who complete the regular course but do not attain the level of “literate” should be developed. This course, adapted from the existing booklet, should focus simple reviews for all lessons in small group formats with peer teaching strategies. All willing learners—even those needing extra time and help—should be supported until they attain a solid level of “literate.” • A streamlining of the administrative procedures, combining required documents and eliminating unnecessary ones, would improve administrative organization of centers and the literacy component as a whole. With support from the excellent IT team at the PACEEQ central office, literacy program statistics should be re-organized into a data base that can be accessed easily and used to inform program decisions. • Enrollments should be finalized after a week of classes to eliminate non-committed learners. Ideas to ensure a committed learner population should be considered: a more comprehensive social negotiation processes, a system of down-payments, etc. • The formative evaluation process, which is important but complex, should be combined with monitoring using a system of clinical supervision. The clinical supervision process would develop reflective practice among facilitators while simultaneously enhancing the monitoring capacity of regional NGO literacy specialists while creating a “community of learners.” Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 8 Conclusion In conclusion, the PACEEQ project gains significant ‘added value’ from the literacy program. The project impact of the literacy component is identifiable in two primary areas, first the activity surrounding the B/APEAE and secondly in the attitudes toward enrollment and support of girls (and all children) in primary schools, including both a better understanding of the issues that are contained in “schools of quality” and also actions that can be taken in the home to support school success of children. A strongly positive impact was documented in both these areas. In terms of improved skill level, board members interviewed use both reading and writing skills for the benefit of the B/APEAE, attend meetings more frequently, participate in school activities, and even debate issues during board meetings. As such evaluation has documented that the PACEEQ literacy program is a solid factor in developing more confident and active members on the B/APEAE of target schools. Board members also report a greater understanding of issues underlying a quality school and their roles therein. Furthermore, respondents indicated that they spend more time now talking with other parents about the importance of enrolling children in school and ideas for following up on students at home, including making time for homework, reorganizing household tasks in favor of girls’ studies, and buying school supplies for their children. The PACEEQ literacy program is helping develop, stronger school-home bonds and better home-based support for student achievement. One of the major concerns of the literacy component is the drop-out rate among enrolled students. When questioned about the reasons that learners decide to quit literacy classes the most common responses were illness and social obligations. When facilitators were asked why people were dropping out, 35% reported that a poor understanding of the importance of literacy among community members was responsible, and another 15% noted that the centers were poorly run. Discussions with learners revealed that some enrolled learners attended only one course meeting; they had not grasped the amount of work that was expected and were intimidated. Some learners enrolled expecting the project to pay stipends to the learners, others were mocked by family or neighbors and pressured to abandon classes. Again better community awareness is indicated. It is reasonable to conclude that if a stronger social negotiation process was used among a wider segment of the community then more people would better understand the importance of literacy and encourage learners while maintaining inputs necessary for a smoothly running literacy center. Available statistics indicate that the PACEEQ program is slightly below the target indicator of 75% of tested learners attaining literacy. Learners need to be in class to be achieving literacy. As expected, when questioned about absenteeism, facilitators and learners discussed missing classes due to illness, social obligations and family chores. The program has instituted a policy of flexible scheduling so the classes can be organized any time of any day to meet the needs of the greatest number of learners. This sound policy helps learners continue to study while fulfilling family obligations. Teaching quality is also considered one of the factors in absenteeism and low achievement levels. Poor classes are less inspirational, learners get bored or lost, and learners feel time spent in classes could be better spent doing something else, all of which may eventually lead to dropping-out. The facilitators are offered just one training Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 9 course under the PACEEQ literacy program. Refresher trainings would reinforce the sound training program that is currently in place. Better classroom practice may inspire the learners to attend classes more regularly and achieve better results. In summary, the PACEEQ literacy program has strong positive impact on community attitudes and practices in support of the primary schools. Beyond the simple but well￾known theory that parents with ANY sort of educational experience are better able to support the education of their own children, the evaluation has documented that learners are more active in the B/APEAE meetings, participate more frequently in debates, use newly acquired reading and writing skills for record keeping, correspondences, and accounts of the APEAE, as well as being more active proponents of gender equity for students in the communities, and more supportive of student achievement in their homes. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 10 Introduction The PACEEQ project is a 4-year, $11 million USAID funded education project designed to enhance community support and equity in Guinean primary education. PACEEQ, in French “Participation Communitaire a l’Education de base pour l’Equite et la Quality,” began operations in August 2001, and now operates in six of Guinea’s eight administrative regions. PACEEQ is managed through a consortium of partners led by World Education Guinea (WEG) that includes four other U.S. based NGOs (Academy for Education Development (AED), Research Triangle Institute (RTI), Save the Children, and Education Development Center (EDC)) along with 25 Guinean NGO partners, 19 of which are involved with the literacy program. The vision of the project is “to improve community participation and gender and rural/urban equity in basic education through increased involvement and competence of civil society, specifically parents’ associations and local alliances.” (PACEEQ Revised Continuation Application, May 2003). As such, each community working with the PACEEQ project begins by doing a participatory community diagnosis to identify the issues that will need to be resolved while moving toward improved quality and equity in the schools. The results from the diagnostic tool inform an educational action plan (plan d’action pour ameliore l’education dans son ecole) in which many of the target communities identified the need for literacy and numeracy classes. Based on this community level need, a strategy to create and support literacy centers was included among the PACEEQ programs. Project indicators in literacy include the establishment of 225 literacy centers to serve 6750 learners, of whom 70% (4725learners) will complete the course and 75% (3543 learners) will be assessed as literate. Within the literacy program to date PACEEQ has supported the creation of 260 literacy centers at the community level, each associated with the Parent Association (Association des Parents d’Eleves et Amis de l’Ecole, APEAE) of the local school. Among the literacy centers, 22 were designed exclusively for women learners. PACEEQ has trained 478 literacy facilitators (alphabetiseurs villageoise), of whom 102 (21%) are women. Project support for the community literacy centers is offered through PACEEQ trained literacy specialists at regional level NGOs, and the literacy component of PACEEQ project. Overview of Evaluation Process This internal, mid-term evaluation of the functioning and performance of the literacy program under PACEEQ was designed in a participatory manner involving the PACEEQ Monitoring and Evaluation team, international consultants and staff working with the literacy centers in Guinea. The need for such an evaluation was identified in early 2004 as the project was preparing to expand the literacy program into both post￾literacy and into new communities. The overall goal of this evaluation was to document the management, functioning, and value added to the PACEEQ project of literacy centers, and to develop a set of recommendations to guide and improve future expansion of the program, while exploring retention, attendance, and attainment issues. The Terms of Reference developed by WEG and RTI in Guinea emphasize the importance of gathering and analyzing data from operational literacy centers to provide Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 11 and overview of the current situation as well as to create a base of information to inform future directions of the PACEEQ and future literacy programs in Guinea. For more detail please refer to attached evaluation overview (J. Spratt, May 2004). The evaluation was designed to be carried out during a six week period (from 15 August through 30 September 2004, later expanded to last through November 2004 due to programming constraints). The evaluation plan was outlined in the three steps below: 1. Review and analyze of the following documents and materials: ƒ Literacy instructional materials, including both the literacy and numeracy booklets and accompanying facilitator manuals, and the post￾literacy materials developed by PACEEQ staff. ƒ Training components for both the pedagogical aspects of the teaching of literacy and the administrative aspects of running a literacy center, as well as training for staff in support of literacy centers. ƒ Quantitative information from each of the literacy centers collected since the inception of the PACEEQ literacy program. These data include information regarding learners’ literacy levels prior to the course, and again as a final evaluation when the course has been completed showing enrollment, drop-out numbers, and the attainment levels at the completion of each literacy cycle. ƒ Field support and monitoring systems, as well as monitoring systems that are in place and operational at the time of the evaluation. 2. Design data collection tools including: ƒ Classroom observation grid for literacy center field visits, emphasis on satisfaction, attendance issues, behavioral and attitudinal changes, and administrative procedures. ƒ Interview protocols for interviews with learners from each of the literacy centers visited, as well as involved community members. ƒ Meeting with evaluation team members drawn from literacy personnel to ensure correct and efficient application of tools, and sound initial compilation of data into informational grids at the regional level. 3. Compile and interpret data at a national level in a report synthesizing collected information with conclusions and recommendations. ƒ Data summarized by region into summary grids ƒ Data entered into a data base for immediate analysis and for use as basis for future decisions and strategic plans and to determine change over time in attitudes and practice. ƒ Incorporate data into draft report presented to representatives of the PACEEQ leadership and the PACEEQ literacy component staff to confirm validity of analysis, and solicit additional input for the final draft of the report. ƒ Produced final report Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 12 The above outlined three-step process, in addition to offering a program overview, will enable the evaluation team to respond to the following key questions: 1. How does the PACEEQ literacy component strengthen a community’s ability to improve quality and equity in schools and community? How does literacy add value to the achievement of the overall goal of the project? 2. What are the factors leading to lack of retention in literacy classes? Are there factors that increase/decrease drop-out rates? 3. What are the barriers to learners achieving their targeted literacy levels? Are there factors that could be better supported in order to augment the achievement of literacy learners? For further details the Scope of Work and Terms of Reference are attached as an annex. Development of the PACEEQ Literacy Program The PACEEQ literacy program was based on the literacy program developed by World Education from 1998-2000 in the region of Mamou as well as literacy experiences of Save the Children. Prior to PACEEQ, World Education Guinea (WEG) pursued two separate branches of literacy. The first was a finite program of integrated health and literacy for emerging literacy among refugee women in Guinea’s Forest Region, which was designed in collaboration with the American Refugee Committee. The second was a World Education initiative based on an expressed and observed need for increased literacy among members of the APEAE in Mamou region. When the PACEEQ program started, a direct collaboration with the Guinean Service National d’Alphabetisation (SNA) was formalized. The Guinean government representatives met with the World Education literacy component leadership to coordinate program efforts, methodologies, and materials. Furthermore a coordinated strategy for materials development and training was agreed upon. Administrative procedures for the creation and management of literacy centers in the PACEEQ rural target zones were also developed. This cooperation between the PACEEQ literacy program and the SNA remains strong, and is one of the factors that add credibility and sustainability to the PACEEQ literacy program. Professional literacy personnel, trained under the Guinean governmental literacy program, were hired by WEG as literacy coordinators and training specialists. Technical staff from World Education’s global literacy programs assisted in the initial design of the two literacy booklets, one for reading/writing and one for numeracy. Although the original literacy program was only implemented with APEAEs in the Mamou Region, the literacy and numeracy materials developed at that time serve as the foundation for the PACEEQ literacy program materials. As agreed upon with the National Literacy Service (SNA), the literacy booklets first developed in Pular were transcribed into Malinke, for use in the regions of Middle and Upper Guinea (Moyenne Guinee and Haute Guinee). As the PACEEQ program expanded to other regions booklets were transcribed into Sou-Sou and Kpele for use in the Coastal and Forest Regions respectively. The pedagogy employed in the booklets is based on the “storyboard” methodology for which World Education is known globally. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 13 Technical support for the literacy booklets was drawn from the PACEEQ literacy staff, the SNA, and World Education literacy specialists. As part of the professional development of the PACEEQ literacy team, the coordinators were sent on a technical exchange program to Mali, for training with the well￾developed World Education Literacy Program that is in place there. The training program included aspects of active and student centered methodologies, practical activities, the importance of integrating content messages (such as health information or school quality issues) into the literacy materials, as well as practice-and-review techniques and formative evaluation practices. These ideas were incorporated into the PACEEQ literacy program processes and training design. Community need for literacy centers is first identified in the Educational Action Plan developed by an APEAE. Next, a process of social negotiation is undertaken. To facilitate the social negotiation on a community level, two sets of criteria were developed in a participatory manner that included learners, teachers, parent association members and members of literacy programs and the PACEEQ project. Communities review and then apply criteria for setting up a literacy center and for selecting community members to become literacy facilitators. The following table outlines the criteria agreed upon for creating literacy centers at a community level: Lists of Criteria for Creation of the Village Literacy Centers Criteria Identifying Community Readiness for Establishment of Literacy Centers Criteria Identifying Potential Candidates for Literacy Facilitators in Communities A community ready to open a literacy center should demonstrate the following: ƒ Existence of a primary school ƒ Available space for literacy classes ƒ Willingness to support a literacy center ƒ Eagerness to attend literacy classes ƒ Presence of resource people to manage and support the literacy centers ƒ Existence of a supervisory committee on the parent association ƒ Means to support literacy facilitator ƒ A chosen class site accessible to many learners from the community ƒ Inclusion of literacy in the Education Action Plan The candidates for the post of literacy facilitator should possess the following: ƒ Good reading and writing skills ƒ Capacity to learn and try new methods of teaching ƒ A stable economic situation ƒ Time to prepare and teach courses ƒ Good communication skills ƒ Willingness to serve as a resource person in the community ƒ Live in the community ƒ Personal commitment to literacy ƒ The respect of the community and of the potential learners The criteria described above help determine community readiness and willingness to support a literacy center, and help initiate the process of actually creating a literacy center. The theory is that because these criteria are clearly delineated and publicly discussed they enabled sincere transparency regarding identified needs, necessary village-level support, and appropriate choice of two literacy facilitators. Criteria are shared with communities during social negotiation sessions carried out in conjunction with the regional NGO staff and the local APEAE of schools in PACEEQ targeted rural Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 14 communities. Based on the criteria, communities organize and begin the process of establishing and running a community literacy center. When a community is ready, the PACEEQ partner NGO signs an agreement with the Board of the Parent Associations (B/APEAE) and two village literacy facilitators, which formalizes the support and outlines responsibilities involved in opening a literacy center. Under this agreement the following support is offered by PACEEQ to each of the Literacy Center: ƒ Pedagogic and administrative training for the two facilitators and for the support team from the local prefectoral NGO. ƒ Materials for the literacy centers including: teacher manuals for literacy and numeracy, student booklets for literacy and numeracy. ƒ Administrative copybooks to record student attendance and progress, inventory controls, lesson plans and visits to the literacy centers. ƒ Regular support visits during which the NGO trainers offer methodological and administrative support for the facilitators and for the literacy centers. ƒ A stipend, offered after having completed 75% of the lessons based on learner attendance. The stipend is a total of 150,000 GNF that is distributed by the B/APEAE to motivate the facilitators and to support didactic materials (including chalk, blackboards, rulers, lamps, etc.) Each community that is establishing a literacy center, presents two candidates that meet the criteria and who have agreed to become literacy facilitators. Literacy facilitators are offered an in-depth, nine-day training that includes sessions on social negotiations, adult learning strategies, storyboard methodology, overviews of the literacy and numeracy booklets, model lessons, practice lessons, review and exercise techniques, evaluation tools (formative evaluation, self-evaluation, pre- and post-testing), as well as management of a literacy center including administrative procedures and documents necessary. Following the training for the literacy facilitators, the literacy center enrolls learners, is given booklets and facilitator manuals and starts the literacy classes. When a literacy center is opened, classes begin by pre-testing learners to establish a base-line measure of literacy skills. A mid-term evaluation is used as a formative tool to inform and guide teaching practice during the course. Upon the completion of the literacy and numeracy course the centers offer a final test which determines the level of literacy obtained by each of the learners. The final test is proctored by an external agent from the regional NGO to determine achievement levels for each learner. Learners that succeed on the final exam are considered graduates and given certificates of literacy. Those who are tested at a level of “semi-literate” or “beginner” are encouraged to repeat the course if and when it is offered again in their village. It should be noted that each PACEEQ literacy center has a slightly different character, based on the cultural and economic requirements of each community. Following a series of social negotiations, the community and the facilitators agree on a contract. Community support offered the facilitator may be monetary, in-kind contributions (such as rice) or agreements to help the facilitator for a specified period of time in fields or other labor. There are some centers in which the facilitator asks for no compensation. Course schedules also vary. Some communities choose to meet every day, others as little as twice a week. Some centers find attendance better if the classes are organized in Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 15 the evening; others prefer morning classes; others still meet both in the mornings and in the evenings. Such organizational details are left with each of the communities in order to maximize the attendance rates and to encourage communities to take ownership of the centers and the processes involved in running the centers. Overview of PACEEQ Literacy Program to Date The PACEEQ literacy program became functional in 2002 when 16 centers were opened in two prefectures (10 in Labe, and 6 in Faranah). The centers expanded rapidly and by the end of 2003, a total of 132 literacy centers were functioning. By August of 2004 a total of 266 literacy centers had opened, and still more communities served by PACEEQ are organizing to open literacy centers within the year. As the number of literacy centers grew, it was observed that women learners were under-served: the illiteracy rate among women was much higher than that among men, but men comprised the majority of literacy learners in most of the literacy centers. Furthermore, women learners in the existing centers (and women who had not yet enrolled) explained that they were not always comfortable expressing themselves freely in mixed gender classes. To address this problem, 12 literacy centers exclusively for women were opened in Faranah and Kankan. The number of women-only literacy centers has continued to expand. By June 2004 there were 22 operational women literacy centers in 5 of the PACEEQ operating regions. A total of 30 women-only literacy centers are projected to open before the end of 2004. The PACEEQ funding is the primary constraint to opening more women-only literacy centers. Project funding limits the number of women-only centers to two per prefecture. However, in recognition of the great need for more women centers and to encourage communities, World Education staff (both in Guinea and internationally) have donated personal money for printing costs and the stipend. To date employee contributions have enabled the opening of approximately eight additional women centers at PACEEQ levels of support. Other demand-driven initiatives were also based on the success of the initial literacy program. Communities and learners requested more classes and continued instruction in literacy, and the following two additional initiatives were developed and implemented: ƒ First, current centers have been “re-opened” to enroll new sets of learners (and re-enroll those learners who either dropped out or did not attain the level of “literate” during their first course). “Re-opened” literacy centers are run entirely on community initiatives beyond the PACEEQ contribution of learner booklets and original literacy facilitator training. As of June 2004 there were 66 “re-opened” literacy centers in two of the operating regions (Kankan and Labe). All the “re-opened” centers are mixed gender, but some women centers are currently discussing the feasibility of re-opening as well. ƒ Second, based on learner requests and community observations, a need for a second level course and post-literacy reading materials in local languages was identified. Post-literacy booklets are currently being developed by the PACEEQ literacy team. The proposed format of the “post-literacy” booklets Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 16 includes a “picture story,” a text for reading and comprehension practice and some questions to review content messages, and to practice writing skills, a format similar to the initial literacy booklets. Given the wide popularity and rapid expansion of the World Education literacy program under the PACEEQ project, it was recognized that this was an opportune time to do an internal evaluation, document work to date of the literacy program. Presentation of Data Collected for the Evaluation Overview and Analysis of Statistics As described in the preceding section the evaluation began with an analysis of statistics and documentation that have been produced by the PACEEQ literacy program. The following table identifies all literacy centers, by region, and by the type of center (women only, mixed gender, or re-opened (mixed)) as well as whether each of the centers is currently functioning—which is to say, operating a literacy program that is active as of August 2004. These data were used to develop the sample population for the evaluation process, so the 75 centers visited could be distributed across regions and types of centers: Description of Centers Boke Faranah Kankan Labe Mamou NZ Total Functioning Includes 41 0 0 48 2 14 105 Women centers 2 0 0 6 2 0 10 Mixed centers 39 0 0 0 0 14 53 Re-opened (mixed) 0 0 0 42 0 0 42 Closed/Evaluated Includes: 0 44 97 20 0 0 161 Women centers 0 4 8 0 0 0 12 Mixed center 0 40 65 20 0 0 125 Re-opened (mixed) 0 0 24 0 0 0 24 Total Centers (functioning or already evaluated) 41 44 97 68 2 14 266 Total # of Centers projected by PACEEQ 40 40 53 62 0 30 225 The PACEEQ literacy centers were initially conceived of to improve the functioning of the APEAE, and in particular the capacity of the B/APEAE. Literacy centers were also Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 17 considered as a means to support educational equity through women in the communities who have a great impact on the education of their children. As such it is interesting to note the gender breakdown of the literacy learners, as well as the number of the members of the B/APEAE that have enrolled in the literacy classes. The following data, drawn from the June 2004 trimester report gives a break-down of the learners enrolled in literacy classes: Region # of Centers Total Women enrolled B/APEAE enrolled Enrollment Total % Total % Boke 41 740 355 48% 259 35% Faranah 44 1,720 877 51% 250 15% Kankan 97 2,085 855 41% 493 24% Labe 68 2,602 1379 53% 768 30% Mamou 2 60 58 96% 2 3% NZkore 14 435 174 40% 117 27% Totals 266 7642 3700 48% 1889 25% Average enrollment: 29 learners per center Attainment statistics are also kept by the literacy component, and they indicate two different ends of the spectrum. First the drop out percentage is shown by the difference between the number of learners enrolled in the course when the program began and the number of learners evaluated at the end of the class sessions. Second, learner achievement is shown by the number of learners attaining the level of “literate” on the final assessment. The following table is drawn from an analysis of the 161 literacy centers where the literacy program has finished and in which the learners have been evaluated by region. Region # Centers Drop-outs closed and evaluated # Learners originally enrolled # Learners evaluated # % # Learners literate % tested learners literate %enroll learners literate Faranah 44 1678 1204 474 28 673 56 40 Kankan 97 2085 1585 500 24 1199 76 58 Labe 62 2180 1656 524 24 1254 76 58 Totals 161 5943 4445 1498 25 3126 70 53 Overview and Analysis of Literacy Component Documentation The documents reviewed include the instructional materials, the training materials, and the support and monitoring materials that are used in literacy centers and during field visits. Samples of each of these documents will be included in the Annex for in-depth perusal, but a brief description of each of the documents follows. BARKY PLEASE ATTACH THESE IN ANNEX Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 18 Instructional Materials: The literacy instructional materials used in the PACEEQ literacy program were a participatory effort between the National Literacy Service (SNA), PACEEQ literacy staff, and World Education. The classroom materials, both literacy and numeracy were designed originally in the local language of Pular. They were subsequently adapted into other local languages, following the same content topic information, but using the language specific alphabets, which have varying numbers of letters and symbols, so the booklets for different languages have slightly different numbers of lessons; Pular has 16 lessons, Malinke has 20 lessons, and both Kpelle and Sousou have 18 lessons. The numeracy booklet has remained largely unchanged beyond key word translations. Literacy Booklets All the PACEEQ literacy booklets are designed to follow the same lesson format. Each lesson begins with a review of the previous lesson to reinforce both skills and content information from the previous class session. Next, Learners are asked to observe and discuss a drawing that illustrates a school quality issue drawn from PACEEQ program goals. Following discussions, a “key phase” is decided upon and written on the blackboard with new letters introduced to form the basis for reading and writing exercises. At the end of each class period, the facilitators and the students complete formative evaluation forms designed to be used as a planning tool by the facilitators and as a support tool for the NGO and PACEEQ staff who visit the centers regularly. Numeracy Booklets The numeracy booklets have 30 lessons which focus on numbers, mathematical symbols, and the operations of addition, subtraction, multiplication, and division. The methodology is designed to move from concrete ideas using manipulatives, to semi￾concrete ideas using pictures of objects, to abstract using number and symbol representations. Booklets are based on a review-and-practice technique and are not dependent on high levels of literacy, using explanations are symbolic rather than word dependent. Facilitator Guides Both the above mentioned booklets have associated facilitator guides. Facilitator guides are essentially the student manuals with inserted text that explains, in a step-by-step fashion, the way to present each activity and every lesson. Each activity is outlined including directions explaining what the facilitator should both say and do, with examples for reference. The facilitator guide is intended to reinforce the active methodology presented during the nine-day training that facilitators attend prior to opening the centers. The guide is clearly organized, comprehensive, and user friendly. Post Literacy Booklets Post literacy booklets are in the initial development stage. Unlike the literacy booklets these three documents are being drafted first in French and will be translated into the four local languages used by PACEEQ literacy centers. All booklets are still in early draft format. Facilitator manuals and training sessions have not been designed yet, nor has a post-literacy program strategy. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 19 The first draft post-literacy manual is based around the theme “a school of quality.” The intention is to reuse drawings from the initial literacy booklets and add additional text and discussion questions have been added. All topics proposed for this booklet have already been covered in the initial literacy booklets. A second proposed “post-literacy” booklet will integrate health messages surrounding HIV/AIDS. It is organized as a summary of a Parent Association meeting and “a day of HIV/AIDs information,” targeting healthy behaviors to avoid HIV/AIDs and then a section presenting an APEAE annual action plan. The logical thread is not clear. A third “post-literacy” book is anticipated, to be based on the importance of education for girls. Again, all the proposed topics have already been covered in the initial literacy booklets, however all draft post-literacy materials are still in the early stages of development. Training Materials PACEEQ program supports the opening of community based literacy centers by offering training to the literacy facilitators. This training is important since most facilitators are not trained teachers—and may have just graduated from the literacy centers themselves. To prepare the literacy facilitators for their role in the classrooms, PACEEQ has designed a nine-day program for facilitators. The stated goal of the training is to know the methodology and the administrative systems for the literacy centers,” with the following objectives: ƒ To master the contents of the literacy and numeracy booklets ƒ To practice teaching both literacy and numeracy classes ƒ To apply the evaluation tools to the class sessions When the training was first developed it was a 13-day program. This was determined to be too long, so the training plan was re-organized to fit within a two-week time frame. The current, nine-day training is offered over the course of 10 days with a one-day break at the mid-point. This is a more reasonable amount of time for a training program, but it is a great deal of information for people to absorb in a limited amount of time. The training plan is laid out clearly and in great detail. Each session has a title, time allotment, a list of objectives, and the procedures to be employed for running the session. Each of the training days begins with an “ice breaker/ warm-up” and ends with an evaluation of the training day. Starting the second day, training days also begin with guided reflection on the previous day. The training topics are logical and well organized. Please refer to Annex G for an overview of the facilitator training program. Assessment, Monitoring, and Support Materials for the Literacy Centers The PACEEQ literacy program has placed a strong emphasis on both academic assessment of learners and on formative evaluation of the literacy classes. This two pronged approach was designed to target teaching practice to learner needs and to encourage reflective practice among facilitators. The learner assessment is administered in three stages. A pre-test enables facilitators to determine student level upon entry into the class and make teaching decisions based on Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 20 that information. A mid-term evaluation is administered to determine if certain parts of the course should be re-taught or reviewed. The final assessment determines if learners have achieved the level of “literate” upon completion of the literacy course. The formative evaluation is expected to be applied each class meeting. There is a two￾page form where facilitators record student responses and a two-page form that is filled out by the facilitator as a self-evaluation. The idea is that this information, gathered at the end of each class meeting enables the facilitator to determine the success of the class in terms of learning and in terms of the classroom activities employed. This knowledge should be used by the facilitator to adapt the lesson plan for the next class and to adapt classroom methodologies based on self and learner responses so methods can be improved in future classes. In addition to the evaluation and assessment tools discussed above, literacy facilitators are required to keep a number of administrative records for the literacy center including: 1. enrollment booklet with learner data 2. attendance booklet inclusive of reasons that learners may have been absent 3. assessment booklet, including data for each learner on pre-, mid- and final tests 4. lesson plan booklet 5. inventory record 6. records of materials borrowed and returned (primarily lesson booklets) 7. visitor logbook The required procedures for proper and timely completion of these administrative documents is included in the initial facilitator training program, so the use and the processes have been explained to the people required to keep these records. Documents are expected to always be available in the classrooms so monitoring visits can confirm they are up to date and used correctly. Literacy centers are supported and monitored regularly by regionally based NGO literacy specialists. The expectation is that the literacy specialist visits each center not less than twice each month. During this visit, the specialist completes a different monitoring form. This form should be the basis of discussions between the facilitator and the specialist with highlights written into the visitor logbook so all visitors can follow progress of the center and facilitator. Overview of Field Data Collected by the Evaluation Team Field data was collected during a two week period at the end of September. Of the intended sites, all 75 were visited. Each visit took approximately three hours. Difficulties encountered by the collection teams tended to be logistic in nature—the rains in Guinea are late and heavy this year so roads to some sites were nearly impassable. Furthermore, the collection of data fell during harvest time in some regions so not everyone was available to be interviewed, although enough were there to complete all surveys at every site. Despite difficulties, all sites and facilitators were visited, a total of 150 learners, 526 members of the B/APEAE, and 261 community members were interviewed. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 21 Upon completion of the collection phase, the evaluation the data were compiled on regional summary grids, presenting a summary of all the data from each of the four interview protocols. The regional data was entered into a database at the central PACEEQ office to assist in analysis and to create an organized source of information into which new data can be added and from which conclusions can be drawn to inform future decisions. In other words, it is hoped that data collected for this evaluation process will be used as a first step in re-organizing information from literacy centers into a database that will be regularly updated throughout the remainder of the project. In addition to the field data collected by teams in the six regions, the international consultant made field visits to three active literacy classes, two of which were supported by the PACEEQ. Observations were made during each of the visit. Copybooks and texts were examined and short interviews were carried out with facilitators, learners, and community members. In general the observations made in the field supported the data that was gathered by the collection teams. Overview of Data by Evaluation Tool Literacy Class Observation Form During this evaluation, a total of 75 literacy centers, in each of the six PACEEQ operating regions were visited. The literacy centers that were visited enroll (or enrolled) a total of 2,249 learners, of whom, 1421 were women (63%), and 528 were members of the B/APEAE (23 On the day that the centers were visited by the evaluation teams a total of 1,536 learners were attending class, so learner attendance was 68%. It is interesting to note, however, that although overall attendance was 68%, attendance among members of the B/APEAE was 84%. When learners were questioned about reasons for non-attendance, 29% cited health reasons, 25% explained that there were social obligations (such as births, weddings, funerals), and 22% reported that people were working in their fields. An additional 16% reasoned that the literacy center was too far to walk regularly for the absent learners, and 7% reported that people did not attend class because the centers were poorly managed and the community didn’t understand the importance of literacy. A clear explanation was not given for what “poor management” of the centers meant exactly. Nonetheless, it can be attributed to either poor day-to-day activity/organization of the classroom or poor running of the center by the community. This idea will be discussed later since there are implications for both the facilitator training program and for the social negotiation process that is implemented prior to the establishment of a center. When facilitators were asked why some people had dropped out of the classes, over a third (35%) explained that the main reason for dropping out was a lack of conviction on the community level regarding the importance of literacy. An additional 15% of the respondents also explained that people had dropped out of the literacy classes because the centers were poorly run. Other reasons mentioned include moving away, social issues, and work either in homes or fields. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 22 Most centers had many more literacy/numeracy booklets than learners. The figures were not reported in Kankan, so eliminating that region from the totals, there were a total of 1651 learners and 3675 booklets. In rough figures this indicates that there was a surplus, across the regions, of about 20%, which, presumably is a 10% surplus of literacy booklets and a 10% surplus of numeracy booklets. The books were reported to be kept in reasonable condition in about 85% of the classes. Administrative records were reviewed in all centers visited and in most cases the different records were up to date and in good order. It was, however, noted that there are several other records that were not reviewed, and that keeping the various records was clearly a strain on the facilitators and the community. Finally, when facilitators were asked if there was anything they would like to report to the evaluation team, after expressing concern about a lack of sufficient quantities of booklets (which does not correlate with other sections of the questionnaire,) they mentioned concern over absenteeism (22%), which has also been discussed in previous sections. Although 11% complained that they needed a classroom devoted to the literacy center—that the place to hold the classes was a problem, and 7% mentioned that the facilitator was not sufficiently supported, with an additional 6% requesting more training for the facilitators, nonetheless, 17% of facilitators noted there was good collaboration between the centers and the B/APEAE. In 95% of the centers that were visited (71 out of the 75), literacy facilitators reported very high levels of satisfaction. The facilitators indicated that approximately 95% of their learners were highly satisfied as well. In discussions with both learners and teachers it was clear that many held strong convictions regarding the importance of literacy classes. One facilitator reported that she had been a learner during the World Education pilot literacy program, and had felt so good about herself after becoming literate that she ultimately became a facilitator, not only for the PACEEQ sponsored literacy classes but for classes with another organization as well. Learner Interview Protocol During this evaluation process, two learners from each center were spoken with individually. Some of the learners were members of the APEAE, others were chosen at random. According to the learner respondents, the age of learners in literacy centers ranges from 15 to 70, but the average age of learners was approximately 40. A total of 150 learners were interviewed for this evaluation, of whom, about 65% were women. When learners were asked why they attended literacy classes the main responses centered on the desire to learn to read and write and help themselves and others in life (44%). Interestingly, the second most common answer was that people attend literacy classes to better support their children in formal schools (30%). When asked the converse, why they did NOT attend literacy classes, 91% explained that illness or social obligations were primary causes of absenteeism. “when I was a learner I understood how much work it took to learn to read and write, now I am trying to make it easy for others to benefit, as I have from literacy classes…” Literacy Facilitator in Siminko Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 23 When questioned about their achievement in literacy classes only 4 people reported that they were only doing “just passably,” while 86 learners reported doing “well,” and 60 considered their work “good enough.” All respondents felt that they were interested in the sessions, and 86% reported being very interested in the literacy lessons. 26% of the respondents reported appreciating the methodologies used by the facilitators in their classes. Learners were asked about the numbers of children in their families that were of school age, and how many of those children were enrolled in formal schools. The percentages varied from region to region with the lowest enrollment in general (72%), and the lowest enrollment of girls (67%), both being in Kankan. The highest percentage of children enrolled (89%) and highest enrollment of girls (93%) both were in N’Zerekore. No enrollment figures were reported in the Mamou interviews. When respondents were asked why they send their children to school the responses fell into three general categories. 48% of the respondents noted that literate people are more responsible and useful for society. Over a third of the respondents (37%) were hoping that their children would then be better able to help support their families. And 15% of the learners hoped their children would “have a better future. When asked why they did NOT send their children to school, 40% reported that children were needed to help in the fields or that the families were too poor to be able to support the school costs for all their children. Another common response (22%) was that there were not classrooms in the school for another class of children or a teacher was not available. 10% of the respondents explained that many people do not understand the importance of sending children to school, and 8% reported that their children were attending Koranic schools. Learners were questioned about the support they offer the students in their family. Over a third (35%) reported that they now buy more school supplies for students in their families. A quarter of the respondents explained that they can now follow the students’ work at home and they organize time for children to study after school hours. An additional 9% reported that they had re-arranged the household tasks more equitably so all the children (not just the boys) had time for home-work. 11% of the respondents explained that after having attended literacy classes they now were more careful about punctuality and attendance than they had been before becoming learners themselves. Learners were asked how their own learning experience in the literacy centers had specifically impacted the educational support they gave the children in their homes. Nearly 40% reported that they check their children’s copybooks and organize review sessions. Another 22% added that they are better able to closely follow their children’s progress. Nearly a fifth of the respondents explained that they now understood the importance of school supplies such as pens and pencils. Finally, 18% of the respondents explained that they were now more concerned with gender equality and tried to reduce household tasks for their girls so the girls could have more time at home to study. “I check my daughter’s copybook each day. I still cannot read what she has written, but now I know that she should have something written each day, so I can tell she has been to school and paid attention.” Learner in Guellin Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 24 Only 3% of the respondents noted that they better understand the roles of the APEAE, but these respondents were not necessarily members. However, when questioned about their support for the APEAE, 86% said they participated in APEAE activities. 84% said they frequently attend meetings, although less than a quarter of respondents admitted to speaking during the meetings. Of the learner respondents, 8% felt that their support was to donate money for the school and assist in school work projects. The learners were offered an opportunity to give comments to the PACEEQ program and 70% of the learners interviewed asked for increased support in the form of financial incentives, supplies, training, and a literacy site. 40% of the learners also asked for support in re-opening the current centers or establishing a post-literacy program. B/APEAE Members Interview Protocol A total of 526 B/APEAE members, 30% women, were interviewed in a focus group format. Across all the centers visited, approximately 40% of the board members were women, and of the 70% of the board members were enrolled in literacy classes, again, 40% were women. Respondents who were literacy learners were asked what they now know and do regarding the B/APEAE that they did not know or do prior to their literacy classes. The largest response (50%) cited more participation in the board meetings, 40% explained that they now know better how to communicate and to organize documents for the board, and 28% explained that their roles are now clearer. As part of their role, 23% of the respondents noted that they follow both the students and the teachers more closely now, and 18% explained that they work to improve relationships between the school and the community while another 11% work to improve gender equity in school enrollment. Respondents were asked about particular aspects of board meetings that may have been influenced by their participation in literacy classes, and 90% reported that they often attend the meetings. More interesting than simple attendance at the meetings, however, 100% of the respondents reported participating frequently in debates, both expressing ideas and supporting ideas of others. Furthermore, in 38 of the 75 centers, the learner board members reported reading official documents, and in 55 of the 75 centers the learner board members reported taking notes during meetings. Only one center had learner board members that did not feel comfortable in either reading or writing. When asked how else the learner board members use their newly acquired capacity in literacy, 50% of the respondents again noted that they are better able to manage the school and B/APEA administrative documents. 12% of the respondents explained they use their new skills to support a re-opened literacy center, and 11% mentioned that they were more confident in following student and teacher progress. The final question probed what needed to be done to support the complete turning over of the literacy center to the community. Nearly a third of the respondents requested manuals for post-literacy classes, and 20% noted the need for careful management of the current literacy booklets. Almost a quarter of the respondents discussed the need for Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 25 better community mobilization in support of literacy, with 17% specifically recognizing the need for better financial and training support for the facilitators. Community Member Interview Protocol During the evaluation a variety of interested community members were interviewed, including local religious leaders, teachers, and community leaders. A total of 261 community members were interviewed, including 59 women (23%). The community interviews were conducted in a focus group format. Community members were asked what they considered the importance of having a literacy center in their community. A third of the respondents felt that the main purpose was solely to decrease illiteracy in the community. More than a quarter of the respondents felt that literacy centers would help with economic development and social cohesion of the community. More specifically, 22% of the respondents explained that a literacy center is a means of focusing community interest on the school. 11% of the community respondents felt that literacy classes would help change attitudes and norms. When asked to discuss the difficulties that they had encountered in running their community literacy centers, the primary concern expressed by 29% of the respondents was that they did not have a stable site for classes (sites are often shared with formal schools so class times are pinched into openings in the school schedule, or are held in people’s homes which is inconvenient) and that there is a lack of materials (though this was not confirmed in the classroom observations). Nearly a third of the community respondents also explained that poverty is a hindrance to the smooth running of the centers, both supporting facilitators financially and supplying classroom materials for facilitators and learners. Again, learners being discouraged by neighbors or family members who did not understand the importance of literacy was discussed in 17% of the interviews. Nonetheless, all community members expressed their hope to continue running subsequent sessions of the literacy classes. Community members were asked about each of their individual contributions to the existing literacy centers. Almost a third of the community members cited contributions of school supplies or support for the facilitator (money, in-kind contributions, or labor). An additional 27% reported that they work to explain the importance of the literacy center to other people in the community. Almost a quarter of the community respondents explained that they had provided a room for the literacy classes, which is remarkable considering 29% of the same group complained about the lack of a stable site. A final 18% felt that their contribution to the literacy center was simply visiting the class site and being moral support as needed. When the community members were queried regarding changes in attitudes in their communities, they first described women being more willing now to send their children to school. Gender equity was also important, with almost a third of the respondents noting that students were now better supported in their homes with time for their schoolwork. An additional 17% noted that girls were given fewer household chores to allow more time for studying. About 10% of the respondents observed people were better about buying school supplies for their children and also participating in the up￾keep of the schools. Fewer early marriages was noted in 5% of the communities. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 26 Final comments from the community member respondents noted primarily that there was an observed improvement in the relationship between the schools and the communities due to the literacy centers, and that in general things in their communities were improving (better management on a family level, better collaboration between the B/APEAE and the school administration, and people in the community mutually supporting each other). A final request was made by 13% of respondents asked PACEEQ to increase training and financial support for the initial literacy centers. Overview of Field Visits to the two Literacy Centers in Mamou Three visits were made to observe the two functional PACEEQ literacy centers in the Mamou region. The first trip was a disappointment because arriving at the site we found the class schedule had been changed and would meet the following day. Nonetheless we visited a literacy class that was held in a nearby community despite the fact that it was not one of the PACEEQ centers. The literacy class was held in a small room in one of the offices of the company that oversees the damn on the Bafing River near Tolo. There were both men and women learners. The class was repeating the words and letter combinations that the teacher had written on the blackboard. Interestingly, the facilitator for the PACEEQ center is also the facilitator for this center, and she was following the PACEEQ program in this literacy class. She was not being compensated for teaching this literacy class, but was hopeful that the organization that runs the damn would begin to do so. The following day we visited the PACEEQ literacy center in Guelin. Two facilitators were present, working on the very first of the literacy lessons. Again, students were repeating words and letter combinations that the facilitators had written on the blackboard. The facilitators told us that it was a review, but it was clear from looking in student copybooks that the class had not been meeting regularly and was essentially non-functional. The learners explained that a number of people had dropped out because the local Franco-Arab teacher had told the community that literacy classes were not worth following, rather, people should be studying the Koran. The president of the B/APEAE was present and seemed supportive of the literacy class, but the facilitators told us they receive no compensation for the class. They told us that they had been illiterate and after attending literacy classes during the pilot phase, they were now trying to help others—but they did not look satisfied, and their talk had the sound of a rehearsed line. Also, both facilitators live in Tolo, not in Guelin, so they had a seriously long walk back and forth to this literacy center and were not true members of the Guelin community. Although there were 30 people enrolled in this center, only 10 learners were present—and although this is registered as a women’s only literacy center there was a man studying with the women. It was explained that he was included because he was a member of the B/APEAE despite it being a women only center. The third visit was to Siminko. There was one facilitator and her assistant who was a young man, but he did not help with the course while we were there. We observed learners repeating words and phrases that the facilitator had written on the blackboard, then two learners went to the board and copied letters. Like the center visited the previous day, there were 30 people enrolled, but only 10 learners present, and again one of the learners was a man, despite the fact that this center is registered as a women-only center. Furthermore, two of the “learners” present turned out to be daughters of the Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 27 facilitator that were actually enrolled in formal school. Learners explained that they had a high number of drop-outs because people that began the course got discouraged when they realized just how much work it would be. On the other hand, the learners that were there were prideful about their work. One learner that told us that she never misses a class and single-handedly makes meals for the facilitator every time class session. It was also reported that the class had mobilized one day to help in the field of the facilitator, however, again the facilitator lives in Tolo which is a very long walk from Siminko. It was clear from learner copybooks that literacy classes are not held regularly, even though the facilitator happened to be teaching the 12th lesson. It should be noted that only two PACEEQ literacy centers were visited due to time constraints of the evaluation contract and the fact that the initial analysis had not taken place prior to the second visit as planned. However, both visit were expected by the facilitator and learners, not surprises. Ideally more literacy centers would have been included on the itinerary, but it was both interesting and informative to see even two centers in action. The observations made during these visits reinforce much of the information gathered by the evaluation teams during visits to the various sites across all PACEEQ regions. Recommendations Program Relationships It is clear that effort has gone into developing and maintaining relationships with the SNA, the national literacy service. This alliance is vital to the sustainability of the PACEEQ literacy program. Given the strong connection that has already been established, I would recommend that an effort be made to deepen the already strong connection with SNA. Ideally, SNA would sanction graduates with official certificates and graduates would be eligible for continued educational opportunities in other areas. Such a connection may also create the possibility of expanding the use of PACEEQ literacy materials into government sponsored classes beyond scope of the PACEEQ. There are several other organizations that are currently working in the domain of literacy in Guinea. An incomplete list includes: Projet Gestion Resource Naturale (PGRN) which is USAID sponsored, Education Pour Tous (EPT) in partnership with SNA and the UN, GTZ, etc. It would be prudent to have a sense of what the other programs have done and are doing in an effort to profit from their experience as well as share the PACEEQ program and expand the PACEEQ impact. These relationships may also be helpful in gathering post literacy materials to distribute under the post literacy program that is currently being developed. One of the other strengths of the PACEEQ literacy program is that the literacy centers are not imposed on communities. Rather, communities identify a need and include it in their Education Action Plans, at which point the process of social negotiation is begun with community awareness raising and organizing people and systems to be leveraged “I am old, but now I can now follow the progress of the young people. I never miss a class,” another learner reported “now I can follow my children at their school; I look in their copybooks and see if they have been writing in them, so I know that they are actually going to school.” Learner in Guelin Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 28 in support of a community literacy center. When an agreement is reached, PACEEQ offers classroom materials and training support. This social negotiation process is a sound strategy from a development philosophy standpoint, and the criteria that PACEEQ have developed to guide the process are a solid step toward institutionalizing this process. However, both data and observations during the visits indicate that the social negotiation process does not go as far as is needed to truly establish the centers in the communities. There are two areas that indicate the social negotiation process should be deepened. The first is a poor understanding or willingness to devote community resources to the literacy center. The second is a negative attitude in communities that influences learners against attending classes. These two areas were identified in interviews when discussing reasons behind absenteeism and drop-outs. The lack of community ownership of the centers was evident when the varied groups interviewed all requested better “motivation” (money) and/or support for the facilitators and centers. Requesting outside support rather than organizing a community level solution to difficulties was wide-spread across literacy centers. Furthermore, there is a clear problem with the classroom areas that literacy classes are held. Many communities asked for support for a space devoted solely to a literacy center rather than borrowed from a community member’s home or a school classroom, thus subject to other uses and needs. A lack of adequate support was also indicated when people explained that drop-out issues were due to “poor management” of the centers. Some communities and facilitators seemed to believe that once a center had been established, more project support may be forthcoming although this was never the case. A more thorough social negotiation process may be able to work toward avoiding all these misunderstandings, and is certainly needed to clarify the roles of the project vs. the roles of the community in the centers. The issue that the communities at large are not convinced of the value of a literacy center is also an issue with implications in the social negotiation process. Many respondents explained that people had dropped out because of negative social pressures from neighbors and family members who were not convinced of the importance of local language literacy. The social negotiation likely includes those people that associated with the formal school and the APEAE, but it can be concluded that it does not include a wide enough segment of the population, a wider range of support is necessary. Again, the social negotiation process that is in place should be deepened prior to establishment of a literacy center, to include more of the community before the center and individual learners are undermined by a lack of support. Statistics The statistics that are kept centrally by PACEEQ on the literacy program are up-dated only for the trimester reports. Centers are opening, or re-opening while others are finishing up and being evaluated, so keeping up-to date information is tricky. It is, nonetheless, important. There is not an organized data system for the literacy component at the central PACEEQ level. Even simple enrollment statistics needed for this report were drawn from the last trimester report which was completed in June2004, data were over 5 months old. Such important program data should be easily available and timely as well as dependable and consistent—a system needs to be put in place. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 29 The system of collection and storing field data is complicated by the vast quantity of administrative documentation that the centers are required to keep and send back to the central level—not all of it well understood. Administrative documentation at the centers should be reviewed and unnecessary pieces should be remove, important pieces should be consolidated (for example enrollment, attendance, and test information could all be stored in one copybook, inventory lists and borrowing/return records could all be kept in one copybook) This would alleviate some of the facilitator and supervisor workload, and would facilitate the efficient collect and organization of needed data. Nonetheless, the PACEEQ program has the assistance of a highly capable information technology management team which is available and helpful. Once the information needs are made clear, the IT team is usually able to find data and put them in useable formats. I strongly recommend that the literacy component coordinate with the IT team to decide what information they need, and re-organize their data collection forms accordingly. Useful data should be available to inform decisions and future program directions. Instructional materials: Initial Literacy and Numeracy Booklets The initial literacy booklets are logically organized and cover discussion topics that are important for improvements in school quality and school equity. The literacy portions of the lessons are well laid out, and the facilitator guide is directly associated with the learner booklets and thus user-friendly. The drawings are, however, poor. Although there is a certain cost involved in additional technical resources such as artists, there is great value in the visual aspects of literacy books. These books are designed for people with low literacy levels, so pictures/visual aspects of the books will be the first and perhaps strongest connection with the learners. More attractive pictures would also serve as publicity pieces for the literacy centers sparking interest and pride in the literacy classes. Beyond the literacy classes, secondary audiences in the communities would see the pictures and become interested in the topics. Furthermore, better pictures could be adapted into posters and other didactic material. I recommend that a local artist be found to improve the pictures in the existing books. Post Literacy Materials: One strength of the PACEEQ literacy program is that it has been responsive to needs expressed or identified on a community level. Communities have requested not only re￾opening current centers, but also supplementary post-literacy courses and additional reading materials for the graduates of the program. The environments where the literacy classes are organized have a scarcity of reading materials in any language, but most particularly in the local languages, so maintaining literacy skills is difficult and worth attending to. In response to this community initiated request, a post-literacy program would be a strong addition to the current PACEEQ literacy. Prior to the development of the post literacy materials, however, serious thought should be given to a post-literacy program strategy. Will post-literacy booklets be used in a classroom format? A community reading room? A lending library? The use of the post￾literacy materials should be decided prior to more effort being put into the designs for Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 30 the booklets, since the intended method of use impacts the design of materials. A training program and facilitator manuals will be necessary to ensure appropriate use and care for the materials on a community level. The post literacy booklets should be interesting to the exact target group that recently completed the first literacy book. Topics should be new and written in lively and varying forms. Creative and new topics would inspire people to pick the booklets, rehashing old topic would be boring. Furthermore, a variety of text forms should be incorporated. Literacy skills are reinforced by reading different sorts of texts—ranging from letters and official memos to traditional tales and theater skits. Most importantly, post literacy booklets should enhance literacy skills while keeping interest high. The booklets currently under development should be reconsidered and reworked to develop new topics and to use varied, attractive, and new formats. Examples of post-literacy pamphlets could include the following: narratives about APEAE members opening a bank account (and including copies of the forms that Mamadou and Bintou had to complete at the bank), descriptions of situations where documents managed by the APEAE were developed or changed, include brief legal documents that explain APEAE functioning; explanations of exam results from a class with a discussion of options and ideas derived from the results; a narrative of traditional stories that have morals or a proverb that could be applied to school quality (“two hands are better than one,” “little by little a bird makes her nest,” etc.) In short, the format should be both lively and varied. To keep interest, content messages should be fresh and new while still reinforcing PACEEQ project focus. Training materials: A sound and detailed training program has been developed by the PACEEQ literacy component. It is logically organized, thorough and methodologically active. The training design is clearly described and would be easy for a trainer to follow, which ensures a level of quality over repeated iterations of the training as new facilitators are invited into the PACEEQ literacy program. It is clear that the literacy program has been reflective about the training plan as evidenced by the move to shorten the training from 13 days spread across three weeks to nine days, admittedly busy, days spread across two weeks. Although this training program is a strong and important aspect of the literacy program it is a great deal of information for people to absorb in a short period of time. The training program would be strengthened by the addition of a program of single day “refresher” trainings, held monthly in regional centers. Refresher trainings could reinforce the active teaching methodologies introduced in the central trainings, and develop a team spirit among regional groups of facilitators. These refreshers would be low cost since they would be held regionally, and run by NGO literacy specialists, but would provide strong, on-going impact on teaching quality. Another area of training that would strengthen the program would be to develop and train facilitators to run a “post-test review course” for learners who only achieved the level of “semi-literate.” This review course could be based on the initial booklet, and run in a guided peer-teaching manner since all the learners would have completed the course already. A post-test review course would allow learners to review the entire Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 31 program relatively quickly and have a second opportunity to sit for the final test without having to re-take the entire initial course. Such a program would increase the number of learners that achieve literacy, and would also support all willing learners with the opportunities they need for success. Monitoring, Evaluation and Support Materials The pre- middle- and post- learner assessments are very well organized. The tests are well designed with a logical sequence of indicative exercises to complete. The directions for proctoring the tests are thorough and clear. With a minimal amount of training, the tests could be applied in a relatively standard way across the project zones. The student assessment tests are a sound basis for determining literacy levels upon enrollment in the courses, in the middle, and upon completion of the course. The formative evaluations are an interesting idea, with practical implication on the teaching practice in literacy centers. The formative process is done verbally so low level literate learners are not required to write complex sentences. However, the forms are too long—oddly enough they are longer than the evaluation form used at the end of the nine-day facilitator training. The many questions on the form take a substantial portion of class time to complete and also make for a complex analysis. Also, since each of the evaluation forms is two pages long, and there are evaluation forms for both literacy and numeracy, one for self-reflection and one for learner responses, a total of eight sheets of evaluation are used for each class meeting. Photo-copying these forms in remote communities is expensive and logistically difficult. Furthermore, many of the facilitators are minimally trained. The idea of reflective practice is complicated. Formative evaluation is given less than 3 hours of attention during the nine-day facilitator training, mostly on how to apply the forms. The real value of the forms lies in the analysis of the content, which is the most complicated part, and emphasized during in the training. Reflective practice could be more effective if the classroom forms were replaced with a model of clinical supervision that could be guided by the NGO literacy specialists who receive training at a central level, and work closely with regional literacy facilitators. A model of clinical supervision would devote training time and efforts toward developing the capacity of the regional NGO literacy specialists, which is logical as the program expands. The NGO literacy specialists collaborate to improve teaching quality, promote reflective classroom practice, and creating a “learning community.” Regular clinical supervision and observations of the literacy centers could positively impact classroom practice and encourage self-reflection and also inform monthly refresher trainings. Information on the clinical observation/supervision model is available though World Education Boston. An overview can be found in: Oliva, Peter & Pawlas, George. Supervision for Today's Schools, 5th Ed. New York: Longman, 1997. Other information is available on the web through schools of education. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 32 Conclusion: In Response to the Key Questions 1. How does the PACEEQ literacy component strengthen a community’s ability to improve quality and equity schools and community? How does literacy add value to the achievement of the overall goal of the project? In conclusion, the PACEEQ project gains significant ‘added value’ from the literacy program. The impact of the literacy component is identifiable in two primary areas. First there is an impact on the school management through a more literate, more confident B/APEAE. The second area of impact is seen in the attitudes of literacy learners toward enrollment and support of their children (and especially girls) in primary schools. Learners displayed better understandings of issues that are contained in “schools of quality,” and also a better grasp of actions that can be taken in the home to support school success of children both as a B/APEAE member and as a parent. A full 70% of the members of the B/APEAE, in the communities visited for this evaluation, were enrolled in literacy classes, so the literacy classes have a major impact on the functioning of the boards. Respondents explained that because of the literacy topics, they better understand their roles and responsibilities as part of the APEAE. After becoming literacy learners, B/APEAE members also reported attending meetings more often and participating in debates more regularly, as well as using both reading and writing skills for documentation of APEAE meetings and activities. The PACEEQ literacy centers’ strongly positive impact is evident in the increased activity of board members both in meetings and with and in support of the schools which indicates, among other things, an increased self-confidence and comfort level in the schools. This self confidence was also demonstrated when board members report spending more time now talking with other parents about the importance of enrolling children in school and ideas for following up the work and the attendance of both students and teachers. The focus of the literacy topics on “a school of quality” encourages communities to discuss aspects of formal schools that may not have been understood prior to classes. Literacy learners understand that they actually can affect their children’s school success. Learners have become better advocates for education quality, 49% of the respondents indicated that they are now more active in the management of the school, indicating at the very least an increase in comfort level participating in school activities and an improved relationship with the school administration. Evaluation data indicate that 35% of the learners are more apt to buy school materials for their children after understanding the importance of pens and copybooks from their own studies, 25% of the learners now follow the academic progress of their children and check their copy books, and 11% report they are more vigilant about attendance than they were in the past. In special support of girls, 9% described how they now re-organize family chores in more equitable ways so all children have time to devote to studiesThis evaluation demonstrates that the literacy program adds value to PACEEQ through helping to change attitudes within households in favor of supporting school quality, and better school-community relations.The PACEEQ literacy program is a strong factor in developing a stronger school-home bond and better home-based support for student achievement. Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 33 2. What are the factors leading to lack of retention in literacy classes? Are there factors that increase/decrease drop-out rates? One of the major concerns of the PACEEQ literacy component is the drop-out rate among enrolled learners. It should be noted that the drop-out rate (from the three regions where drop-out data was available) was, on average, 20% of the original enrollment. When questioned about the reasons that learners decide to quit the literacy classes the most common responses were illness and social obligations. During the evaluation, interestingly, when facilitators were asked why people were dropping out 35% explained that it was due to a poor understanding of the importance of literacy classes among learners and among the community at large. Another 15% of the facilitators noted that people were dropping out because the centers were poorly run. It is reasonable to conclude that if there was a stronger social negotiation process among a wider segment of the community, more people would have a better understanding of the importance of literacy, and be more willing to support learners and maintain a literacy center in better condition. Drop-out statistics, from the PACEEQ literacy centers demonstrate that on average, 70% of the tested learners are attaining literacy which is slightly lower than the target indicator of 75% of the learners tested attaining literacy. When questioned about absenteeism, beyond illness and social obligations which were the most common responses, people spoke about the problem of missing classes due to work needs, either in fields, in homes, or for commerce. Being in class is important to achieving literacy, and the PACEEQ literacy program has instituted a policy of flexible scheduling so literacy classes can be organized any time of any day to meet the needs of the greatest number of learners. This sort of flexible and leaner-sensitive policy could help learners continue to study while fulfilling their family obligations. Teaching quality is also considered one of the factors in absenteeism and drop outs since poorly run classes are less inspirational, content seems difficult, and time spent in classes seems a waste. The facilitators under the PACEEQ project are generally only minimally trained, and the methodologies observed in the three classes visited indicate that the active pedagogy is neither well understood nor well mastered. In order to improve the classroom practice, short, monthly, “refresher training” courses for facilitators should be organized on the regional level. Improving classroom practice may inspire learners to attend classes more frequently and achieve better results. 3. What are the barriers to students achieving their targeted literacy levels? Are there factors that could be better supported in order to augment the achievement of literacy learners? Achievement issues have been covered in responses to the previous key questions, but in short, additional training support for facilitators through a refresher training strategy will improve facilitator skill levels and enhance learning. Moving from the model of formative evaluation to a model of clinical supervision will increase the classroom support improved facilitator practice and create a “learning community.” A strengthened process of social negotiation will enhance the community support for the Internal Evaluation of PACEEQ Literacy Centers/August-November 2004 Page 34 individual learners as well as for the literacy centers. The development of a “post-test review program” for those learners that are close to achieving literacy but need extra support to pass the final tests will encourage willing learners to continue working rather than to abandon their studies. A post literacy program will offer a further incentive to attain the level of “literate.” And finally, better statistics will show the situation more accurately across the PACEEQ regions. In summary, the PACEEQ literacy component is progressing well. The program has developed a comprehensive process of establishing literacy centers in the communities with support from community members and the local B/APEAE. An initial literacy booklet, an associated facilitator guide, and training program have been designed in the four major languages of Guinea. A process of social negotiation to be employed in communities considering opening a literacy center has been outlined. And a total of 260 centers have been opened to date which exceeds the project target of 225 by 15%--with still other centers poised to open soon. The project has been sensitive to needs identified on the village level, and has re-opened centers to accommodate large numbers of learners, organized literacy centers for women only so the women can learn in more comfortable environments, and a post-literacy program is being designed. The impact of these centers is clear. Beyond the simple but well-known theory that parents with ANY sort of educational experience are better able to support the education of their own children, the evaluation has documented that learners are more active participants in the B/APEAE meetings, participating more frequently in debates, and using their reading and writing skills for the record keeping, correspondences, and accounts, as well as being more active in school activities and enhancing collaboration between the school and community. Learners have also become advocates in the community for enrolling children in school, being proponents of gender equity in the schools, and better supporting student achievement in their homes. The literacy centers are well on the way to enhancing PACEEQ project goals. Recommendations for future research include the following ideas: Document the result of increased literacy in a multiple case-study format by following BAPEA members in various communities over time (from illiteracy, though the PACEEQ literacy classes, to post-literacy) and document changes in levels of activity on B/APEAE work, levels of participation in the BAPEAE and other community initiatives, and any changes in their homes to support school success of their children. A similar sort of research could also be done from the point of the children enrolled in the formal schools. A group of students could be followed throughout a year (or if possible longer) to examine their attendance, participation, success, and attitudes toward school and the ways that their parents support changes as their parents move through a literacy course. It would also be interesting to conduct a study that uses a control group of communities that have B/APEAE activity supported by PACEEQ, but have not organized literacy centers in their communities in comparison with similar communities that have organized literacy centers. This would be a way to identify impact that could be attributed to the literacy center as opposed to attributed to other project interventions.