Zambian Ministry of Education: Education Management Information Systems (EMIS) And Related Activities External Program Review Report Glory J.T. Makwati Mpongo R. Malyenkuku Tegegn Nuresu Wako 10 April 2004 Contract GS-10F-0112J Task Order No. 690-M-00-02-00001-00 EMIS and Related Activities American Institutes for Research USAID/Zambia SO2: Improved quality of basic education for more school-aged children IR 2.3: Improved information for education decision making processes This document was created using Adobe Acrobat 6.0 Professional Submitted by: American Institutes for Research 1000 Thomas Jefferson St. NW Washington, DC 20007 Phone: 202.403.5000/Fax: 202.403.5979 www.air.org Zambian EMIS and Related Activities External Review Zambian Ministry of Education Education Management Information Systems (EMIS) And Related Activities External Program Review Report Mpenzeni Park Middle Basic School, Chipata, Zambia The Evaluation Team: Mr. Glory J.T. Makwati, ADEA Working Group on Education Statistics - Team Leader Ms. Mpongo R. Malyenkuku, Education Consultant Mr. Tegegn Nuresu Wako, ADEA Working Group on Education Statistics Report commissioned by: American Institutes for Research (AIR) Submitted to: USAID Zambia and Ministry of Education Zambia 2 ADEA- WGES/NESIS TABLE OF CONTENTS Page 1. Acronyms 1 2. Executive Summary 2 3. Background 5 4. Methodology 6 5. Limitations 7 6. Objectives 8 7. Achievements 8 8. Task 1: EMIS 9 9. Task 2: Capacity Building 22 10. Task 3: Assessment 24 11. Task 4: Community Data Collection and Use 25 12. Suggested Future Direction 25 13. Sustainability 27 14. Conclusion And Summary Of Findings 29 15. Recommendations 30 16. List of Interviewees 34 17. References 35 18. Appendix 1. EMIS and Related Activities Equipment distribution tables 2. EMIS External Evaluation Terms of Reference 3. EMIS Training Break-down and Programme 36 Zambian EMIS and Related Activities External Review Acronyms AIR American Institutes for Research ASC BESSIP Annual School Census Basic Education Sub-Sector Investment Program CHANGES Community Supporting HIV/AIDS, Nutrition, Health, Gender and Equity education in Schools CSO Central Statistical Office EBS Educational Broadcasting Services ECZ Examinations Council of Zambia ED*ASSIST Education Automated Statistical Information System Toolkit EMIS Education Management Information Systems IRI Interactive Radio Instruction MOE SADC Ministry of Education Southern African Development Community NESIS National Educational Statistical Information Systems SPSS Statistical Package for Social Sciences USAID United States Agency for International Development WGES Working Group on Education Statistics ADEA- WGES/NESIS 1 Zambian EMIS and Related Activities External Review Executive Summary Background The Education Management Information Systems (EMIS) unit within Zambia’s Ministry of Education (MOE) was working well in the early 1990’s when it participated in a pilot study on school records management. Together with other sub-Saharan countries, coordinated by the Working Group on Education Statistics (WGES) and the National Statistical Information Systems (NESIS), Zambia took school records management as its priority needs. This was a good beginning towards strengthening the management of school records in an effort to achieve data quality that comes from schools. However, this effort did not take root as expected. This problem, coupled with professionals who left the ministry, weakened the EMIS, which eventually died. The system for data capturing then was the Integrated Microcomputers Processing System (IMPS). This program was used as a data-capturing tool. However, the IMPS became outdated because of lack of maintenance and ownership. There was no regular updating of the program. The Zambian EMIS began to deteriorate for various reasons and finally everything came to a standstill. It was revived as a result of an agreement between the Ministry of Education (MOE) of Zambia and the United States Agency for International Development (USAID). Financial assistance was provided to the MOE to collect, organize, and use educational information for the ministry. USAID contracted the American Institutes for Research (AIR) to execute the pilot project. AIR further sub contracted the Academy for Education Development (AED) to implement the project. Objectives The objective of this assessment is to follow up on the implementation process of the Zambian EMIS as funded by USAID and implemented by AIR and the Academy for Educational Development (AED) in collaboration with the MOE. The objective of Zambian EMIS, as stated in the statement of work, is to “rebuild and strengthen the capacity of the ministry to collect, organize, store, share and disseminate educational information for planning and management at all levels.” 2 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review Achievements The project has achieved a great deal in terms of equipment procurement and installation, systems development, capacity building, networking, and rendering technical assistance to other departments especially the Examinations Council of Zambia (ECZ), Education Broadcasting Services (EBS) in connection with the Interactive Radio Instruction (IRI) program, and other activities related to Education Management Information Systems (EMIS). A summary of these achievements is given below: EMIS: The data-collection instrument was revised and distributed to schools and data collected for two years during the project implementation, The Education Automated Statistical Information Systems Toolkit (Ed*ASSIST) program was used to capture and retrieve data. Completed data for 2003 was copied to CD and distributed to provinces, districts and schools. From the CDs, standard reports can be produced. Historical data were imported from previous collections for the other two previous years 2000-01, making four years data available to users. The departments in the MOE’s headquarters are connected to Internet although the original plan anticipated only email services. Capacity building: A series of training activities were conducted for headquarters, two pilot provinces and nineteen districts in computer usage and literacy. More advanced training was given for EMIS staff on application packages including Access for a duration of one month, in South Africa. Short-term training was also given to a network technician from outside the ministry. Instructional modules were also produced to supplement other program training materials. Assessment: There is a need to enhance the capacity within the Examinations Council of Zambia (ECZ) in the development and implementation of competency tests. The grade 4 competency tests were developed, administered and a draft report produced. The objective of this exercise was to monitor learning achievement and learning processes at Interactive Radio Instruction (IRI) centers. The exercise was initiated in collaboration with the Educational Broadcasting services (EBS). . ADEA- WGES/NESIS 3 Zambian EMIS and Related Activities External Review Community Data Collection and Use: The objective of the community data collection and use component of the project is to assess information needs of the communities and establish factors that contribute to the quality of education. An attempt was made to identify the information needs that can be incorporated into the EMIS data collection process. The pilot project commissioned a study to identify possible areas of cooperation between community data collection services and EMIS. The study was completed and a report produced. Gaps The projection and simulation model outlined in the plan came up late in the plan. This activity has not yet been done. This is a program to assist planners to do projections and simulations and come up with alternative options and scenarios for planning and policy development. It is highly dependant on the availability of data from EMIS. Therefore, it can only be done efficiently when source data from EMIS is ready. One option for the projection and simulation model is to use the four-year historical data that is currently stored in ED*ASSIST. The second option is to develop the projection and simulation model further, if the project is to be taken to scale and more data is made available Off shoots These are activities that were done although they were not in the original plan of the project: Departments and sections of the MOE at headquarters, are now connected to the Internet. Initially this activity was planned to give email services only. The connection to Internet was an additional initiative by program implementers. A sub-regional international conference was conducted as part of Technical Assistance to the Examinations Council of Zambia (ECZ). Conclusion A lot of effort has been put in order to achieve the objective of enabling the MOE to collect and process information on education. A single annual data collection instrument has been produced. It will be administered in 2004. Previously there were separate data collection instruments for different levels of educational institutions, i.e. basic, secondary and community schools. The data capturing and retrieval computer program, ED*ASSIST, was adopted for entering, processing and producing reports. This program has made it possible to produce timely compiled reports and basic indicators of the education systems’ performance. These data 4 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review have been produced and distributed on CDs to MOE headquarters, provincial and district offices and external users. It should be noted however that the ED*ASSIST is a very versatile package that is capable of generating a large number of reports reports. It has great pontential and it is easy to use. While these efforts are remarkable in terms of reviving a system that had almost died, much more effort in terms of training and systems maintenance is required to bring the system up to the level of efficiency in relation to increasing and institutionalizing the use of information for decision-making purposes. The project has succeeded in laying a sound foundation for the future development of EMIS. This is a challenge for the MOE and cooperating partners. Background Historically the government of Zambia has committed itself to national development through education. Education is regarded as a potent tool for socioe-conomic development and social transformation. Although education in Zambia is the responsibility of government, the role of other actors, such as communities, religious organizations, nongovernmental organizations, and cooperating partners is recognized and encouraged. On its part the Government of Zambia is committed to “protect the rights of individuals, promote social well being and achieve a good quality of life for every person.” (Educating Our Future, National Policy on Education, p2 1996). In an attempt to achieve its goals, the Government of Zambia launched the Basic Education Sub-Sector Investment Program (BESSIP) in 1999. This program was mainly funded by the government, USAID, and other co-operating partners. In line with world trends and declarations, such as the 1990 Jomtien Declaration on Education For All (EFA) and the Dakar World Conference on Education for All (WCEFA) in 2000, Zambia has set broad strategic objectives to realize its vision of providing education for all its citizens by 2015, in accordance with the extended EFA decade goals and the UNESCO Millennium Development Goals (MDG). Some of the goals are: ADEA- WGES/NESIS 5 Zambian EMIS and Related Activities External Review 6 ADEA- WGES/NESIS • To increase enrolment rates and improve learning achievement at primary school level • To improve the quality of basic education for more school-aged children This means “improved access, retention and achievement.” Improved achievement on the other hand meant more pupils learning relevant, basic skills and competences. In order to achieve the above goals it became imperative to develop a strong Statistical Information System (SIS) as a basis for the enhancement of sound educational planning and decision-making. The need for timely, accurate, regular and user-friendly statistical information is a prerequisite to ensure the achievement of the above goals. Through various contractual arrangements/agreements the Government of Zambia and USAID decided to invest in the development of an Education Management Information System (EMIS) as a basis for the achievement of agreed strategic plans. The project was to start on a pilot scale involving Eastern and Southern provinces. Although there were decided advantages in taking all provinces together to minimize disparities, the pilot project was implemented in the two provinces indicated above in order to provide experience before taking the project to scale. The pilot project has been running for two years, and it will end in March 2004. Assessment The objectives of this assessment is to look through the activities as planned and see how successfully the project was implemented by describing the level of achievements, shortfalls, and suggest a way forward for future improvements towards the achievement of overall goals of the project. Methodology In this assessment an attempt is made to understand and document to what extent the on￾going revival project of EMIS is being implemented by stakeholders: the MOE and AIR, taking into account the progress made and problems encountered. No attempt will be made to Zambian EMIS and Related Activities External Review assign responsibility for the execution of any activity. General observations will be raised indicating challenges for project /program development and implementation. The methodology used in this assessment was 1. A review of documents: plan documents, reports, memos and comments made on specific issues on areas related to the project. 2. Meeting implementers: Permanent secretary (MOE), Chief of Party (AIR), Director of the Directorate of Planning and Information, MOE EMIS team, ECZ, Siavonga and Chipata district offices, and Chipata provincial offices. The assessment team also visited Kabwe district in Central province and Lusaka district offices in Lusaka although the two were not in the pilot project. 3. Using observation methods by looking at the printouts of educational statistics and asking the statisticians to demonstrate how they actually used the software they have acquired from the headquarters. 4. Holding meetings with other stakeholders, e.g. Education Broadcasting Services (EBS), USAID, etc. In the group discussion guiding questions were used to enhance the discussion process during all the meetings. Limitations The assessment team had more time and exposure with a wide range of stakeholders. However the terms of reference indicated that the MOE was not to be the subject of the assessment. This made the assessment task more complex because the pilot project was implemented in conjunction with the MOE. Since the project has laid the initial foundation for development, the MOE has to take up certain responsibilities in relation to future project development. This situation has led to the making of general, instead of specific recommendations related to the future of the project, in some cases. The assessment team believes that the future of the project is subject for negotiation between the MOE and the supporting agency(ies). In the light of this it is difficult to apportion responsibility for certain issues. ADEA- WGES/NESIS 7 Zambian EMIS and Related Activities External Review Objectives The objective of the project as stated in the Terms of Reference (TOR) is summarized under the following four points. 1. To provide technical assistance to the MOE’s EMIS unit to develop, pilot, improve and implement an education management information system at the national level and selected districts in Southern and Eastern provinces and to develop and implement a plan enabling the MOE to collect analyze, manage and use timely educational information to make education decisions at all levels of the education system, including headquarters, provincial, district, school and community levels throughout Zambia. 2. To improve the MOE’s and others’ capacities to collect, share, analyze, disseminate, and use information at all levels of the educational system to make informed educational decisions. 3. To collaborate with MOE’s headquarters, provincial, and district education offices, and relevant partners working in Southern and Eastern Provinces to develop and/or refine community-based models for collecting and utilizing MOE and/or community schools data so that such data are harmonized with and, where indicated, included in the EMIS. 4. To “collaborate closely with the Examinations Council of Zambia and relevant units of the MOE in developing and implementing a training plan that will enhance the capacity of Examinations Council personnel, in competency test development, implementation, and use.” Achievements In this section of the report a presentation of the activities of the project is made, based on the following: group discussions with concerned parties, observations from school visits, as well as visits to districts and provincial offices. A summary of observations is recorded below in the order they were documented in the scope of work. 8 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review EMIS EMIS is a system within the MOE responsible for educational information with the vision of achieving informed decision-making and planning. Thus, by collecting, organizing, processing, analyzing, publishing, disseminating educational data, and collecting feedback information, the MOE seeks to reinforce the enhancement of educational development. The enhancement of education can be achieved through the clear formulation of the vision, correct understanding of the environment, efficient use of feedback, correct diagnostic analysis of the current system – outlining, detailing and identifying the current problems and efficient planning and correct incorporation of a monitoring mechanism in the planning and implementation of the system. Comprehension of this vision implies correct understanding of the system (the whole) and its components (the parts): collection, processing, analysis, publication, dissemination and efficient use of the feedback information. On the one hand the whole without the parts is an empty box, while the strength of parts makes the whole even stronger. Any shortcomings in the whole or part of the components weaken the system and may lead to the death of the system. That is to say if the quality from the data collection instrument is taken away, the quality of processing, and analyzing is also lost. The EMIS implementation process started with a needs assessment by the project in conjunction with the MOE. The Chief of Party (COP) representing AIR was responsible for day to day project activities and he worked and continues to work closely with the Planning and Information Directorate of the MOE. The project was divided into four primary tasks: Task 1: Education Management Information System (EMIS) The assessment team learned that the EMIS needs assessment was done when the project started. Accordingly computers were procured (hardware, software, peripherals). These included Personal Computers (PC’s), servers, printers, projector(s), computer tables, fax machines etc. (Please see appendix 1 for more details). The project team was very focused in the execution of the above tasks. The training program was also well organized and all related activities were completed on schedule. The distribution of equipment was accompanied by ADEA- WGES/NESIS 9 Zambian EMIS and Related Activities External Review distribution tables and inventory forms and related backup materials to ensure the safety and accountability for the equipment. It is advantageous for the ministry to transfer historical data from the old IMPS system to ED*ASSIST. This will enable the ministry and other users to benefit from time-series analysis that is often required to measure progress. The ministry has also revived the data-collection system, through this project, by designing a single data collection instrument, the Annual Schools Census (ASC), which is used for collecting data from schools. Accordingly, school data for the previous two years (2001 and 2002) were collected processed and analyzed. This was another leap forward in terms of availing information to users. The different offices of the ministry are connected to Internet and currently use the facility. The necessary hardware and software required are in place through this project. This made communication easy and useful between different departments and the rest of the world. Data collection Data collection is done through the Annual School Census (ASC), which is designed at the MOE’s headquarters and distributed to schools through provincial and district offices. The return rate was appreciated at the time of the EMIS mid-term review in 2003, as almost all schools responded on time in submitting the 2002 questionnaire. Return rates of 96% for 2001 and 98% for 2002 respectively, were recorded. The 2003 return rates were however lower at 93% for primary schools and 89% for secondary schools. This lower return rate was in spite of the training activities conducted at district and school levels, related to procedures on completing the ASC and the importance of timely returns. The causes of this lower than expected situation needs to be looked into with a view to improving the overall quality of statistical information. The use of the Questionnaire Tracking System (QTS), which is part of the ED*ASSIST system can be used to follow up on defaulting schools. In 2003 the ASC was revised for the collection of data for year 2004. This revised ASC seeks to collect core data on enrolment, teachers and schools including community schools and IRI centers. Data on these items can therefore be collected through pilot studies or surveys 10 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review conducted at intervals of three to five years. Training in completing the ASC for 2004 has already been done. However schools have not yet submitted the returns as these have to be completed on 31st March, 2004. In discussions with some interviewees, they felt that the ASC was still big and vague in some sections. They were advised to continue making suggestions for improvement because the process of improving the structure and content of the ASC was dynamic rather than static. The users of the ASC felt that its structure should be modified so that instructions appeared on the same page or immediately below the sections they referred to. More training, especially at school level, is necessary to further improve the correctness of information in the ASC. In this regard the team learned that through training given to district and school representatives, comments were collected from schools. This is a job well done. It is through such comments and discussions that the ASC can be improved from time to time. The standard instrument in terms of content and size will take shape through continuous feedback from the users and other stakeholders. The possibility of different interpretations of the items in the instrument should be eliminated through discussions and debates on the content and layout of the ASC. In the assessment it was learned that there is a possibility, perhaps for the benefit of the school, that some head teachers tend to inflate or deflate data. This may be so, because of: 1. Low level of awareness about the use, importance and value of accurate information. 2. Lack of feedback information to schools on how the data are used. This problem can be tackled through training of teachers and head teachers on the use of information for planning and monitoring purposes at school, district and provincial levels. The other method that may be used to improve the quality and accuracy of data is to conduct sample surveys that allow a look back to the ASC sent to centers by checking through the copy that remained at school. A few sample schools can be taken and checked for correctness as a move towards improving the quality of data at school level. The use of time-series data – comparing data over time – can also improve the quality of data. This can be done by ADEA- WGES/NESIS 11 Zambian EMIS and Related Activities External Review increasing the capacity of analysts at all levels so that they are able to do the analysis and compare data over time. Although the current ASC is comprehensive in that it collects data for IRI centers as well, it does not collect data for the Non-Formal Education (NFE) sector. Basic data such as number of providers by type and number of learners by sex etc. can easily be collected and analyzed with the rest of the EMIS data. Information on these sectors will help the MOE to assess progress in the overall development of basic education in general and Education for All (EFA) in particular. It should however, be noted that everything needed cannot be collected through the ASC. The regular ASC has to focus on the core data elements such as schools, teachers, enrolments etc. The rest of the data needed may be collected from secondary sources and through sample surveys. EMIS can take part in such sample surveys and pilot studies and collaborate with researchers to obtain information to fill the gaps. Information on examinations data can be obtained from the ECZ. The ECZ and the MOE need to coordinate their data systems so that they can share information easily. Discussions should be held between the MOE EMIS team and ECZ to work out strategies for collaboration. The Human Resources Directorate of the ministry is already using important information collected on the teachers. This database is used in the restructuring process that is going on within the ministry, as well as for human resources allocation and other administrative issues. Information teacher dynamics such as attrition, transfers, resignations, retirement etc. can be calculated and produced from this database. Schools use different lists and registers as data sources for the ASC. This is inconvenient and it may introduce errors. In ADEA’s earlier studies on school records management, in which Zambia was a participant, it was suggested that a document called the school abstract be made available to schools. The abstract is a summary statistical table, at school level meant for reference, reporting and use of statistics for planning at that level. Schools need to prepare this abstract prior to completing the ASC instrument. The schools can prepare this abstract 12 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review easily by using all teachers, especially class teachers who participate in the preparation process. It should also be underlined that this summary table encourages the use of statistics at school level both for planning, monitoring of activities and providing information to users. Experience has shown that the continued use of these tables can increase the level of information awareness at school level. Recommendations 1. The ASC is still too large to be taken as ‘standard.’ Some of the contents, especially those on static data related to infrastructure, can be tackled by sample surveys while other contents in the instrument can be collected at intervals of 3 or 5 years as appropriate. 2. In addition to the regular statistics of the formal system, provisions have to be made to collect information from other sectors such as higher institutions, Early Childhood Education, Non-Formal Education, and ECZ. 3. Data collection guidelines in the ASC should be placed together with the tables they refer to. Experience has shown that not many people read instructions that are separate from the questions even if they are detachable. 4. In order to improve the rate of returns of the ASC, the MOE should delegate more responsibility to districts on follow up for the ASC. Districts are nearer to schools than provinces and the headquarters. It is easier for districts to follow up on school instrument distribution and returns. 5. The MOE should develop and publish the first school abstract for all schools as a guide. Schools can then develop their own abstracts and use them to complete the ASC and report to higher authorities. Above all the abstract can be used for monitoring, planning and school development in general. 6. The survey administration plan the (Gantt chart) should be made available to all users in the provinces, districts, schools and headquarters. The chart should show the following activities: procurement of printing services for the ASC questionnaires and distribution of the same to schools, time for data collection, submission of completed questionnaires by schools, processing, analysis, publishing, distribution and feedback of information. ADEA- WGES/NESIS 13 Zambian EMIS and Related Activities External Review 7. Static data relating to toilets, classrooms teachers’ houses etc. should be left out in an effort to make the instrument smaller. The items left out do not change annually. Data processing Data processing is one of the core functions of EMIS. Data obtained from schools has to be entered into computers for processing. This process involves the use of computer programs or codes used to prepare entry templates or forms and report formats. The entry requires the use of user-friendly entry templates and user-friendly retrieval formats. The project/ministry has been able to enter data into the computer from the ASC and prepare outputs for users of different kinds. The EMIS unit has also been able to prepare output from the database, burned CD’s with 2000-03 statistical information, and distributed them to provinces and districts. This is good progress towards achieving the set objectives. The project has been able to make information available on time, for the core data elements. However, one cannot say the MOE EMIS unit has the skills, techniques and tricks, to manage and explore the potential of the available database content and quickly respond to the frequent demands from users, especially decision and policy makers. There is still need to develop more skills. EMIS staff should be able to build upon capacity and abilities obtained so far in order to design a database, relate different tables, design forms, design queries, design report formats, connect forms and sub-forms and write computer codes that enable EMIS professionals to coordinate and use the different objects of the database. Then and only then can EMIS professionals be comfortable with the use of all these highly interrelated skills. Abrupt progress is not expected in this short time of project implementation. The basic skills training in computers are just the beginning of a long journey in skills development. The output expected at this stage includes compiled tables and graphs with analytical chapters targeting the general users of the information at the ministry level and beyond. The aim is to make basic information available. Furthermore, the EMIS unit is expected to comfortably manipulate the database and respond to ad-hoc queries as they arrive. These queries are too many and demand in-depth knowledge of the system in order to be able to work around joining tables and making queries. This is the responsibility of the MOE EMIS unit and should be accomplished by local professionals with some assistance from external technical assistance, when necessary. 14 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review The EMIS unit has also come up with summary statistics, disaggregated at school levels, on a CD for internal and external users. These are detailed tables from the main database used to produce sub reports. These are extracts from the completed database of Access software containing statistics on schools, pupils, teachers and facilities. Users need to know the database itself in order to make queries and produce necessary reports that are not included in the options given on the CDs. Those who do not have orientation on the use of the CDs have found it difficult to use it as expected. A short demonstration by the provincial officers, is necessary so that they can orient the district offices on the technicalities of using the CDs. ED*ASSIST is used for data capturing and retrieval. The right of use has since been given to the ministry together with the source code. This is good progress. Although three (3) EMIS unit staff have gone for advanced training in MS Access programming, more staff and additional training should be provided to further enable staff to modify the source code so that the EMIS unit can update the data capturing templates and retrieval of reports as demanded. It is important to remember that both the data-collection instrument and demand from decision and policy makers change from time to time. Hence, it is absolutely necessary for EMIS personnel to be very conversant with the manipulation and use of ED*ASSIST including the modification of the program source code to be able to use it. It is equally important that the EMIS personnel are able to produce the kind of data sets produced on CD. The EMIS unit should produce this kind of data sets on a recurring basis. However, the current arrangement is that most of the data processing work is done by a consultant from AED, which developed ED*ASSIST. When the data capturing module needs updating, it has been sent to the implementing partner’s Washington, D.C. office for modification to the data capturing templates. This is a wedge that is hampering progress. This has a big implication on the sustainability of the program in the future. Apparently there were no arrangements within the project for the AED consultant to be under studied by the local personnel. This is one of the major shortfalls in the program implementation and an impediment for the program as a whole. The transferring of skills to the locals should be done through advanced training for two to three officers in each area of need. It is necessary to train more people in an area because of high attrition rates within the MOE. ADEA- WGES/NESIS 15 Zambian EMIS and Related Activities External Review Computer programs used for data capturing and retrieval purposes need such skills and knowledge to maintain them while they are used and over time. The EMIS unit needs to develop such skills by training its professionals at the Headquarters, provincial, and district offices. The establishment of an in-house-training program should be considered by the MOE as it is a less costly option. Problems can occur any time, anywhere, so any system needs consistent maintenance. There should be someone out there attending to program (software maintenance) as problems occur. It is therefore imperative that the current project or MOE invests in a targeted training program that enables Zambia’s EMIS to stand on its own feet. Recommendation 1. There should be no limitations or proprietary control in any way over the use of ED*ASSIST especially when it is meant for development functions. Specifically the EMIS unit should be able to explore, modify and amend the computer program codes as needed. 2. An in-depth training on Access database and Access programming is a necessity. Training on creating databases, tables, and querying the database, must be given top priority as the whole data is maintained using ACCESS and the training be given on the job while using the actual database for creating, manipulating and retrieval activities. Equally important, more professionals in provinces and districts should be included in such training. 3. It is also a good idea to include output that is specific to the needs of known users, such as UNESCO Institute of Statistics (UIS), CSO, SADC, and institutions in the sub-region. Data analysis Data analysis is another core element in the cycle of EMIS activities. This is required at the decision and policy level. Core indicators of the education systems’ performance need to be made available. This usually involves identifying data for analysis, joining tables, making graphs, calculating indicators and writing narrations of what the indicators are revealing so that users especially decision and policy makers can understand and use. In this regard the EMIS unit has produced selected indicators for users. These include access, efficiency, quality and gender equity. 16 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review However, the yearly abstract or annual statistical bulletin has not yet been produced. The abstract should contain analysis needed by the policy makers. The abstract targets users that do not have computers to utilize the copy of statistics and indicators contained on the CDs. The abstract also has the narration that explains what these indicators mean as relates to policy and decision-makers. They contain outcome of the year or time-series analysis that can be used by policy or decision makers. There is need for the planning unit to compile a report on indicators for planners and policy makers as it relates to the existing demand based on available data for primary users of the information within the MOE. It usually involves EMIS, planners and other relevant sections of the ministry to produce such a vital summary indicators report for use. The target group for such a publication is the top decision makers and planners who are not expected to read through the large abstracts produced on regular basis. The ministry has also collected school coordinates over the last years using Geographical Information Systems (GIS). This is a great leap forward in terms of strengthening the EMIS unit of the ministry. The GIS database is connected to the main school database. This additional information helps not only to produce information on a map but also to locate schools with more precision. It gives, for example, distance between schools, districts and provinces and other locations. It is also possible to demonstrate necessary indicators on a map thus providing key information for planners and decision-makers adding efficiency to planning, implementation, and monitoring processes. In the light of the availability of this GIS information, it is recommended that: 1. Planners and technical EMIS staff be trained on the use of software for GIS (ArcView). 2. Planners are trained on principles of GIS and educational mapping. The use of such information for educational planning development and monitoring cannot be over emphasized. ADEA- WGES/NESIS 17 Zambian EMIS and Related Activities External Review Publication It is said, “If information is not published it is not done.” This is not to undermine the efforts made in collection, processing and analysis but to emphasize the importance of the product reaching the user and making an impact on educational development. The project has assisted the ministry to produce statistics and indicators on schools that it has distributed to provinces and districts for use. This is a great achievement in two years of its inception. While this is significant progress, the use of such information is confined to those who have access to the use of a computer. It is also an encouragement for users to have it available in print form. It is strongly recommend that this be complemented by a print copy, so that the information may have a wider coverage. Indicators on gross enrolment ratios, net enrolment ratios, pupil-teacher ratios and several others were developed by the project and they are now available. It is hoped that more effort will be made in the near future to put together reports and tables for publication. At this stage of revitalization of educational information, and progress towards standardizing the data collection instrument and training personnel who fill out the ASC, it is logical to recommend that the core data outputs and indicators be published to start with, and distributed to districts and provinces for example comparative performance among provinces or districts. Within the district, a good start may be in comparing the performance of schools so that each can see where it stands. The team was encouraged to note that a draft report on publications produced by the EMIS unit is now at an advanced stage. It is being processed to incorporate comments made by the Permanent Secretary. Hopefully this publication will be out soon. It is also useful where possible to publish basic facts, figures and indicators on the web for dissemination to attract more users and gain a wider publicity. 18 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review Distribution If the users have not seen the product, they can not use it, and if they cannot use it they cannot do sound planning and decision-making. This retards educational development and it has a negative impact on overall economic and social development. Distribution is one of the important functions of EMIS that promotes the use of information for development. When the information is published and not distributed to users, the whole process of data collection and processing it is not only a failed job but also a waste of resources and a retardation of development. As stated above, the statistical tables produced by the central EMIS unit have been distributed to provinces and districts, and other internal and external users on Compact Disc (CD). The forthcoming publication will have to be distributed to users too. The distribution of statistical information should include provinces, districts, and schools. However, such analytical reports in the form of publications may be very expensive. In this case, one can consider the option of selecting summary analysis relevant to each unit and distribute them accordingly. Distribution also includes different departments at headquarters, experts, decision support systems, and external users such as national and international organizations and institutions, governmental and nongovernmental organizations. In order to enhance the information distribution process, a list of relevant government and non governmental institutions should be maintained in EMIS to make sure that each one of the institutions receives a copy whenever it is produced. It is important also to make room for receiving feedback from users by inviting them to send comments on the structure and content of the abstract and other publications containing educational information. On the other hand, it is equally important to make an arrangement and receive other publications, from other sources such as government and nongovernmental institutions, that are relevant to the EMIS work. This could be information on economic or social indicators. For example we may want educational performance of a certain district with the level of economic status of that district, the educational level of a certain province with the prevalence of HIV infections etc. Dissemination and Information services Dissemination means not only distribution but also the ‘marketing’ of the information products. Experience has shown that demand for information is low in many African ADEA- WGES/NESIS 19 Zambian EMIS and Related Activities External Review countries. Hence, the effort is to increase demand. Such activities as presentation of information (e.g., to workshop participants), briefings to higher bodies, production of brochures, abstracts and posters can be used to increase demand. In short, efforts should be made to make users aware of the existing information and analytical results at their disposal for use in development activities in their areas of operation and accountability. Information services refer to an organized way of providing services to clients demanding information for use. This basically is useful for increasing the level of demand for information use. Increased demand is one of the success factors in efforts made to strengthen EMIS in the ministry. As part of their dissemination strategy the project and MOE has produced a brochure called “Just ASC” the acronym used to indirectly refer to ‘ask’ asking for information, but directly refers to information produced from the Annual School Census. This is one method of creating awareness. The issue of low demand for information came up during discussions on project implementation in the ministry. How does one increase demand for information? The reality is to invest in quality and produce quality information in order to achieve increased demand. When statistics are correct and of good quality, they will attract more users. The other factor is to market the products. Brochures, abstracts, and indicator reports are only some of these products. Internet connection and communication The different departments of the ministry are connected and use Internet facilities. This was not part of the initial project activity. The connectivity component of the project was realized through collaboration and support from The Danish International Development Agency (DANIDA). Provinces and districts are not yet connected. Related training was also given to one person who can maintain and provide services in this area. Training was given to the senior network administrator from ECZ as part of capacity building. However this person left the MOE shortly after returning from this advanced training. For sustainability of the Internet Communication project, it is important to train additional personnel from within the EMIS unit or any other related section of the ministry 20 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review for effective implementation, maintenance and troubleshooting of the networking activities, especially if and when the program is taken to scale. Capacity building within the MOE will have to continue even after the donors are gone. On the other hand, one person alone cannot guarantee project sustainability. More than one person should be trained in any given area to mitigate the loss of personnel through natural attrition, transfers, resignations etc (two to three people per area would be ideal). Two or three people should be trained in basic maintenance skills to ensure the provision of services in the future. The use of Internet connection and services boosts the level of communication within the ministry, including provinces and districts The way forward is to connect all provinces on to the Internet at least in the short run and gain experience and knowledge that can be useful for later implementation of a similar program at district level. Recommendations 1. Training relating to Internet connectivity should be given to two or three members from the MOE, preferably from the EMIS unit. When an appropriate candidate is not available at the headquarters, recruitment can be made from provincial and district offices. 2. All provinces should be Internet connected so that information can be communicated between headquarters, provinces, districts and within the various departments of the ministry. Data quality It is an important fact to note that data obtained from schools should be correct, timely and usable. Data quality depends, among other things, on the quality, motivation, and determination of both the producers and users. Core users, in this case, are planners and decision-makers in the MOE. Poor quality data does not only mislead planners and decision￾makers but also leads to a waste of resources. Obtaining quality data also depends on the level of awareness of the personnel engaged in data collection and the degree to which school head teachers, teachers and record officers assigned to complete the ASC are ready to shoulder the responsibility. ADEA- WGES/NESIS 21 Zambian EMIS and Related Activities External Review Data quality should be addressed at all levels from school to the MOE headquarters, as it greatly affects the outcome of the information used for planning and decision-making. Possible mechanical errors when inputting data, file handling, aggregation etc. need to be attended to carefully at all stages. Efforts can be made by preparing preliminary analysis and making users aware of the dangers of poor data. Above all, addressing the problem of data quality at school level pays off at the end. Human resources requirements After the ministry’s restructuring, posts have been filled and the EMIS unit has a greater number of personnel than ever before. However, the unit’s head has already left the ministry. This position of management is crucial, especially at this time of revival, in coordinating the work of computing and statistical analysis as well as managing the requirements that emanate from the decision makers. Without proper management, it is difficult to achieve desired goals. The post of a programmer also needs to be seriously looked at now. If the project is taken to scale then the position of programmer is critical. If it is not possible to hire new personnel, it is important to look for potential candidates within the ministry at all levels. Task 2: Capacity Building Capacity building is to enable individuals and organizations to perform appropriate and sustainable functions as required by the system(s) they happen to be working in. Ability to perform core functions of EMIS, design methods, defining objectives, managing efficiently and understanding challenges of development needs is crucial. It is with this understanding that the findings under the component of the project – capacity building, will be discussed. The Directorate of Planning and Information should be encouraged by the MOE to assume a more visible position in the development of the EMIS program, while the consultants play a supporting role and gradually recede into the background, Currently the pilot project staff have become a reference point even to the Senior management. During the assessment it looked like the project staff were the focal point in terms of providing information. MOE staff did not look confident enough to provide information and kept on referring queries to the Chief of Party or the Senior EMIS Specialist. In the light of this situation it was felt that they should step up and become more visible to take over in preparation for the future without project staff. On the other hand project staff should refer requests for information to 22 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review the local staff instead of answering themselves. This will enhance the development of confidence and transfer of skills to ensure sustainability of the program at national level. In any case this is the essence of capacity building. In order for a capacity-building program to be effective within a short time, the strategy of training of trainers (TOT) can be used. In this strategy, the central EMIS personnel can effectively be trained and used to train provincial offices and through time the provincial offices should be trained to train district offices. The central EMIS personnel can be used when necessary to assist district offices in training. The district offices can be made responsible for follow up of the survey administration and training of teachers who assist the school administration in compiling school statistics. The central and provincial EMIS can still play a big role in devising strategy, standard setting and providing feedback down the ladder of administrative units. Important factors can be noted here: 1. The use of information should be encouraged at all levels of administration especially at school level. Schools should use the information they produce at school level for planning, monitoring and evaluation. Provinces and districts can do similar analysis at their level by comparing districts’ and schools’ performance. 2. Continuous assessment in identifying gaps on training n and use of the feedback information for further training programs is desirable. eeds Some officers received training in various skills such as systems development, basic computer literacy etc. It is also necessary to train users in basic computer maintenance skills. Basic computer literacy training was given to headquarters staff, pilot provinces and districts in the second year. In addition, more advanced training was given to three (3) EMIS personnel during the second year of the plan implementation. These efforts in capacity MoE EMIS Province EMIS District EMIS Schools Constituency Ward Zone • Use • Monitoring survey • Feedback • Use • Feedback back information Strategy • Continuous Assessment • Training of Trainers • Feedback information • Planning • Monitoring • Evaluation • Training Use Use Use Use MoE EMIS Province EMIS District EMIS Schools Constituency Ward Zone • Use • Monitoring survey • Feedback • Use • Feedback back information Strategy • Continuous Assessment • Training of Trainers • Feedback information • Planning • Monitoring • Evaluation • Training Use Use Use Use ADEA- WGES/NESIS 23 Zambian EMIS and Related Activities External Review building have enabled EMIS professionals to use the Ed*ASSIST database and make some queries to a limited extent. In the light of the limited skills of the EMIS staff there is need to offer targeted on-the-job￾training, in Access programming, especially covering the following essential topics: creating tables, joining tables, relating tables, and making different queries, creating forms and preparing report formats, through access programming and application digest. Another area of training needed is analysis and use of data. There are a number of core indicators and graphs produced as a result of the revived program. Hence, a closer collaboration is needed between EMIS and Central Statistical Office (CSO) to be able to obtain school age population figures needed for the production of core indicators of education systems performance. The team learned that the single age breakdown of population is expected from CSO. While this is a good idea, EMIS personnel are encouraged to contact CSO for the required breakdown and at the same time get the necessary skill to break the 5- year age group by single year age group in-house, using techniques such as Sprague multipliers, used in many other countries. Task 3: Assessment AIR provided technical assistance to ECZ as part of the capacity building initiative. An expert in competency-based testing visited Lusaka three times and each of the consultancies visits lasted two weeks. AIR advised ECZ through the development of competency tests for mathematics and reading in grade 4. The test instrument was designed for application in regular government schools and IRI centres. This test was administered in January 2003, marked, and a draft comparative analysis report prepared. A sub-regional international conference on Learner Assessment for improved Educational Quality was successfully conducted as part of the technical assistance component. The workshop was held in Livingstone, Zambia from 30 June to 2 July, 2003. The countries that participated were Botswana, Kenya, Malawi, Namibia, South Africa, Sudan, Tanzania, United States of America, Zambia and Zimbabwe. The workshop provided an opportunity for exchange of ideas and best practices. In addition to the technical assistance already given the ECZ has indicated that there is still need for further capacity development for its staff. 24 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review The ECZ indicated that they still require technical assistance support in the form of an external consultant based in Zambia. The ECZ has expressed a great need for assistance to enhance their capacity in the areas of sampling designs and strategies, test construction, item analysis, determination and assignment of performance levels to students, data analysis, SPSS, project monitoring and evaluation and project management. (Repetition) Task 4: Community Data Collection and Use The objective of this component of the project was to complement data collected by EMIS with data collected by other sources like the HIV/AIDS unit with in the MOE, the Communities Supporting Health, HIV/AIDS, Nutrition, Gender and Equity Education in Schools (CHANGES), IRI programme through the Educational Broadcasting Services (EBS) and others programmes such as UNICEF. To this end a consultant undertook a study to identify data elements that are collected by these programs and that can be incorporated into the regular data collection system in EMIS. A report on the study was produced. It is incorrect to assume that EMIS can collect all data needed by the ministry for planning, monitoring and decision-making purposes. Additional information useful for these undertakings has to be sought from other agencies especially CSO, research institutes, other government ministries, nongovernmental organizations and other secondary sources. Hence the need to, 1. Strengthen the relation between the research institutions and the research unit of the MOE in order to share good practices and experiences. 2. Augment quantitative data with qualitative data by conducting pilot studies as may be necessary. Suggested Future Direction This project has successfully been implemented within two years with notable indicators of success. It is not correct to assume that EMIS can be built within a short time. The success of the project has become a challenging development that needs to be continuously reinforced and strengthened by both the MOE and the cooperating partners. ADEA- WGES/NESIS 25 Zambian EMIS and Related Activities External Review The effort to build such a system should be taken to scale to include all provinces, districts and schools. However, the logical thing to do is to take the effort step by step so that one can easily learn from the experience and reinforce the activities before taking it to scale nation wide. If the project is taken to scale, it is recommended that all provinces be covered as an experience-building i communication. This essentially means computerizing all provincial offices, connecting them through Internet, building both indiv and institutional capacity, regularly monitoring progress, and eventually decentralizing the system further to districts. A broad nitiative and connected to the center for ease of effective processing and idual spectrum of training needs he challenges of decentralization need to be addressed. These include procurement and ll re s regards training, a good strategy is: rs (TOT’s) as much as possible. The central EMIS time T distribution of hardware, software and other equipment, Internet connectivity, training at a levels and other logistics. Sometimes the role, need and importance of training at various levels is overlooked. If the project is taken to scale, provinces and districts will undergo mo or less the same information cycle, especially as it relates to processing, analyzing, and use of information. In addition they will be responsible for follow up and reporting to higher bodies. A 1. To use the Training of Traine can be used to train provincial officers, and the provincial offices in turn can train district offices and schools starting from basic computer literacy. More advanced training can be planned for both province and districts professionals at the same 26 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review 2. at all levels, SPSS training for experts in the ministry especially the ECZ). Sustainability ean the continuation of the EMIS, running efficiently into d initially, even after the external assistance is no longer available. This stablished and equipped the training center for the ministry and this is an indicator of such a re, and in r portion of commitment is to come from inside EMIS. EMIS by its very nature is an nvironment of learning. It takes individual interest and commitment to engage in nior staff) low r of the focus. The planners and decision-makers at all levels including province and districts of the ministry are primary users of educational information. To plan and offer a targeted, on-the-job-training (e.g., computer programming for EMIS 3. To include all relevant candidates from the onset in order to achieve a wider coverage and save on costs. Sustainability is understood to m the future, as intende means keeping the existing system functioning and reinforcing the established system with new vision and new ideas. For this system to be sustainable, the capacity of local staff has to be developed in all respects. In this regard, the managerial, professional and technical skills need to be strong enough to meet the challenges. On the other hand, a strong political commitment is desirable. The Ministry of Education has e commitment. The EMIS structure has been established and professionals hired. This is another indicator of commitment. However, to keep the system going and strong, a continuous encouragement and assistance is needed from decision-makers. Allocating a budget for training, updating equipment, procurement of necessary hardware, softwa other consumables should be fulfilled as much as possible in order for the system to rema strong. The othe e self-learning. Learning from one another in groups in this favorable environment needs self-motivation and commitment. The MOE management (PS, the Directors and Se should sustain the current good working relationship between members of its staff and al discussions, sharing of experience, and debate on issues that affect the performance of EMIS and develop skills to tackle problems. They need to know, 1. Know what the user wants and put users at the cente ADEA- WGES/NESIS 27 Zambian EMIS and Related Activities External Review 2. Aim at good quality product and services. Good quality information is more likely to be used than a poor quality information thus increasing demand for educational information. Hence, the focus is made to put the user up in front and render good The abo awaren are of the power of formation. The EMIS unit has the responsibility and ability to market the product. This can so 1. A focused training in (computer literacy, basic skills, advanced programming, ct is to go to scale. 2. Continued external assistance is needed for sometime to come until the local staff y, 4. ters 5. mation should be encouraged to quality services by producing credible education information. ve are some of the factors that lead to sustainability. Another success factor is the ess and ability to market the product. Many users may not be aw in be done through production of analytical reports, indicators, training materials etc. It can al be done by presenting the outcome of the year, or changes over time to workshop participants, briefing to higher bodies, publishing, group discussions etc. For this system to be sustainable the following should be followed: systems maintenance or analysis etc) is necessary if the proje managers and professionals become comfortable and are able to develop, modif change, update and enhance the source code to make it usable for the ministry. 3. Decentralize the system including data processing to provinces and districts. The process will involve a series of training activities. Connect all the provinces to Internet facilities. Train technicians and troubleshoo from within the ministry. Planners from the Directorate of Planning and Infor use information from EMIS. 28 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review Conclusion and Summary of Findings Although the MOE is not subject for assessment their role and support for the project ha sto be recognized. The MOE has demonstrated great commitment and support for the project by providing office space. They have also provided the requisite human and materials resources to ensure successful implementation for the project. The ministry has also demonstrated its commitment by equipping the training laboratory and supporting the training component of the project and facing up to the challenges of continuing with the project when donor support comes to an end. The USAID assistance to Zambia’s Ministry of Educations’ EMIS programme and related activities, as implemented by the American Institutes for Research (AIR) has achieved a lot within the two years of the project. It has managed to collect data, countrywide, for the last three (3) years, 2001-2003. The outputs are already made available to users within the ministry and outside on CDs. Different departments of the ministry are connected to Internet thus facilitating efficient communication between departments. A series of training activities were given to the selected provinces and districts in the pilot areas and the main office. Training was also given to EMIS staff at a higher level. An Internet connectivity technical person was also trained as part of EMIS capacity building. Training has to continue in a well-designed and focused manner. The EMIS unit has also supported the technical assistance to different departments of the ministry. Accordingly a consultant was brought in to help the ECZ to assist in the competency testing for the ministry’s IRI program. A draft report was produced to this effect. The project in collaboration with ECZ organized an international conference on Learner assessment for improved Educational quality. This was a notable achievement, which was not in the original plan. The flexibility of the project team is commendable as it ensured greater enhancement of capacity building, by taking on extra items that became necessary for the success of the project. The project also conducted a study on ‘community data collection and use’ to identify the demand and scope of the problem and available data sources to supplement the EMIS schools data bank. ADEA- WGES/NESIS 29 Zambian EMIS and Related Activities External Review On the whole good progress was made during the two years of the project implementation. It is time to plan for a decentralized system in which provinces and districts play a great part in supporting the headquarters and themselves in building EMIS at their levels and begin to use information for planning and decision-making. The current EMIS pilot project has done much to revive the information management sector of the MOE. The project has made a substantial breakthrough in creating awareness of the importance of statistical information within the MOE and other users of this information. Positive attitudes have been developed within a short time. Virtually all the major actors within the MOE acknowledge that the project has made a change in relation to the provision of information for decision-making and educational planning. The ministry used statistics from EMIS to prepare the strategic planning document as well as in preparing budget for the sector. The project staff have also demonstrated a very high sense of commitment to duty and their tasks. It is hoped that this commitment can be sustained in the future. Recommendations EMIS 1. It is recommended that the project should be taken to scale. This time including all the provinces so that they can all benefit equally from the project and reduce disparities that may occur. 2. Strengthen capacity at all levels using the strategy of ‘Training of Trainers’ so that data processing can be done at all lower levels of administration through the cascade model. 3. Connect all provinces to Internet to ease communication between the headquarters and provinces. 4. Continue with computer literacy training both horizontally and vertically. Always use internal capability to do that whenever possible. 5. Create a favorable working environment where staff can learn from one another and strengthen the relation between EMIS on the one hand and users on the other, so that effective utilization of information is achieved. Other units of the Directorate of Planning and Information should be exposed to EMIS so that they can demand data for use in their own plans and programs. 30 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review 6. Assist schools with the production of ‘school abstract’ formats that can be used for school summary statistics. This abstract can be useful in many ways, such as providing schools with ready reference material for reporting to higher offices as well as encouraging schools to produce indicators for planning and monitoring at their own level. The abstract will contribute to data quality as well. 7. There is a possibility for inflating information coming from schools. Hence, it is recommended that a pilot study be undertaken by central EMIS with provincial and district offices should verify the collected data. A small, not costly, sample can be used to lessen the problem. When this is done once a year, in conjunction with the awareness training and feedback information to schools, this could prove effective in the long run. To improve the quality of data the annual school census has to be filled in by the school head who should be held responsible for the accuracy of data coming from schools. The school head should certify the information correct, before submitting the questionnaire to the District Education Office (DEO). 8. The ASC could be reduced by removing sections that do not change drastically annually. Such information can be collected at intervals of 3 to 5 years or through sample surveys. 9. EMIS should start producing annual statistical bulletins for distribution and dissemination to the users. These may include some trend data from 1996-1999 to fill the trend gap. 10. EMIS should endeavor to include and use selected statistics for national examinations in the annual statistical bulletins and statistical reports. 11. EMIS should explore possibilities of using EBS facilities to explain and guide schools prior to the date of compilation of the school census. Capacity Building 1. A multi-pronged training program should be used to provide the required skills. The various components of the training program can run simultaneously. This would cover awareness, computer literacy, middle level training in report writing and production to higher level/advanced levels relating to programming and systems development. A thought should be given to the utilization of local skills from local institutions such as universities and the private sector. Reference to and over reliance on implementing partners has detrimental long-term effects. A core ADEA- WGES/NESIS 31 Zambian EMIS and Related Activities External Review team of Trainers of Trainers should be put in place using existing human resources within MOE. The team can train others as part of their duties. Sharing of resources between and among departments, provinces, districts, schools etc. can greatly reduce costs, while improving the efficiency of the system. 2. Train a computer programmer locally who can modify, update and maintain the system currently used. The target groups for this focused training are head quarters, provincial, and district office statisticians or mathematicians currently working in this area. 3. Computer literacy training has to be continued to province and district level personnel currently working in these offices. It always pays off to train those who have computers at hand. 4. Relevant training has to be given, to increase analytical capacity of the EMIS unit of the ministry headquarters as well as the province and district staff personnel. 5. A well designed training program for schools is necessary to make school teachers aware of information, and the use of information by producing school level indicators, for example teachers can participate in school level research activities, and organize information at school level and report efficiently. Assessment 1. The ECZ has requested technical assistance specifically in the area indicated in the scope of work. We suggest that project implementers and ECZ sit together, discuss and come up with a statement of understanding. They should produce a proposal and discuss it with project implementers. 2. There is a need to train ECZ staff in the use of basic computer application packages such as SPSS. Community Data Collection and Use As it stands now, EMIS is just beginning to organize the regular data collection system of the ministry. The question of community data relates very much to research undertakings and the utilization of secondary data. To get around this problem, one can consider: 1. strengthening the research section of the ministry to do the research work in collaboration with other research oriented projects under the ministry 32 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review 2. EMIS increasing its capacity to collect and utilize secondary data related to education. 3. EMIS needs to cooperate and work closely with research section of the MOE and other research projects and utilize the data collected from research findings as secondary data. ADEA- WGES/NESIS 33 Zambian EMIS and Related Activities External Review List of Interviewees 1. Mrs. B. Y. Chilangwa – Permanent Secretary, MOE 2. Dr. Lawrence Musonda – Director of Planning, MOE 3. Ms. Beatrice Mugwagwa – Systems Analyst, MOE 4. Mr. B. Musonda – Senior Statistician, MOE 5. Dr. George Caldwell - AED Consultant 6. Mr. R. D. Warrick – Chief of Party (AIR) 7. Mr. M. Hazemba - DESO, Siavonga district. 8. Mr. Dan Nyumba - Statistical officer, Statistical Officer, Siavonga District 9. Mr. A. C. Kapita – Head Teacher Siavonga Upper Basic School 10. Mr. J. B. Chilaka – PEO Eastern Province 11. J. C. Musonda – SPO, Eastern Province 12. G. Chiziba A/SEO, Eastern Province 13. Mr. D. Chirwa – S/EXO , Eastern Province 14. D. H. Mwanza – EXO , Eastern Province 15. Mr B. G. Phiri EXO , Eastern Province 16. Ms. Irene Jere - Data E OP, Eastern Province 17. Mr David Lungu – Statistician, Eastern Province 18. M. S. Tembo – R/o, Eastern Province 19. R. L. Kasoma – R Accountant, Eastern Province 20. D. C. Mvula - B.O. , Eastern Province 21. S. C. J. Banda – Planning Officer, Chipata District, Eastern Province 22. Mr. R. G. Tembo – Senior Human Resources Officer, Chipata District 23. Ms. Chilaka – Head Teacher Mpezeni Middle Basic School, Chipata District 24. Ms. W. Chilala - USAID Education Specialist 25. Dr. C. Chipoma – USAID Education Specialist 26. Dr. Rick Henning USAID – USAID Education Advisor 27. Mr. Mwansa - PEO Kabwe, Central Province 28. Mrs. Bwalya – Senior Planning Officer, Central Province 29. Mr. S. Chumbula – A/Statistical Officer, Central Province 30. Mr. Lubinda – Controller EBS Ms. G. Hamaimbo – EBSMrs. Mubiana - EBS Ms. F. Ngoma - Data Entry Operator, Central Province Mr. Charles Ndakala – Senior Systems Analyst, MOE 34 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review References 1. Educating Our Future- -National Policy on Education Ministry of Education, Lusaka, 1996 2. Report on the Community Sensitization and Mobilization Campaign, Research and Verification Exercise Conducted in Sinavonga District – Livingstone , Zambia, December 2003 3. EMIS Program- The way Forward, Capacity Building and Strategic Growth 2004 to 2005 4. Sub – Regional Conference on Assessment – 30 June to 2 July 2003 5. Assessment of Current IT Capabilities and Connectivity – December 2003 6. Zambia DHS EdData Survey 2002 – Education Data for Decision Making – U.S. Agency for International Development – April 2003 7. Mid-erm Assessment of USAID Support For Zambia’s Education Management Information Systems (EMIS) And Related Activities – G. Makwati et al, 2003 8. Education Management Information Systems and Related Activities. Task 4 Community Data Collection and Use – Consultancy Report – M. R. Malyenkuku ADEA- WGES/NESIS 35 Zambian EMIS and Related Activities External Review Appendix 1 – EMIS and Related Activities Equipment Distribution Table Full Equipment Distribution D D i s k S t o r r c V i e & S p C D P r oj Headquarters, Eastern and Southern Province Locations Servers PCs Compute r Table s UPS Units Surge Pro tect o rs Laser Printer s Printer cables Laser Toner Portable HDD CDRW Drive s CDRW Media Dust Cover Kits Keyb o ards C a ge b oxes Fax Mach n e s Fa x Toner F l o p p y Disks PC Cleaning Kits XP Pro Norton Anti-V i rus Norton Utilities A acial Analyst Network Cards L ector an d Case MS Exchan g e MOE HQ PU-Budget and Planning 1 1 1 1 1 1 1 1 1 1 1 1 1 1 PU-Planning and Research 1 1 1 1 PU-Information Systems 3 5 26 8 5 6 1 15 30 18 11 5 10 5 4 4 16 11 8 3 1 20 1 1 PU-School Infrastructure 1 1 1 1 1 1 1 1 1 Sub-Total 3 8 29 11 8 9 4 15 30 18 11 8 10 8 4 4 16 11 0 11 3 1 20 1 1 Eastern Provincial Planning Office 2 1 2 2 1 1 5 1 9 2 2 1 1 4 9 2 1 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 e 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Chadiza 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Chama e 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Katet Lundazi 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Mambw Nyimba 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Petauke 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Chipata 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Sub-Total 18 9 18 18 9 9 45 0 1 81 18 0 18 9 9 36 81 0 18 1 0 0 0 0 36 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review Full Equipment Distribution Headquarters, Eastern and Southern Province Locations Servers PCs Compute r Table s UPS Units Surge Pro tect o rs Laser Printer s Printer cables Laser Toner Portable HDD CDRW Drive s CDRW Media Dust Cover Kits Keyb o ards C D D i s k S t o r a ge b oxes Fax Mach n e s Fa x Toner F l o p p y Disks PC Cleaning Kits XP Pro Norton Anti-V i rus Norton Utilities ArcVie &Spacial Analyst Network Cards LCD Projector an d Case MS Exchan g e Southern Provincial Planning Office 2 1 2 2 1 1 5 1 9 2 2 1 1 4 9 2 1 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Livingstone o 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Kalom Namwala 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Monze 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Gwembe 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Sinazongwe 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Choma 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Mazabuka 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Siavonga 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Kazungula 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Itezhitezhi 2 1 2 2 1 1 5 9 2 2 1 1 4 9 2 Sub-Total 24 12 24 24 12 12 60 0 1 108 24 0 24 12 12 48 108 0 24 1 0 0 0 0 Total 3 50 50 53 50 30 25 120 30 20 200 50 10 50 25 25 100 200 0 53 5 1 20 1 1 Procured 3 50 50 53 50 30 25 120 30 20 200 50 10 50 25 25 100 200 53 5 1 20 1 1 Signature of reciept: Date: ADEA- WGES/NESIS 37 Zambian EMIS and Related Activities External Review Appendix 2 – EMIS and Related Activities External Evaluation TOR Terms of Reference Evaluation of USAID’s Support for Zambia’s Education Management Information System and Related Activities Background In 1999, Zambia’s Ministry of Education (MOE) launched its sector reform program, the Basic Education Sub-Sector Investment Programme (BESSIP). BESSIP’s primary objectives are to increase enrollment rates and improve learning achievement at the primary school level. These objectives are being attained through enhancing on-going programs and developing new ones. BESSIP is financed through a partnership of government and external financing agencies for an estimated total of $840 million for the 1999-2006 period. USAID/Zambia Country Strategic Plan’s second strategic objective, improved quality of basic education for more school-aged children, supports BESSIP’s objectives. USAID/Zambia defines “quality basic education” as “improved access, retention and achievement.” Improved access means more school age children, girls and boys, rural and urban, receiving basic education. Improved retention means more pupils continuing and completing primary schooling (grade 1-7). Improved achievement means more pupils learning basic skills and competencies. In conjunction with this strategic objective, USAID has a related intermediate result, namely “improved information for education decision-making processes.” USAID measures achievement of this result through two indicators: (1) production of an annual school census statistical report, on time and (2) number of administrative units producing periodic reports. To address this intended result, USAID supports Zambia’s Education Management Information System (EMIS) through a technical-services contract with the American Institutes for Research (AIR) and its subcontractor, the Academy for Educational Development. This contract began in December 2001 and is scheduled to end in March 2004. The main objective of the Zambian EMIS is to improve the MOE’s capabilities to collect, organize, process, store, share, and disseminate education information for informed planning and management of education at all levels. A related objective is to establish an EMIS platform that provides the MOE at all levels with access to that information and to establish and enable the MOE to use simulations and modeling systems, based on real, current EMIS data, to plan and to project needs and resource requirements within the MOE, based on various best/worst-case scenarios of the future. Furthermore, AIR is tasked to collaborate with the Ministry of Education and the Examinations Council of Zambia to enhance Zambia’s assessment of learning outcomes through its initiative on competency testing. AIR is addressing four primary tasks. First, AIR is providing technical assistance to the MOE to develop, pilot, improve, and implement an education management information system at the national level and selected districts in Southern and Eastern Provinces and to develop and implement a plan enabling the MOE to collect, analyze, manage, and use timely 38 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review ADEA- WGES/NESIS 39 education information to make education decisions at all levels of the education system, including headquarters, provincial, district, school and community levels throughout Zambia. In accordance with the terms of its contract with USAID, AIR is required to have the EMIS operating in at least two provincial office levels and 19 districts offices by the end of the contract. In addition, AIR is tasked to establish within the MOE a functioning simulation and modeling system with which planners can identify future resource needs, based on current data and various best-case/worst-case scenarios. Second, the contract requires AIR to improve the MOE’s and others’ capacities to collect, share, analyze, disseminate, and use information at all levels of the educational system to make good educational decisions. As part of the second task, AIR has provided improved information technology (computers, internet hookups, etc.) as required by the EMIS decentralization strategy in the Southern and Eastern provinces where such do not exist. Decisions about which equipment to purchase were based on a needs assessment to determine precise hardware and software specifications and needs required to support installation and operation of the EMIS in a phased manner and to ensure maintenance and service support for a minimum number of years. As a further contribution to capacity building, AIR conducted a training needs analysis, to assure its compatibility with and/or integration into the MOE overall planning for human resources development, and to provide the necessary support to ensure that capacity to use EMIS and to make decisions based on EMIS-generated information at all levels is developed. The needs analysis identified staffing and skill requirements in view of the new EMIS, a greater understanding among all levels of personnel of the kinds of information they need to make good decisions, and hardware/software to be used, and included recommendations for institutionalizing information management functions at all levels, but especially at the district levels. Third, the contract requires AIR to collaborate with MOE’s headquarters, provincial and district education offices, and relevant partners working in Southern and Eastern Provinces to develop and/or refine community-based models for collecting and utilizing MOE and/or community school data so that such data are harmonized with and, where indicated, included in the EMIS. Fourth, the contract requires AIR to “collaborate closely with the Examinations Council and relevant units of the MOE in developing and implementing a training plan that will enhance the capacity of Examinations Council personnel in competency test development, implementation, and use.” Statement of Work The Association for Development of Education in Africa (ADEA) will take the lead in completing a summative evaluation of the success and progress of USAID’s support for the Zambia EMIS and related activities. In completing this task, ADEA shall: • Propose a team leader and one other team member; the candidates proposed, preferably from southern Africa, should have prior experience with education statistics and their collection, management, use, and reporting; • Work with two Zambian team members, including one from the MOE, to be selected in consultation with USAID/Zambia and the Ministry of Education; • Meet with representatives of the MOE, USAID/Zambia, and AIR in Lusaka to discuss AIR’s program-related efforts and these representatives’ assessment of the program; Zambian EMIS and Related Activities External Review 40 ADEA- WGES/NESIS • Visit one or two provinces (and/or district education offices within these provinces) selected in consultation with USAID and the program’s chief of party, to compare and contrast their use of EMIS and information technology. USAID/Zambia will provide further guidance about the purposes of the site visit(s). • Review program documentation, including relevant portions of AIR’s contract with USAID and AIR’s periodic progress reports submitted to USAID; • Provide a draft written evaluation of the EMIS program and its related activities to AIR and USAID no later than March 5, 2004. The evaluation process shall also include: a) an interim, oral presentation to and discussion with USAID on or about the midpoint of the team’s presence in Zambia; and b) an oral presentation to and discussion of preliminary results with USAID and the Ministry of Education in Lusaka on or about February 27, 2004. • Revise, as necessary, the draft written evaluation in response to comments from USAID, the MOE, and AIR. • Submit a final written evaluation to USAID and AIR no later than March 12, 2004. This evaluation shall draw conclusions based solely on the data collected and, shall, if applicable, direct all recommendations for change to either USAID/Zambia or to AIR. The final report shall be objective and shall be based on a comparison of anticipated results with accomplishments or progress toward these results. The MOE shall not be the subject of the evaluation but will be given the opportunity to review and comment on the draft evaluation report. USAID, AIR, and the MOE shall receive a copy of the final report. The evaluation shall include an introduction to the program and history, a list of appropriate indicators, a description of achievements, and an explanation of activities not completed. The evaluation shall also provide suggested future directions for the program as well as recommendations for its enhancement or improvement. During the course of the evaluation, the following questions should be addressed, in addition to other questions that may be deemed to be of interest or importance: • What progress and accomplishments can be attributed to the program’s efforts? • How well has the program achieved its desired outcomes? Answering this question should include consideration of the program’s required deliverables and whether these deliverables were completed satisfactorily. The evaluation team shall also consider whether there were changes to the program’s initial scope of work. If so, how were such changes made and why? • What are the program’s strengths and weaknesses? • Is the EMIS system introduced during the program sustainable after the end of donor support? If not, why, and what must be done to ensure or promote the system’s sustainability? • Are there new or additional activities or tasks that could usefully contribute to the strengthening and enhancement of the EMIS and the efforts of the Examinations Council of Zambia over the next three to five years? The intended outcome is a fair, objective, and independent appraisal of the program’s progress and achievements throughout the entire program period. Zambian EMIS and Related Activities External Review Appendix 3 – EMIS Pilot Training Break-down and Programme Cost Summary Sheet for Five Months of EMIS Training A) Lusaka EMIS Training 1. Subsistence allowance No. of staff Rate No. of nights Total Provincial Education Officers (PEO) 4 45,000 30 5,400,000 Senior Education Officers (SEO) Planning - Provincial 4 45,000 30 5,400,000 Statistical Officers (SO) - Provincial 4 40,000 30 4,800,000 District Education Officer (DEO) 19 45,000 30 25,650,000 Education Officers (EO) - District 25 45,000 30 33,750,000 Statistical Officers (SO) or designated EO backup - District 19 40,000 30 22,800,000 PIP - Northern 1 45,000 30 1,350,000 Special Advisor - Western 1 45,000 30 1,350,000 Totals 77 100,500,000 Note: The number of nights are based on the number of training days that a given staff member will receive Moe staff within Lusaka are not included in the totals. B) Transportation to and from Lusaka Transportation allowance via public transport No. of staff Rate Total Southern Livingstone 6 35,000 2,100,000 Gwembe 3 34,500 1,035,000 Siavonga 3 25,000 750,000 Namwala 3 30,000 900,000 Kalomo 3 33,000 990,000 Choma 3 28,000 840,000 Kazungula 3 46,000 1,380,000 Monze 3 22,000 660,000 Mazabuka 3 18,000 540,000 Itezhitezhi 3 22,000 660,000 Sinazongwe 3 35,000 1,050,000 Sub-total 36 10,905,000 Eastern Lundazi 3 55,000 1,650,000 Chipata 6 50,000 3,000,000 Nyimba 3 35,000 1,050,000 Chadiza 3 25,000 750,000 Chama 3 55,000 1,650,000 Mambwe 3 75,000 2,250,000 Petauke 3 55,000 1,650,000 Katete 3 28,000 840,000 Sub-total 27 12,840,000 Northern Kasama 7 75,000 3,500,000 Western Mongu 7 75000 3,500,000 Sub-total Northern/Western 14 7,000,000 Total Transportation Allowance: 30,745,000 C) Fuel Cost to and from Lusaka for Provincial Staff Southern Distance Fuel Total for Total for 10 ADEA- WGES/NESIS 41 Zambian EMIS and Related Activities External Review Cost R/T R/T Livingstone 497 3,174 694,090 6,940,903 Eastern Chipata 557 3,174 777,884 7,778,839 Northern Kasama 858 3,174 1,198,248 6,647,626 Western Mongu 476 3,174 664,763 11,982,485 Total 33,349,853 D) Accommodations and Meals 171,882,000 E) Grand Total for EMIS Training 336,476,853 42 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review 12 34 56 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Sunday Monday Tuesday Wednesday Thursday Friday Saturday HQ Senior Staff Education Officers - District 30 31 Statistical Officers - District January 2003 PC Basics MS Word Introduction - 1 ½ days MS Excel Introduction - 1 ½ days OutlookIntroduction WordIntroduction ED*ASSIST - DDM ED*ASSIST - QTS & IDEA PC Basics PEOs, SEOs,SOs, HQ Staff, PIP￾Northern,Technical Advisor-Western MS Word Introduction - 1 ½ days MS Excel Introduction - 1 ½ days ED*ASSIST - DDM PC Basics MS Word Introduction - 1 ½ days MS Excel Introduction - 1 ½ days ED*ASSIST - QTS & IDEA PC Basics ConfirmedEMISTraining Program Revised04/02/03 ADEA- WGES/NESIS 43 Zambian EMIS and Related Activities External Review 1 2 3456 78 910111213 1415 16 17 18 19 20 21 22 23 24 25 26 27 28 Sunday Monday Tuesday Wednesday Thursday Friday Saturday District Education Officers HQ Senior Staff Education Officers - District PEOs, SEOs,SOs, HQ Staff -PIU, PIP-Northern, TechnicalAdvisor￾Western February 2003 MS Word Introduction - 1 ½ days MS Excel Introduction - 1 ½ days ED*ASSIST - DDM PC Basics InternetSearching Excel Introduction Outlook Calendar Features MS Word Intermediate - 1 ½ days MS Excel Intermediate - 1 ½ days ED*ASSIST - DDM InternetSearching MS Word Intermediate - 1 ½ days MS Excel Intermediate - 1 ½ days File Management InternetSearching ConfirmedEMISTrainingProgram Revised04/02/03 44 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review ADEA- WGES/NESIS 45 Zambian EMIS and Related Activities External Review 1 23 4 5 67 8 9 10 1112 1314 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Sunday Monday Tuesday Wednesday Thursday Friday Saturday Statistical Officers - District District Education Officers HQ Senior Staff 30 31 Education Officers - District March 2003 MS Word Intermediate - 1 ½ days MS Excel Intermediate - 1 ½ days ED*ASSIST - DDM InternetSearching MS Word Intermediate - 1 ½ days MS Excel Intermediate - 1 ½ days File Management InternetSearching File Management PowerPoint Introduction Outlook Filters and Rules File Management MS Word MS Excel ED*Assist ConfirmedEMISTrainingProgram Revised25/03/03 InternetSearching 46 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review ADEA- WGES/NESIS 47 Zambian EMIS and Related Activities External Review 12 34 5 67 89 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Sunday Monday Tuesday Wednesday Thursday Friday Saturday Statistical Officers - District District Education Officers HQ Senior Staff Education Officers - District April 2003 PEOs, SEOs, SOs, HQ Staff -PIU, PIP-Northern, TechnicalAdvisor￾Western MS Word MS Excel ED*ASSIST MS PowerPoint Introduction File Management Confirmed EMISTraining Program March - 30 March - 31 May - 1 May - 2 Revised 25/03/03 MS PowerPoint Introduction ED*ASSIST OutlookWebAccess MS Word MS Excel ED*ASSIST MS PowerPoint Introduction MS Word MS Excel OutlookWebAccess ED*ASSIST Queries MS PowerPoint Introduction MS Word MS Excel OutlookWebAccess 48 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review ADEA- WGES/NESIS 49 Zambian EMIS and Related Activities External Review 12 3 45 6 7 89 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Sunday Monday Tuesday Wednesday Thursday Friday Saturday PEOs, SEOs, SOs, HQ Staff - PIU, PIP-Northern, Technical Advisor-Western Statistical Officers￾District District Education Officers Education Officers￾District ConfirmedEMISTrainingProgram May 2003 Revised25/03/03 ED*ASSIST MS PowerPoint Introduction MS Word MS Excel File Management ED*ASSIST MS PowerPoint Introduction MS Word MS Excel File Management ED*ASSIST MS PowerPoint Introduction MS Word MS Excel File Management MS Word MS Excel ED*ASSIST File Management CompletionCeremony 50 ADEA- WGES/NESIS Zambian EMIS and Related Activities External Review 12 34 5 6 8 9 10 11 12 13 15 16 17 18 19 20 22 23 24 25 26 27 29 30 Sunday Monday Tuesday Wednesday Thursday Friday Saturday PEOs, SEOs, SOs, HQ Staff - PIU, PIP-Northern, Technical Advisor-Western District Education Officers Statistical Officers￾District Confirmed EMISTraining Program Revised25/03/03 MS Word MS Excel ED*ASSIST CompletionCeremony MS Word MS Excel ED*ASSIST CompletionCeremony MS Word MS Excel ED*ASSIST CompletionCeremony ADEA- WGES/NESIS 51 Zambian EMIS and Related Activities External Review 52 ADEA- WGES/NESIS