KON RAY BOARDING SCHOOL AND CENTRAL HIGHLANDS EDUCATION PROJECT END-OF-PROJECT EVALUATION January 2013 This publication was produced for review by the United States Agency for International Development. It was prepared by Julian Watson and Erica Holzaepfel through Social Impact, Inc. EAST MEETS WEST FOUNDATION KON RAY BOARDING SCHOOL AND CENTRAL HIGHLANDS EDUCATION PROJECT END-OF-PROJECT EVALUATION DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is intended for limited internal distribution and is not yet available in printed or online versions. Documents will be made available through the Development Experience Clearing House (http://dec.usaid.gov). Additional information can be obtained from Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc., to the United States Agency for International Development under USAID Contract No. RAN-I-00-09-00019, Task Order AID-486-TO-12-00002. CONTENTS ACRONYMS................................................................................................................................................................ ii 1 Executive Summary ............................................................................................................................................ 1 1.1 Evaluation Purpose ................................................................................................................................... 1 1.2 KRBS Project Scope ................................................................................................................................. 1 1.3 Evaluation Scope and Methods ............................................................................................................... 2 1.4 Summary of Findings and Conclusions ................................................................................................. 3 1.4.1 Sustainability of the Kon Ray Boarding School ............................................................................... 3 1.4.2 Sustainability of Other KRBS Project Activities .............................................................................. 4 1.4.3 Management of KRBS ......................................................................................................................... 5 1.4.4 Ethnic Minority Student’ Access to Education................................................................................ 6 1.4.5 Children With Disabilities’ Access to Education ............................................................................. 7 1.4.6 Teacher Training ................................................................................................................................... 7 1.5 Selection of Recommendations ............................................................................................................... 8 2 Introduction ......................................................................................................................................................... 9 2.1 Education in the National Context ......................................................................................................... 9 2.2 Ethnic Minority Policies and Practices ................................................................................................ 12 2.3 Central Highlands Socio-Economic Context ...................................................................................... 13 3 Project Description ........................................................................................................................................... 14 3.1 Project Logic ............................................................................................................................................ 14 3.2 Project Background ................................................................................................................................. 14 3.3 History of Project Agreement Modifications ...................................................................................... 15 3.4 Theory of Change in an Evolving Project ........................................................................................... 16 3.5 Boarding School or Semi-Boarding School ......................................................................................... 17 3.6 External Factors Affecting the KRBS Project .................................................................................... 18 4 KRBS Project Evaluation Design and Methodology .................................................................................. 19 4.1 Evaluation Focus ..................................................................................................................................... 19 4.2 Evaluation Methodology ........................................................................................................................ 20 4.2.1 Review of Project Documents and Background Reports............................................................. 20 4.2.2 Semi-Structured Key Informant and Group Interviews ............................................................... 20 4.2.3 Focus Group Discussions ................................................................................................................. 21 4.2.4 Direct Field Observations ................................................................................................................. 21 4.3 Limitations on the Evaluation ............................................................................................................... 21 4.4 Findings ..................................................................................................................................................... 22 4.4.1 Overarching ......................................................................................................................................... 22 4.4.2 Funds Received From the GVN ...................................................................................................... 23 4.4.3 Income-Generating Activities ........................................................................................................... 24 4.4.4 Parental and Community Support .................................................................................................... 26 4.5 Conclusions .............................................................................................................................................. 26 5 Sustainability of Other KRBS Project Activities ......................................................................................... 27 5.1 Micro-Credit Loans ................................................................................................................................. 27 5.1.1 Findings ................................................................................................................................................ 27 5.1.2 Conclusions .......................................................................................................................................... 28 5.2 Agricultural Extension ............................................................................................................................ 28 5.2.1 Findings ................................................................................................................................................ 28 5.2.2 Conclusions .......................................................................................................................................... 28 5.3 Early Childhood Care and Education Centers ................................................................................... 28 5.3.1 Findings ................................................................................................................................................ 28 5.3.2 Conclusions .......................................................................................................................................... 29 6 Management of the Kon Ray Boarding School Project ............................................................................. 29 6.1 Project Management ............................................................................................................................... 29 6.1.1 Findings ................................................................................................................................................ 29 6.1.2 Conclusions .......................................................................................................................................... 31 6.2 Partnership Management ........................................................................................................................ 31 6.2.1 Findings ................................................................................................................................................ 31 6.2.2 Conclusions .......................................................................................................................................... 32 7 Ethnic Minority Students’ Access to Education .......................................................................................... 32 7.1 Findings ..................................................................................................................................................... 32 7.1.1 Baseline Data ....................................................................................................................................... 32 7.1.2 School Enrollment .............................................................................................................................. 32 7.1.3 Academic Performance ...................................................................................................................... 33 7.1.4 Barriers to Education ......................................................................................................................... 33 7.2 Conclusions .............................................................................................................................................. 33 8 Children with Disabilities’ Access to Education.......................................................................................... 34 8.1 Findings ..................................................................................................................................................... 35 8.2 Conclusions .............................................................................................................................................. 36 9 Teacher Training ............................................................................................................................................... 36 9.1 Findings..................................................................................................................................................... 36 9.2 Conclusions .............................................................................................................................................. 37 10 Overarching Recommendations ..................................................................................................................... 37 ANNEX A: KRBS Project Evaluation Statement of Work ............................................................................... 42 ANNEX B: KRBS Project Cooperative Agreement Modifications .................................................................. 45 ANNEX C: Evaluation Limitations and Change of Evaluation Scope ............................................................ 47 ANNEX D: Vietnam KRBS Project Evaluation Plan ......................................................................................... 51 ANNEX E: Interview and Focus Group Discussion Guides ............................................................................ 62 Department of Planning and Investment .......................................................................................................... 62 Education Promotion Association ..................................................................................................................... 63 Provincial Women’s Union.................................................................................................................................. 64 National Ministry of Education and Training .................................................................................................. 65 Current Students at Kon Ray Boarding School ................................................................................................ 66 Kon TUM Provincial Department of Education and Training ..................................................................... 67 ANNEX F: Evaluation Key Informants and Stakeholders ............................................................................... 69 ANNEX G: Documents Requested From EMW ................................................................................................ 72 ANNEX H: KRBS Project Revised Evaluation Schedule .................................................................................. 73 FIGURES Map 1: Kon Tum Province, Central Highlands Vietnam ...................................................................................... i Map 2: Regions of Vietnam ......................................................................................................................................... i Map 3: Districts of Kon Tum Province .................................................................................................................... i KRBS Project End-Of-Project Evaluation i Map 2: Kon Tum Province, Central Highlands Vietnam ←Kon Ray Boarding School is approximately six km due north of Kon Ray Town Map 3: Districts of Kon Tum Province Map 1: Regions of Vietnam ii KRBS Project End-Of-Project Evaluation ACRONYMS BOET Bureau of Education and Training CECI Canadian Centre of International Development and Cooperation CEM Committee for Ethnic Minorities CIDA Canadian International Development Agency CoAg Cooperative Agreement COR Contracting Officer Representative DOET Department of Education and Training DOFA Department of Foreign Affairs DPI Department of Planning and Investment ECCE Early Childhood Care and Education (Centers) EFA Education for All EMW East Meets West Foundation EPA Education Promotion Association FGD Focus Group Discussion GVN Government of Vietnam ILMC Improved Livelihoods for Mountainous Communities INGO International Nongovernmental Organization KII Key Informant Interview KRBS Project Kon Ray Boarding School and Central Highlands Education Project KRBS Kon Ray Boarding School KRD Kon Ray District KTP Kon Tum Province MDG Millennium Development Goal MOET Ministry of Education and Training MoE Team EMW’s Internally-Commissioned Evaluation Team MOLISA Ministry of Labor, Invalids, and Social Affairs PM Project Manager PPP Purchasing Power Parity PS Project Supervisor RFA Request for Applications SEQAP School Education Quality Assurance Project SI Social Impact SOW Statement of Work TIP Trafficking in Persons TNA Training Needs Assessment UD University of Danang UN United Nations UNESCO United Nations Education, Scientific and Cultural Organization USAID United States Agency for International Development VGOS Vietnam General Office of Statistics VHLSS Vietnam Household Living Standards Survey VIE Documents in Vietnamese Language Only VND Vietnamese Dong WU Women’s Union KRBS Project End-Of-Project Evaluation 1 1 EXECUTIVE SUMMARY 1.1 EVALUATION PURPOSE This evaluation results from a request by USAID/Vietnam for Social Impact Inc. (SI) to evaluate the East Meets West (EMW) Kon Ray Boarding School and Central Highlands Education Project (KRBS Project). The KRBS Project Cooperative Agreement (CoAg) was awarded in September, 2006 for a period of two years at a cost-sharing level of $512,906 USD. The agreement was significantly modified two times over its lifespan; first in 2007 to expand program activities and increase the funding level by $229, 897 USD, and again in 2008 to extend the completion date from July 2008 to July 2012 while once again, expanding the scope and activities of the agreement and revising the total obligated amount to $1,412,171 USD. The twelfth, and final, CoAg modification in December 2011 raised the total obligated amount to $2,512,127 USD. The purpose of this final evaluation is to provide the United States Agency for International Development’s Mission in Vietnam (USAID/Vietnam) with insight into the effectiveness of the KRBS Project. The evaluation, designed to assess the strengths and weaknesses of the project’s activities, is intended to inform future development strategies as well as to provide the Mission with recommendations about how to best engage with the Government of Vietnam (GVN) in the education sector. USAID/Vietnam’s primary concern for this report was to explore the appropriateness of the project and to determine whether or not future opportunities and interest exist within the Central Highlands for projects of a similar design and scope. A secondary purpose of the evaluation was to assist the Mission in meeting its USAID Forward goal of strengthening monitoring and evaluation to improve their decision-making process. 1.2 KRBS PROJECT SCOPE Under the original agreement, the KRBS Project was concerned with designing and building a boarding school for ethnic minority children in Kon Ray District (KRD) within Kon Tum Province (KTP), of the Central Highlands region of Vietnam. The objective of the project was to improve access to education for ethnic minority children, as well as children with disabilities. The school was intended to be a model learning center, providing specialized opportunities for skill development through construction of technical facilities for sewing, carpentry, agriculture, and tailoring, among others. While the first phase of the project was predominantly focused on construction, it also included activities for teacher training on how to work with children with disabilities, and agricultural extension and micro-credit loans for students’ families. In September 2006, EMW submitted a proposal to USAID requesting supplemental funding to continue and enhance several of the original project activities as well as to expand the scope of the project to include a selection of new activities. Specifically, phase two of the KRBS Project involved a greater focus on teacher training, expanding this activity beyond the KRBS and beyond KRD to engage teachers at early childhood care and education centers (ECCE) in several other districts throughout KTP. With the majority of the infrastructural construction work completed under the first proposal, phase two included activities to install an electrical transformer, a clean water system, and extensive landscaping. The agricultural extension services were proposed for expansion to an additional 90 farmers in KRD, while the services and support to children with disabilities was increased to provide medical assistance, prosthetics, assistive devices, and physical therapy. 2 KRBS Project End-Of-Project Evaluation EMW submitted a third and final proposal to USAID in July 2008 to request an extension of the agreement through July 2012 and increased funding to support additional teacher training at KRBS and across ECCEs in KTP, as well as to refurbish 25 ECCEs through construction of various educational facilities. Phase three of the KRBS Project also contained a substantial emphasis on achieving sustainability for the KRBS, largely through the design and implementation of various school-based income-generating activities such as raising pigs, poultry, and fish; constructing a nursery garden, growing corn and dragon fruit; hosting educational trainings; and supporting voluntourism.1 1.3 EVALUATION SCOPE AND METHODS This summative evaluation examined each of the three KRBS Project phases, from project inception in September 2006, to project completion in July 2012. The evaluation was conducted over the period from August 2012 through October 2012 with deployment to Vietnam taking place between September 23 and October 16, 2012. Primary data collection in Kon Tum Province was conducted over a three-day period from Monday, October 1 through Wednesday, October 3. USAID/Vietnam identified four, key evaluation questions to be investigated by this evaluation. Due to limitations placed on the SI Evaluation Team (the Team), the evaluation scope was refined to examine a select few of the original evaluation questions. (The evaluation statement of work [SOW] can be found in Annex A: KRBS Project Evaluation Statement of Work. Comprehensive information about the change in evaluation scope can be found in Annex C: Evaluation Limitations and Change in Evaluation Scope. Additional information on the breadth and depth of the original evaluation questions, as well as a discussion of the limitations to the evaluation, is included in the body of this report.) The four, overarching evaluation questions driving the focus of this evaluation include: 1. Boarding School Project Sustainability 2. Sustainability of Other Project Components 3. Has Phase I Met its Stated Goals and Objectives 4. Has Phase II Met its Stated Goals and Objectives To answer the evaluation questions, the Team employed a mixed-methods design, combining descriptive and analytical techniques to answer strategic design and performance assessment questions to understand what happened and why. The evaluation was organized into four phases: (1) planning, document review and finalization of evaluation plan; (2) conducting the evaluation; (3) data analysis and oral presentation; and (4) reporting and dissemination of results. The following data collection methods were applied at various phases of the evaluation: Review and analysis of information from key project documents and reports, as well as background data sources, such as World Bank and UNICEF studies Semi-structured, key informant interviews (KII) Semi-structured, group interviews Focus group discussions (FGD) Direct field observations The list of persons contacted is included in Annex F: Evaluation Key Informants and Stakeholders. 1 Voluntourism is travel which includes volunteering for a charitable cause. KRBS Project End-Of-Project Evaluation 3 1.4 SUMMARY OF FINDINGS AND CONCLUSIONS 1.4.1 Sustainability of the Kon Ray Boarding School 1.4.1.1 Findings School Management reported that they did not receive management training to support the transition process between EMW and GVN, nor had they been prepared for the completion of the KRBS Project and its final handover. Two members of the school management board did not know that the KRBS Project had closed. This contradicted the information given verbally to the Evaluation Team by EMW, who stated that handover training had been provided.2 School management reported that they had received some management training from the DOET, but the Team was not able to establish the frequency or content of DOET training. The Team observed that the school’s superior facilities were not operating at full capacity, with 68% of available student placements filled. The school has two vocational classrooms, one for sewing and one for electrical studies, but the school lacks permanent teachers to support these classrooms. People with the appropriate skills are hired from the community to provide courses at various times throughout the year. These courses are for Grades 8 and 9 and the classes run four or five times a week over the course of four to five months. Inspection of the vocational training classrooms showed that the equipment had not been used in the recent past.3 Neither the school management nor the DOET shared accounting records of government funding for the school with the Evaluation Team. Both bodies assured the Team that the KRBS receives its full statutory funding, including funds from Program 135. (The funding of all schools in Vietnam, but particularly boarding schools, is complicated because money comes from many separately accounted sources. Being a semi-boarding school, KRBS is covered by three directives: (1) public financing for education; (2) policies on semi-boarding students and semi-boarding schools; and (3) operational regulations on semi-boarding schools.4) In order to make up for the anticipated shortfall in income, EMW changed the emphasis of the component from agricultural learning to income generation. By the time EMW’s 2011 Annual Report for the project was published, the concept of the agriculture learning center had been dropped from the progress reports altogether and the report stated, “The KRBS had conducted several income-generating activities to cover its project deficit as well as provide food for 172 students.” The KRBS farming activities no longer have any of the characteristics of an agricultural learning center. The Team was unable to obtain any evidence that the small-scale vegetable production on the school grounds was being used for any purpose other than providing minor supplements to school meals. 1.4.1.2 Conclusions During its brief time at the school, the Evaluation Team was given no cause to doubt that the school was providing a positive education to the students. There appeared to be good staff/pupil relationships and the atmosphere was conducive to learning with appropriate classroom infrastructure and resources. The school is functioning according to the standards set by the GVN, but the complimentary activities at the 2 EMW, 2012 3 The failure to meet target enrollment is discussed in Error! Reference source not found. 4 Interview with MOET 4 KRBS Project End-Of-Project Evaluation school, for which special facilities were built and investments made under the KRBS Project, are not functioning on a regular basis. This is partly because they fall outside the GVN statutory remit for funding, but also because the opportunity to motivate the community to support the school through good management practices has been missed. As well as being an underutilization of an expensive asset, the protracted, reduced enrollment has the effect of also reducing the income from the GVN projected when the school was conceived. The activities that fall outside the GVN statutory remit for funding have not been sustained and the estimated future earnings from income generating activities has been consistently over optimistic. Much of the GVN funding of semi-boarding schools is related to the numbers of students in the school and the substantial shortfall in the actual enrollment over the original estimated enrollment has affected the school’s income. To design a school for an enrollment of 250 and only achieve 170 has had an impact on the education that the school can provide. Notwithstanding the school’s income-generating activities and parental contributions, the funds available from the GVN will only allow the KRBS to function as a semi￾boarding school for 180 students—70 students below the target enrollment. 1.4.2 Sustainability of Other KRBS Project Activities 1.4.2.1 Findings All components of the KRBS Project are now closed. Apart from KRBS itself, all the activities, including training, micro-credit, agricultural extension, and ECCE rehabilitation were designed and scheduled to be completed in July 2012. In the first phase of KRBS Project, EMW fully funded loans of 10 million VND over a period of 36 months,5 administrated by the People’s Committee of Dak Tu Lung to parents connected to the school. There were 28 households under the micro-credit subcomponent; each household borrowed 10,000,000 VND. All households invested in cows and two also invested in goats. However, the loans were too large and did not perform. The GVN, which closely monitors and regulates microcredit mechanisms, suggested that placing the WU in charge of the loan process would help to create more trust and would be more acceptable to the authorities. The Team visited two ECCEs, Hoa Mai Kindergarten and Nang Mai Kindergarten, that had benefitted from upgrading. Both were in suburbs of Kon Tum Town. Hoa Mai Kindergarten in Doan Ket Commune consisted of three campuses with very different levels of facilities. The core campus, on the road, was immaculate, with bright, clean classrooms and a wide range of playground equipment, as well as a new kitchen built by EMW. The building work appeared to be of high quality. Since the addition of a kitchen, enrollment has increased from 50 to 160 students. Enrollment was almost exclusively of Kinh ethnicity. Proper kitchens do not exist at either of the two campuses most used by the ethnic minorities. The teachers at one of these campuses travel to the main campus to get food for students. At the other, teachers cook for children on a single gas ring. Each of these campuses has a fluctuating enrollment of between 20 and 25 students. 5 Information of micro-credit households (VIE) and a sample of Need Assessment KRBS Project End-Of-Project Evaluation 5 The second campus visited, which had an exclusively ethnic minority enrollment, had not been selected by the Kon Tum authorities for improvement and was not, therefore, the responsibility of EMW. It was located in a compound that included a hospital and accommodation for lepers brought for treatment in Kon Tum. This campus was for the children of parents suffering from leprosy and the kindergarten was dilapidated. KRBS Project money had not been spent on renovating the facilities, as it had not been selected by the provincial authorities. 1.4.2.2 Conclusions Due to reductions in time allocated for field data collection, the Evaluation Team was not able to meet and interview recipients of micro-credit loans under the KRBS-Project. However, insights and statements that the Team gathered in interviews and discussions with EMW and a member of the WU, who participated in a meeting with the Kon Tum Provincial Departments, indicated that close cooperation with the government, and the selection of a trusted partner in the WU, had led to successful management of the micro-credit activity. Furthermore, using the WU to source funds through the social banks guaranteed the sustainability of this component. Two ECCE do not form a sample large enough to make any meaningful evaluation of the performance of this component. EMW, for many years, has supervised a range of construction and renovation activities in Vietnam and the work that the Team viewed was of a high standard. The evidence of teachers making toys and learning materials lead to the conclusion that integration of the training with the building rehabilitation is symbiotic. 1.4.3 Management of KRBS 1.4.3.1 Findings During the KRBS Project, the school reported to EMW after each activity, such as the teacher training, vocational training, and agricultural extension activities. They also provided annual reports to EMW. The Team was provided with 19 of the 25 progress reports submitted by EMW to USAID. The first is dated October–December 2006 and the last, the 25th progress report, April–June 2012. They are formulaic, often repeating entries from previous reports. The Team was unable to obtain any indication that USAID had responded to these reports in writing. One change in particular that underscores the management challenges associated with this agreement is EMW’s failure to enact most of its award obligations targeted at the inclusion and support of disabled children. Phase I had included access to education for disabled students as one of the project goals. EMW interpreted this in terms of physical access and duly included it in the construction brief. Phase II enhanced this component by adding access for children with disabilities to medical and educational services as well as proposing targeted training for teachers to work with children with disabilities. The last mention of access for disabled students in the progress reports is in July–Sept 2008, where it states under Output 1: “Facilities will be made accessible for disabled students; Indicators: door sizes, corridor widths, washroom dimensions and ramp access compliance with Americans with Disabilities Act (ADA) or regional standards. Achievement date: Completed.” While management of the construction of the school was transparent, adequately recorded and efficient, management of the ‘soft’ elements of the project was not. The ‘soft’ elements lacked a clear, mutual understanding of the objectives of the project between the two parties, who did not attempt to resolve the challenges through communication or documentation. 6 KRBS Project End-Of-Project Evaluation 1.4.3.2 Conclusions The lack of management continuity on both sides of the cooperative partnership posed difficulties in the Evaluation Team’s efforts to ascertain a clear and cohesive history of the project. Based on the above findings, the Team concluded that project management must have also suffered from this discontinuity. Phase 1 of the KRBS project – the construction of the school - was efficiently managed. EMW was aware of the need to sub-contract management of the micro-credit and agricultural practices components but in phases 2 and 3 the project management lost site of the wider objectives that had been brought into the project through the numerous CoAg modifications. While EMW cooperated well with the University of Danang in delivering the teacher training it did not ensure that key elements of the agreement such as the special emphasis on inclusion of children with disabilities were implemented. The rehabilitation of ECCE facilities in phase 3 was capably managed and efficiently executed. In short the construction components of the KRBS project were well managed but there was a lack of direction in the management of other constituents and the overall holistic vision of the project was missing. 1.4.4 Ethnic Minority Student’ Access to Education 1.4.4.1 Findings KRBS enrollment for the 2012–2013 school year was recorded at 170 students. The enrollment level has remained below capacity for the three years the school has been in operation. The first project proposal document stated that the KRBS originally was designed to serve three of the six communes in KRD. None of the individuals the Team interviewed or met with were able to explain why the school does not serve children from all six district communes, particularly in light of the school’s under-enrollment. Several provincial and district authorities, as well as KRBS management and teachers, attributed the under-enrollment to the number of school-aged children in the communes.6 At the same time, the majority of current students involved in the focus group discussions (FGDs) referenced siblings, neighbors, and friends who remained at home in their villages and were not attending school. Interviews with the District People’s Committee and the District Bureau of Education revealed the existence of a full boarding school in KRD Center since 1992. The Team spoke with EMW staff as well as the country director, none of whom reported to have been aware of this boarding school. KRBS management and teachers did not have data, reports, or even anecdotal information about student performance at KRBS. The only comment on student academic performance was made by the group of teachers the Team interviewed at KRBS, who reported that the performance of ethnic minority students was considerably lower than the performance of Kinh students. Teachers reported that the most significant outcome of ethnic minority students’ tenure at KRBS was their exposure to, and development of, modern life skills. 1.4.4.2 Conclusions In the absence of baseline data, current district-level statistics on student enrollment and progression rates, as well as data on the total number of schools in the district serving each grade-level, compared with the total number of ethnic minority children of school-going age at each age level, would have helped to inform the evaluation about possible changes in access resulting from the KRBS Project. Due to the lack of these key, secondary data sources, the Team relied heavily on interviews with provincial and 6 (BOET, 2012) KRBS Project End-Of-Project Evaluation 7 district authorities as well as with administrators, teachers, students, and parents of students from the KRBS. These key informants were unable to provide the Team with this basic information. Because student academic performance and educational outcomes are not monitored, it is not possible to assess the performance of KRBS in developing educational and employment opportunities for the ethnic minorities, or whether KRBS has had any effect on the academic achievements of ethnic minority students. The absence of reliable data on the basic indicators for access to education serve to compound the lack of unified understanding about the problem of ethnic minority students’ access to education, which appears to be poorly understood by the primary groups seeking to improve the problem. 1.4.5 Children With Disabilities Access to Education 1.4.5.1 Findings EMW was not able to provide the Team with information on the status of children with disabilities in KTP or KRD. Although EMW submitted a report on the state of children with disabilities in KRD in an annex to their second proposal to USAID, the Team was unable to obtain a copy of this report from either USAID or EMW. Both interviews and observations indicate that significant attention was given to the infrastructural design of the KRBS to better-accommodate students with disabilities. However, none of the respondents at either KRBS or EMW were able to tell the Team anything about the prevalence of children with disabilities, the nature of the most common disabilities, or the connection between KRBS design features and facilities and access to education for children with disabilities. 1.4.5.2 Conclusions The Evaluation Team was unable to determine why the project activities targeting children with disabilities were not carried out. If these activities were dropped in a formal manner, the changes were not documented. The Team was unable to establish where the demand for the facilities and special activities targeting children with disabilities was established. Though the EMW proposal for funding these activities cited a special study that was conducted in KTP on children with disabilities, EMW was not able to produce the report upon the Team’s request. 1.4.6 Teacher Training 1.4.6.1 Findings Professors from UD, in conjunction with staff from the Kon Tum DOET, conducted a training needs assessment (TNA) among teachers in KTP to ascertain the areas which required the most training. Professors reported that DOET staff provided them with a list of training topic areas for ECCE, primary school, and KRBS teachers. DOET instructed the professors to work from the list of topics to conduct the TNA. The goal of the TNA was to present the training topic areas to a sample of teachers to determine the topics in which they most required training. UD professors reported that the suggested list of training topics did not include techniques for teaching children with disabilities. Furthermore, professors reported that they were not informed that the training should focus on working with children with disabilities and none of the training curriculums they developed and delivered included a focus on this population. At the same time, USAID reported that they were not informed of changes to the 8 KRBS Project End-Of-Project Evaluation training curriculum and neither EMW project reports nor USAID agreement modifications indicated formal changes agreed upon by USAID. 1.4.6.2 Conclusions Because EMW did not monitor or seek to measure the outcomes of their training courses, they were unable to determine whether or not the training was appropriate and useful, nor whether teachers retained information, applied knowledge, or developed skills as a result of the training. Furthermore, KRBS teachers and management were unable to report whether teacher training affected student learning and educational outcomes. 1.5 SELECTION OF RECOMMENDATIONS The process of repeatedly adding, by modification, disparate components to an existing agreement should be avoided. In the future, a CoAg on human trafficking should not end up rehabilitating kindergartens and subcontracting agricultural extension components. The use of modifications to substantially change the objectives of a CoAg should be discouraged in the interests of clarity and focus of purpose. The KRBS Project began as an Anti-TIP CoAg funded from Cambodia, and evolved from a construction project into a program that incorporated agricultural extension and micro-credit as well as teacher training for early learning. Amending the CoAg at such a fundamental level strained the management of the project, increased the difficulties of proper monitoring, and complicated accounting systems. It also led the implementing partner into executing sub-programs in areas where it lacked expertise and experience. While varying an existing CoAg is a less bureaucratic option than creating a new agreement, care should be taken that the gains of such a course of action definitively outweigh the disadvantages that may follow. With the advances being made in broadening education opportunities and access being afforded to ethnic minorities under GVN Program 135 and the World Bank SEQAP (US$186 million, of which US$27.9 million came from the Vietnamese government and the rest financed by the WB), the findings of this evaluation lead the Team to believe that USAID’s resources would be better used outside the basic education sector. In the future, USAID and EMW should ensure that projects are designed and implemented with a comprehensive understanding of the problem they are seeking to address and a firm grasp of the local context. At the planning stage, a thorough needs assessment should always be conducted to inform the design of any development intervention. Following the needs assessment and the design of a project implementation proposal, comprehensive baseline measurements of all project indicators should be established to provide the foundation for both routine project monitoring and outcome analysis upon project completion. Performance monitoring plans (PMPs) should be required for all projects, particularly those with multiple follow-on proposals characterized by broadening scopes, objectives, and activities. Continued under-enrollment misses the opportunity for an additional 70 ethnic minority students to benefit from the superior facilities supplied at high cost. Through negotiation with GVN, USAID should encourage KTP and KRD officials to open the schools’ catchment area to include the other three communes in the district, in order to increase the enrollment rate and take advantage of the school’s full capacity. Close cooperation with organizations that have long established connections with provincial authorities (such as the WU and DU) greatly increases the likelihood of positive outcomes from initiatives in Vietnam, in general, and the Central Highlands, in particular. Partnering for implementation with KRBS Project End-Of-Project Evaluation 9 institutions that have strong and established connections with GVN authorities at all levels will help to ensure that projects meet their goals. 2 INTRODUCTION This report begins with a review of the education context in Vietnam, starting with an examination of national policies and structures, and refining the discussion to explore provincial and district-level policies and structures. The report then considers the socio-economic context of the Central Highlands region of Vietnam paying particular attention to the ways in which these policies and structures affect the education of ethnic minority populations such as those targeted and served by the KRBS Project. The report progresses with an in-depth description of the project, starting with an explanation of the design logic and transitioning into an appraisal of the KRBS Project background and context, the history of agreement modifications, and an analysis of the project’s evolving theory of change over the course of the three major phases of the project between 2006 and 2012. The project description concludes with an explanation of Vietnam’s boarding school policies and then explores two significant events that shaped the implementation of the project. Following the project description is an overview of the evaluation methods with a thorough description of the limitations placed on the evaluation. The report closes with an extensive discussion of the findings and conclusions for each evaluation question and a final section on comprehensive recommendations. 2.1 EDUCATION IN THE NATIONAL CONTEXT Vietnam has made great advances in the years since the U.S.-Vietnam Bilateral Trade Agreement on December 10, 2001. The country has met its Millennium Development Goal (MDG) of eradicating extreme poverty, with the number of its population living on less than US $1.25/day falling from 64 percent in 1992 to 16.85 percent in 2008.7 The gross domestic product grew by an estimated 5.9 percent in 2011,8 despite the global economic recession. By the World Bank’s definition, Vietnam is a “Middle-Income” country, whose average per capita income rose from $790 USD in 2007 to $1,168 in 20109—a 48 percent increase in three years. The GVN aspires to achieve a per capita income level of $3,000 (in current U.S. dollars) by 202010. Nevertheless, the country remains low in the United Nations (UN) Development Index ranking (128 out of 187 countries in 2011). According to the Canadian International Development Agency (CIDA), the low ranking is explained, in part, by the fact that “some sectors of the population, particularly the rural poor and ethnic minorities, do not benefit equally from economic growth.”11 The conclusion of the World Bank’s Country Analysis in 2009 was that while education is a factor, it is only one of a number of interdependent factors restricting development among the ethnic minorities in Vietnam. 7 2008 is the last year for which this poverty comparison is valid, Source explanation of poverty definitions as used by the GVN see § Ethnic Minority Policies and Practices page 9 8 Source http://www.indexmundi.com/g/g.aspx?c=vm&v=66 9 (U.S. Department of State, 2012) 10 (World Bank) 11 (Canadian International Development Agency, 2012) 10 KRBS Project End-Of-Project Evaluation In July 2003, the Prime Minister of Vietnam approved the country’s Education for All (EFA) Action Plan 2003–2015. 12 This framework set targets and provided a guide for cooperation between government, international organizations, bilateral donors, and NGOs toward achieving EFA goals. As a low-income country in 2003, Vietnam was eligible for assistance from the UN’s EFA Fast-Track Initiative (FTI), which matches national education strategies with donors’ efforts to accelerate progress towards MDG 2, universal primary education by 2015.13 In 2010, the United Nations Education, Scientific, and Cultural Organization (UNESCO) led a review and update of the country’s EFA action plan and produced the following recommendations: (1) ensure that the most disadvantaged have access to at least one year of quality preschool education, (2) strengthen Vietnamese language skills through mother-tongue and inclusive education and (3) improve the EFA quality assurance mechanisms through enhanced capacities of local education managers. The education structure in Vietnam aligns to international norms. It is divided into five levels: preschool, primary school, secondary school, high school, and higher education. Preschool is further divided into playgroup for ages two to four, and kindergarten for ages four to six. These levels are not compulsory, but are both highly encouraged and almost universally used where facilities are available. They are centered on the early childhood care and education centers (ECCE), which grew from the crèches associated with collective farming co-operatives that were closed in 1986.14 Primary school runs from first through fifth grades, normally for children ages six to 11. The Vietnam General Office of Statistics (VGOS) estimates an enrollment rate of 96 percent. Lower secondary school has four grades, sixth through ninth, for ages 11 to 14, and upper secondary supports three grades, tenth through twelfth, for ages 15 to 18. In 2006, the net enrollment of ethnic minorities at primary education level was 89 percent, while that of Kinh and Chinese was nearly 98 percent. A World Bank study15 estimates that over half of all children not in school come from the poorest fifth of the population, and half of that poorest fifth are from ethnic minority populations.16 The Ministry of Education and Training (MOET) has overall responsibility for education; the Minister of Education reports to the Prime Minister. The decision-making system is heavily decentralized in the provinces (departments of education) and districts (district offices of education). Both layers of government are also responsible to their respective People’s Councils. Vietnam’s ongoing decentralization of public finance accounts and service delivery is placing great demands on provincial governments. School management boards report to the Departments of Education and Training (DOET), which are responsible for monitoring standards of education, budgeting, and auditing. The GVN has made outstanding progress in education. In 1945, more than 95 percent of the adult population was illiterate. By the year 2006—after part 1 of Program 135, the socio-economic development program for the most vulnerable communes in ethnic minority and mountainous areas, had been implemented—the situation had been reversed and nearly 93 percent of adults were literate, according to the 2006 Vietnam Household Living Standards Survey (VHLSS) data. However, ethnic minority populations still lag behind their Kinh counterparts. In 2006, four percent of all children did not attend primary school, but that figure increased to 10 percent for ethnic minority children. Of all Kinh 12 Government Document No: 872/CP-KG Dated 02/07/2003 13 http://www.unesco.org/new/en/hanoi/education/education-for-all/ 14 Education Financing and Budgeting in Viet Nam; IIEP Publication 2001 15 (World Bank, 2009) 16 (World Bank, 2009) KRBS Project End-Of-Project Evaluation 11 and Chinese children, 97 percent were enrolled in primary school, 93 percent in lower secondary school, and 70 percent in high school. By comparison, ethnic minority enrollment saw 90 percent in primary school, 81 percent in secondary school, and only 44 percent in high school.17 Preferential Policies for Education of Ethnic Minorities have been applied by GVN and were still in force in 2012. These policies are presented in the following table. Table 1. Preferential Policies for Education of Ethnic Minorities18 Data in the World Bank policy study, Country Social Analysis Ethnicity and Development in Vietnam (2009),20 praises the GVN for progress made, but states that minorities continue to lag behind the Kinh and the government’s educational policies have not succeeded in closing the gap. It also points to the fact that differences in education achievement between minorities and Kinh remain substantial. There are several obstacles to the pursuit of equitable EFA groups in Vietnam. Kinh do far better than the rest of the ethnic groups in terms of time spent in school and in graduation rates. In 2009, ethnic minorities still suffered dropout rates almost twice those of their Kinh counterparts. There was no single reason cited by the World Bank, but it was attributed to “many, mutually reinforcing reasons: poverty, long distance to school, lack of self-esteem, language barriers, poor nutritional status, and high opportunity costs when children’s labor is needed at home.” Minority children generally enrolled later than Kinh. Authors of the policy study suggested21 policies to rectify the imbalances, including: (1) to increase the number of schools, thereby reducing the distance to 17 (VHLSS, 2006) 18 (World Bank, 2009) 19 These policies are subject to discretion in semi-boarding schools but applied universally in full boarding schools. 20 (World Bank, 2009) 21 Ibid POLICY APPLIED TO STUDENTS IN COMMUNITY SCHOOLS APPLIED TO STUDENTS IN FULL BOARDING SCHOOLS FOR ETHNIC MIORITIES19 School fees exemption or reduction All school fees exempted or reduced School fee exempted, scholarship for living expense Textbooks and notebooks provision Applied to Program 135 communes and border communes Scholarships, being entitled to borrow textbooks Direct appointment/ affirmative action (‘cu tuyen’) Minority quotas for upper secondary graduates to go on for further education and training (30–50 people/year/province) Boarding school pupils’ health insurance VND 35,000($2.25)/ year is paid; mosquito net provided; traveling fees granted, etc. Pupils learning in local districts’ boarding schools (at lower secondary level) Pupils learning in the provincial boarding school (at upper secondary level) Teacher allowance Teachers get salary and allowance for the stipulated hours teaching in the class, allowance for geographical region and other benefits Teachers get salary and allowance specified for special schools. Vocational training centers School fees exempted or reduced School fees exempted or reduced; each pupil is granted VND 20,000 ($1.29) for examination fees. 12 KRBS Project End-Of-Project Evaluation school; (2) to implement a more active kindergarten policy; (3) to improve bilingual education; and (4) increase the number of minority teachers. The World Bank study revealed that the most common reason why children drop out of school is because the household lacks money for school fees, particularly for informal charges and fees for supplies22. “Increasing subsidies to secondary education in ethnic minority boarding schools and preschool kindergartens appear to be policy alternatives.” 23 The problem of ethnic-minority access to education among those who live in remote areas has given rise to a particular system of boarding schools that supplement the general education system in Vietnam: fully subsidized boarding schools at the provincial level and “semi-boarding” schools at district level. During the 1990s, the government earmarked considerable sums to support education for mountainous and minority areas, which led to the construction of approximately 400 boarding schools. By 2001, 43 provincial boarding schools, 10 central boarding schools, 190 district-level boarding schools, and 104 boarding schools operated at the commune level (as of 2006, there were more than 500).24 Boarding Schools, under provincial supervision, must cater to upper-secondary pupils. Each province must contain at least one boarding school. Pupils reside there for the full two terms of each level. All costs are met by the GVN. Students receive free tuition, accommodation and textbooks, and live full-time at the school during the school year Semi-boarding schools, under district supervision, are set up for ethnic minority children whose families have settled permanently in economically disadvantaged areas. Enrollment must include more than 50 percent of pupils from ethnic minorities and at least 51 percent of the students must be boarders (49 percent can attend daily and live at home). A semi-boarding school receives an annual state grant, used to supply and maintain housing, kitchens, dormitories, bathrooms and sanitation facilities, and a water supply. It also receives a per capita allowance to buy and repair fitness equipment, sports equipment, musical instruments, televisions to serve cultural activities, and medicines for health. 2.2 ETHNIC MINORITY POLICIES AND PRACTICES Fifty-three officially recognized ethnic groups reside in Vietnam. Eighty-five percent of the population identifies as ethnic Kinh, who dominate the political, economic and social affairs of the country. The remaining 15 percent is composed of groups that range from over a million to just a few hundred people, mostly concentrated in the uplands of the North and the Central Highlands. The equality and rights of every ethnic person living in Vietnam has been clearly stated, at the highest level, in the 1992 constitution. The Prime Minister’s Directive 525/TTg of November 1993, on the continuation of socioeconomic development of mountain areas, and Decision 656/TTg in 1996, on the socioeconomic development of the Central Highlands, still provide an overall policy framework for the accelerated development of mountainous areas and areas where ethnic minority populations reside. Among other points, it advocates the reinforcement of the existing education system, the adaptation of the education and training programs to local conditions, and the encouragement of informal education efforts. 22 (World Bank, 2009) 23 (World Bank, 2009) page 101 24 (World Bank, 2009) KRBS Project End-Of-Project Evaluation 13 In 2005, the average family annual income in KTP was between $110 and $120 per year (the poverty line nationwide was then set at $240 per family) and educational attainment was very low. It was likely that no more than 10 percent of local students finished 10th grade. Access to schooling for ethnic minorities presented a major challenge. Vietnam’s high-quality household surveys25 measure poverty in three different ways. The first is by the cost of a food and non-food consumption basket, with the food component providing a daily intake of 2,100 calories per person per day. Individuals are considered poor if their daily per capita expenditure is not enough to afford this basket, and extremely poor (or “food poor”) if their daily per capita expenditure is not enough to purchase this amount of calories, were they to spend all their expenditure on food. The second measure, used by the Ministry of Labor, Invalids and Social Affairs (MOLISA), is determined by the number of households that fall beneath a given income, disaggregated into urban and rural. However, in practice, local MOLISA officials determine which households fall under these poverty lines through a mix of methods, including village discussion, surveys, and local officials’ personal knowledge. Therefore, these poverty lines can vary across administrative units and can be affected by subjective judgment. The third approach is to use an international poverty line measured in Purchasing Power Parity (PPP) dollars, which can be converted to the local currency through comparable international price surveys.26 The GVN, through the authority of the Committee for Ethnic Minorities (CEM), has made efforts to integrate ethnic minority populations into mainstream Vietnamese economic life. Its key initiative, Program 135, with an allocated capital of Vietnamese Dong (VND) 14,000 billion ($67.2 million) in Phase II (2006–2010) helped to raise the living standard of ethnic minorities and populations living in remote, mountainous areas. The initiative effectively reduced the poverty rate from 47 percent in 2006 to 28.8 percent in 201027 (the MOLISA method of poverty assessment is used to determine allocations within the education system).28 It provides decentralized assistance to communes with high ethnic populations and also provides educational support to poor students. 2.3 CENTRAL HIGHLANDS SOCIO-ECONOMIC CONTEXT Sixty-one percent of ethnic minority peoples were still poor in 2004 compared with only 14 percent of the Kinh and people of Chinese ethnicity. In 2006, as the KRBS Project began, a CEM/Multidimensional Poverty Index meeting on Ethnic Minority Poverty took place in Hanoi. A paper prepared for this meeting, “Explaining Ethnic Minority Poverty in Vietnam,” reported that the low enrollment rates among ethnic minority populations for lower secondary school in the late 1990s were improving, but past low￾enrollment rates resulted in a population of young ethnic minority adults with significantly inferior educational attainment, compared with their Kinh counterparts. The paper stated that “this may well be perpetuating the dominant view that ethnic minorities are ‘backward’.” The problem is grounded, in part, by the deficiency of the education system to deliver a service that produces comparable educational outputs for ethnic minority populations and the Kinh. In 2009, under the School Education Quality Assuarance Project (SEQAP), the World Bank produced a Policy Framework For Ethnic Minority Development. It observed that “ethnic minorities in Vietnam now represent 39 percent of all poor people, despite making up only 14 percent of the country’s population.” 25 Produced by Vietnam Household Living Standards Survey 26 (Dang, November 2009; revised January 2010) 27 Vietnam: Successful reduction in poverty under Programme 135 Posted on January 12, 2011 by BJ Murphy 28 (DOFA, 2012) 14 KRBS Project End-Of-Project Evaluation 3 PROJECT DESCRIPTION 3.1 PROJECT LOGIC EMW’s project managers and country directors collaborated with the DOET to informally assess the educational situation of ethnic minorities, visiting KTP several times to study their needs and shortfalls in the broadest context29 (in 2005, no systematic survey of the population of KTP or KRD had been carried out). EMW concluded that, because the majority of ethnic minority populations lived in remote, often mountainous areas, with poor or nonexistent transportation infrastructure, poor educational outcomes among ethnic minority students could, in large part, be attributed to the logistic or financial difficulties of access to school. The result of this informal research and discussions with the DOET was a concept note, on which the original proposal for the KRBS Project was based. Building small schools to serve scattered, low-density villages had often proved ineffective. Distances from homes to school were too great for many students to walk twice a day. It was difficult to find teachers to work in remote areas, and most targeted students did not speak Vietnamese as their first language.30 Boarding schools, a solution preferred by local educational authorities because it was in line with MOET policies, would create better learning environments with improved educational outcomes, including a lowering of the drop-out rate, improved academic performance, improved promotion rates (from lower secondary to upper secondary), and better preparation for entering the workforce. The KRBS Project aimed to address the financial and logistical challenges of non-boarding schools, by establishing a self-financing, semi-boarding school that, through sound financial management, would reduce the burden of the financial and social cost of school to ethnic minority families while also addressing the costs associated with commuting. 3.2 PROJECT BACKGROUND EMW reported31 that prior to USAID involvement, the Kon Tum People’s Committee, in coordination with the University of Da Nang, which had been tasked with improving educational outcomes in Kon Tum, asked EMW to assist by constructing a boarding school in the province. EMW prepared a concept note to build a boarding school in KTP and felt, following initial discussions with the DOET, which had facilitated some preliminary studies, that it could obtain both permission from the GVN and funds from Atlantic Philanthropies32 to build a boarding school in KTP.33 The province said it welcomed the funding for this purpose in KRD because of the lack of infrastructure and facilities there. The province proposed that this semi-boarding (bán trú) school should be able to receive 250 students with classrooms, dormitories, kitchen, library, laboritory, and working areas for teachers and board management. The province asserted that if the donor could provide the construction costs for the school, the province 29 (EMW, Program Development Director, East Meets West Foundation, 2012) 30KRBS MOA 3: 30 September 2006 Attachment A; Program Description Kon Ray Boarding School page 12 31 Information in this paragraph was supplied at the briefing held between the Evaluation Team and EMW in the Danang office on 28 September 2012 32 http://www.atlanticphilanthropies.org/ 33 Tu, N. T. M., 2012. Program Development Director, East Meets West Foundation [Interview] (13 October 2012). KRBS Project End-Of-Project Evaluation 15 would guarantee land law procedures, costs for ground cleaning, meals for students, salary for teachers, and costs relating to operation when the school was in place.34 On June 8, 2006, USAID issued a Request for Applications (RFA) to establish programming in the Central Highlands, “to implement a program to support raising the economic status and social welfare of the indigenous peoples of the Central Highlands of Vietnam.” EMW responded with the KRBS Project proposal, which met the needs of the district as expressed in the DOET’s official letter to EMW regarding the construction of a semi-boarding school in KRD.35 In the early 2000s, very few international nongovernmental organizations (INGOs) were authorized by the GVN to work in the Central Highlands, but the East Meets West Foundation (EMW) had been working with the ethnic minority communities in Vietnam since 1988 on a variety of construction projects, including hospitals, kindergartens and primary schools. As an organization with a track record of successful construction projects in this particularly challenging region, EMW was a good partner for USAID. The Evaluation Team was not able to ascertain at what stage EMW abandoned the idea of cooperating with Atlantic Philanthropies. 3.3 HISTORY OF PROJECT AGREEMENT MODIFICATIONS As a modification to the Anti-Trafficking in Persons (Anti-TIP) CoAg already in place between EMW and USAID, EMW was awarded additional funding for the KRBS Project to increase access to education for ethnic minorities and the disabled children in KTP. The modification that constituted this new program was issued for the period of September 2006 through September 2008, under a cost-share arrangement with USAID providing funding in the amount of $512,906 and EMW contributing $127,989, for a total funding level for the KRBS Project of $640,894. On June 23, 2006, the GVN set out its initial contribution to the project in an official letter36 responding to EMW’s proposal. In August 2006, the GVN gave further details of its commitment to the KRBS Project in an official letter from DOET to EMW37. The province planned the detailed budget as follows: 1. Annually operation cost for 12 months, for a total of 250 students, at the minimum rates in accordance with the current regulations: a. Teacher, staff and management board salary (for 20 personnel, 1.7 million VND per month): 408 million VND b. Books and notebooks for teachers and students: 30 million VND c. Teaching equipment: 50 million VND d. Electricity cost: 12 million VND e. Administrative cost: 100 million VND f. Meals for students (290,000 VND per student per month): 870 million VND g. Management cost: 20 million VND h. Health care at school: 50 million VND i. Guard: 12 million VND j. Other costs (clothes, physical education costs): 80 million VND 2. Costs for ground cleaning, garden, gate and fencing 34 Official letter of DOET to EMW Ref. 489/CV-SGD&DT to respond to EMW’s proposal on construction of semi-boarding school in KRD (VIE) 35 Ibid (VIE) 36 DOET to EMW Ref.489/CV-SGD&DT to respond to EMW’ s proposal on construction of semi-boarding school in KRD (VIE) 37 DOET’s letter to EMW on detailed budgets of the province in the KRBS Project. KRBS Project, Ref. 621/CV-SGD&DT (VIE) 16 KRBS Project End-Of-Project Evaluation Beginning with the original ANTI-TIP Co-Ag, a total of 12 modifications, many of them incremental funding actions, were made to the agreement over its lifespan. Eight of the modifications were related to the KRBS Project, while the remainder dealt with the ANTI-TIP agreement. Among the eight KRBS Project modifications, three were driven by proposals for additional funding, generated by EMW, and submitted to USAID. On September 27, 2007, EMW submitted a proposal to USAID requesting an additional $229,897 to expand the program beyond the construction of the boarding school. USAID approved this proposal, which constituted the sixth agreement modification and launched Phase II of the project. On July 31 2009, USAID modified the agreement for the ninth time to provide $227,505 in additional funding for EMW’s third proposal, in order to pursue project activities focusing on sustainability. For a comprehensive overview of the modifications of this agreement, see Annex B: KRBS Project Cooperative Agreement Modifications. 3.4 THEORY OF CHANGE IN AN EVOLVING PROJECT Over the course of its lifetime, the KRBS Project developed into three phases, which combined both ‘hard’ and ‘soft’ components to reach certain change objectives. The hard components concerned construction and renovation—bricks and mortar—while the soft components included capacity building, training, poverty reduction, and improvement of agricultural practices. Access for the disabled to education remained a cross-cutting issue throughout each phase of the project. Phase I evolved primarily as a construction project, with the provision of opportunities for local families to improve their incomes and farming techniques as a secondary objective. In this phase, the problem of disabled children’s access to education was also understood to be an infrastructural issue, which could be addressed in the design of the school facilities. To accommodate the education of children with disabilities, the inclusion of wheelchair access was an integral part of the building design.38 The objectives were to (1) improve educational outcomes for disadvantaged, ethnic minority students in KRD; (2) develop valuable skills in students to enable them to become productive members of society; (3) improve access to education for disabled students in KRD; and (4) create opportunities for local families to improve their incomes and farming techniques. The primary theory of change in Phase I of the KRBS Project posited that building a boarding school, where children would not be required to commute on a daily basis, would serve to increase ethnic￾minority children’s access to lower-secondary education. The KRBS Project hypothesized that by providing dormitories, students would be able to stay on-site at the school and an improvement in class attendance rates and reduction in drop-out rates would result, therefore leading to greater student educational attainment. The school would include a cafeteria, with meals three times daily. An improved nutritional status, it was hypothesized, would also enhance students’ academic performance. Finally, building new, free dormitory facilities for teachers would attract and retain qualified teachers, which in turn would improve the quality of lower-secondary education. Phase I also proposed that (1) training teachers how to teach students who are physically disabled would increase educational and vocational opportunities for the disabled; and that (2) increasing available micro-credit for parents and (3) improving agricultural practices in KRD would increase household incomes, removing poverty barriers to education and resulting in increased secondary 38 See below § Phase I: Access to Education for Disabled Students KRBS Project End-Of-Project Evaluation 17 school attendance. The hypothesis supporting improved attendance rates and reduced drop-out rates due to boarding was later questioned. Boarding schools were criticized in some quarters as an inefficient solution when local educational infrastructure still lags behind. Nonetheless, they were seen as appealing to local education officials as a means of achieving the education goals of the government.39 In Phase II, the KRBS Project took on less construction, but retained the emphasis on training academic and vocational teachers at KRBS and continued to expand the agricultural extension training in the district. At the request of USAID, improving access to medical and educational services for children with disabilities was given substantial weight. Phase II primarily was designed to address the non-infrastructure issues of disabled children’s access to education. The challenge was defined primarily in terms of teachers’ ability to effectively instruct children with disabilities, but also in terms of supplementary assistance with medical needs and assistive devices. The 2008 USAID Vietnam Central Highlands Needs Assessment Report identified children in especially difficult circumstances (CEDC) as one of the five main vulnerable groups in the Central Highlands.40 The theory of change for this phase proposed that providing teachers with highly specialized training would better prepare them to understand the needs of, and prepare them to assist, students with disabilities. Phase I had defined the problem of disabled children’s access to education purely in terms of an individual’s physical ability to attend school. Phase II hypothesized that providing children with disabilities with appropriate and targeted educational instruction, good medical care, and assistive devices would enhance these children’s chances of receiving a quality education. The KRBS Project’s theory of change for Phase III focused on the problem of KRBS’ financial sustainability and proposed that implementation of income-generating activities for the school would close the financial gap between GVN’s subsidies and KRBS’ operating costs, leading to the school’s financial sustainability. In addition to addressing the issue of financial sustainability in Phase III, at the request of the Kon Tum Provincial Authorities,41 the KRBS Project also broadened its scope, shifting its focus from activities related specifically to the KRBS, to a program with more generalized support for improving basic education in KTP. Enhanced teacher training at KRBS continued and the KRBS Project expanded to include the development of twenty-five ECCE across the province expanding into five new districts, providing teacher training to ECCE staff and renovating twenty-five kindergartens. The logic of Phase III further proposed that enhancing ECCE through a renovation program in the province would strengthen the early-learning substructure of the education system. 3.5 BOARDING SCHOOL OR SEMI-BOARDING SCHOOL Based on a thorough review of project documents and interviews with USAID and EMW, the Evaluation Team concluded that USAID believed it was funding a full boarding school. Furthermore, the Team was unable to discern whether or not USAID was aware of the significant differences in funding support from the GVN depending on the semantic difference between “boarding school” and “semi-boarding school”. Nowhere in the original application42 is the phrase ‘semi-boarding’ used in reference to the proposed school at Kon Ray. In the official response of DOET Kon Tum to EMW’s proposal letter, the provincial authorities asked for funding of a semi-boarding (bán trú) school and a lab center for KRD. While GVN 39 (World Bank, 2009) 40 (Clark, 2008) 41 (EMW, Staff Discussions, 2012) 42 Application to USAID for Ethnic Minority Boarding School in Kon Tum, Vietnam USAID-RDM/A-Vietnam-486-06-011-RFA. Submitted by the East Meets West Foundation 18 KRBS Project End-Of-Project Evaluation requested a semi-boarding school and EMW planned to build a semi-boarding school, all discussions with USAID included reference to a “boarding school.” The school’s status as a semi-boarding school versus a full boarding school was not properly considered at the project design phase. The school therefore was adopted as a semi-boarding school and has received funding from GVN according to that status. This has placed large financial implications for the amount of GVN funding that would be available at the end of the CoAg to continue supporting the KRBS. 3.6 EXTERNAL FACTORS AFFECTING THE KRBS PROJECT According to EMW,43 two external factors significantly affected the KRBS Project: inflation and the removal of two of the three communes in the school’s catchment area previously classified as poor communes. Vietnam experienced rapidly fluctuating inflation rates during the implementation of the KRBS Project. In January 2006, at commencement of project implementation, the rate was approximately eight percent and fell to near six percent at the beginning of 2007. By the middle of 2008, it had reached 28 percent; this occurred just as the main work on the construction component was being implemented. By the end of 2009, the rate fell to three percent, but climbed, more or less steadily, to approximately 23 percent in July 2012, when the KRBS Project closed.44 The Evaluation Team was told by EMW45 that the inflation rate of 28 percent in 2008 particularly affected the price of building materials and the transportation of those materials to KTP, where access was not easy. According to EMW, this inflation was a factor behind EMW’s action to seek and receieve incremental funding from USAID on six occasions between September 2007 and December 2011. As inflation was outside EMW’s control, they explained that this justified the large difference between the original project budget estimates and the actual costs. The second factor affecting the KRBS Project involved the removal of two of the three communes from the KRBS catchment area. At the time the original proposal was written, all three communes within the catchment area were classified as extremely difficult (the lowest poverty rating), and therefore were supported comprehensively under the government poverty policies— Program 135, in particular. However, every three years the government reviews each communes’ classification to determine whether economic circumstances have improved. A change in poverty status among two of the school’s three feeder communes reduced subsidies under Program 135.46 In effect, the success of Program 135 had taken these communes from ‘extremely difficult’ to ‘poor’ as defined by MOLISA. This meant that children from these two communes no longer qualified for certain subsidies under the regulations for student funding.47 It also meant that the Kon Tum People’s Committee approved the school as a semi-boarding only and it did not qualify for full boarding-school status.48 This change allowed the school to receive subsidies from GVN for salary and teaching activities, but not to cover the costs of boarding students. The classification from full to semi￾boarding school substantially reduced the school’s projected income and caused it to rely more heavily on 43 (EMW, Program Development Director, East Meets West Foundation, 2012) 44 (Trading Economics) 45 (EMW, Staff Discussions, 2012) and (EMW, Program Development Director, East Meets West Foundation, 2012) 46 (EMW, Staff Discussions, 2012) and (BOET, 2012) 47 For details of the regulations for student funding see § Funds received from the page 18 48 See Table 1 KRBS Project End-Of-Project Evaluation 19 its own resources. EMW estimated the primary cost no longer covered by this change in subsidy, food for the students and teachers, to be approximately $30,000 per year. 4 KRBS PROJECT EVALUATION DESIGN AND METHODOLOGY 4.1 EVALUATION FOCUS While in Vietnam between September 24 and October 16, 2012, the SI Evaluation Team focused on project activities in the period from the inception of the KRBS Project in September 2006 through completion of the project in July 2012. At the planning stage and on arrival in Vietnam, the Team had been asked by USAID/Vietnam to examine four key evaluation topics.49 1. KRBS Project Sustainability 2. Sustainability of other program components 3. Has Phase I met its stated goals and objectives? 4. Has Phase II (cost extension) met stated goals and objectives? The evaluation of question one involved an examination of financial sustainability, with a particular emphasis on the schools’ income-generating activities; management sustainability, with an appraisal of the KRBS’s management team and staff and their ability to maintain and operate the school in terms of planning, curriculum, training, personnel, procurement, and long-term strategy; technical sustainability, which considers whether the school is designed with the appropriate systems and tools, maintenance personnel, and resources to repair and service the facility at a high standard; and environmental sustainability, including and assessment of school grounds and resources for water supply, waste disposal, erosion control, and landscaping. The evaluation of question two demanded an examination of the activities and objectives associated with teacher training, which primarily focused on the provision of special skills to teach children with disabilities; an assessment of the refurbishment and teacher training activities and objectives targeting the 25 ECCEs throughout KTP; an investigation of the agricultural extension trainings for farmers in KRD and parents of students attending KRBS; and an analysis of the micro-credit loan and early childhood education promotion program facilitated by the Provincial Women’s Union (WU). Questions three and four were concerned with understanding whether or not ethnic minority and children with disabilities’ access to education had been effected by the project, whether or not educational outcomes for disadvantaged, ethnic minority students in Kon Ray District improved as a result of the KRBS Project, whether or not KRBS students acquired technical skills as a result of the project’s focus on practical trade skills such as carpentry and sewing, whether or not local families’ incomes and farming techniques had improved, whether or not the KRBS was a sustainable entity, whether or not 25 ECCEs had been refurbished, and whether or not teacher’s capacity to work with disabled, ethnic minority students had been improved. 49 The detailed Evaluation Focus as planned is given in Annex C: Evaluation Limitations and Change of Evaluation Scope 20 KRBS Project End-Of-Project Evaluation 4.2 EVALUATION METHODOLOGY The evaluation investigated the performance of KRBS Project activities by applying a mixed-methods approach, combining descriptive and analytical techniques to answer strategic design and performance assessment questions to understand what happened and why. The evaluation was organized into four phases: (1) planning, document review and finalization of evaluation plan; (2) conducting the evaluation; (3) data analysis and oral presentation; and (4) reporting and dissemination of results. The following data collection methods were applied at various phases of the evaluation: Review and analysis of information from key project documents and reports, as well as background data sources, such as World Bank and UNICEF studies Semi-structured, key informant interviews (KII) Semi-structured, group interviews Focus group discussions (FGD) Direct field observations The list of persons contacted is included in Annex F: Evaluation Key Informants and Stakeholders. 4.2.1 Review of Project Documents and Background Reports USAID and EMW shared a selection of project documents with the Team, which the Team closely consulted and used to inform the design of the evaluation. The Team also requested several additional documents from EMW, only some of which they were provided.50 Key contractual documents, such as the agreement modifications, provided the Team with the foundational history of the project. Project documents, including proposals and annual reports, were examined to compile the list of key project stakeholders and beneficiaries. The Team independently collected background reports and documents, such as World Bank and GVN policy documents, to improve their understanding of the state of education and ethnic minority populations in Vietnam. Several documents critical to the evaluation were impossible to obtain, either because they did not exist, or simply were not provided to the Team, including budget proposals, monitoring reports, education statistics, baseline surveys and special studies, such as the one conducted by EMW on the People With Disabilities in KRD. 4.2.2 Semi-Structured Key Informant and Group Interviews Overlaying the four evaluation questions identified by USAID with the background reports and project documents, the Team compiled an all-inclusive list of key informants and developed a comprehensive evaluation plan, which can be found in Annex D: Vietnam KRBS Project Evaluation Plan. The evaluation plan identified which key informants were able to inform each of the areas of inquiry and accompanying sub-questions. The Team assembled a matrix linking key informants with evaluation questions, from which were formulated semi-structured questionnaires and FGD guides for each individual and group. Interview guides were broken down by question area, i.e., the sustainability of KRBS management, with several sub-questions to cover different aspects of management sustainability. Following interviews and group discussions, the Team reflected on and discussed the details of the interview to corroborate findings and build consensus on emerging conclusions and recommendations. Specific time was also set aside for evaluation members to synthesize, compile and refine their notes. The evaluation design included informants from stakeholder and beneficiary groups at all levels of the project, from the feeder 50 See Annex G: Documents Requested from EMW KRBS Project End-Of-Project Evaluation 21 villages and communes of KRBS to the National Ministry of Education. A complete list of the key informants, stakeholder, and beneficiary groups targeted in this evaluation is included in Annex F: Evaluation Key Informants and Stakeholders. The Team’s original evaluation methodology focused on collecting primary data from project stakeholders and beneficiaries in KTP over a two-week period. Key informant and group interviews were planned with each of the critical departments within both the provincial and district governments, including those listed in Annex F: Evaluation Stakeholders and Key Informants. Interview and focus group discussion guides can be found in Annex E. 4.2.3 Focus Group Discussions The original evaluation plan included FGDs with several stakeholder and beneficiary groups; in practice, due to data collection limitations (discussed in a separate section) FGDs only included current students at the KRBS. The Team conducted eight FGDs, four with male students and four with female students. FGDs were comprised of three students from each grade level in each group. The Team divided into two sub-teams, with the women facilitating the FGDs with the female students and the men facilitating the FGDs with the male students. 4.2.4 Direct Field Observations The primary objective of the first and second phases of KRBS Project was to build the infrastructure for the KRBS and to upgrade the educational facilities of 25 ECCEs. Field observations were thus an important component of the evaluation methods. Due to significant limitations on the time available for data collection, direct observation focused primarily on the KRBS. The Team spent two days at the KRBS, with limited time to tour the facilities and observe whether key features of school construction, such as the irrigation system and vocational classrooms, were being maintained properly and served their intended purpose. The Team visited the student dormitories, classrooms, kitchen, dining hall and administrative buildings. 4.3 LIMITATIONS ON THE EVALUATION The Evaluation Team faced considerable restrictions due to the significant reduction of time allowed by the Kon Tum provincial authorities for field-level data collection. The original evaluation proposal and the contract between USAID and Social Impact (SI) included two full weeks of data collection in KTP. One-week prior to deployment to Vietnam, the Team was informed by USAID that KTP provincial authorities had not approved the evaluation proposal for two weeks of data collection in KTP, and instead stipulated that only six days would be allowed. Upon arrival in Hanoi, Vietnam, the Team met with USAID staff at the Mission and re-confirmed the revised evaluation schedule that included six days for primary data collection in KTP. Two days later, when the Team was at the Hanoi airport departing for central Vietnam, USAID informed the Team that KTP authorities had further reduced the approved time for data collection in KTP from six days to three days. The Evaluation Team made additional revisions to the schedule based on three days of data collection according to a clear understanding between USAID/Vietnam and SI that the limitations would significantly hinder the Team's ability to fully and adequately answer all evaluation questions included in 22 KRBS Project End-Of-Project Evaluation the TOR.51 Full details of the limitations to this evaluation can be found in Annex C: Evaluation Limitations and Change of Evaluation Scope. In addition to the substantial limitations on time allocated for data collection, the Team was also limited by their restricted mobility within KTP. Originally the Team had planned to visit several districts throughout the province where EMW had refurbished ECCEs and worked with the WU to provide early education promotion to remote, ethnic minority populations. Upon arrival in KTP, the Team was informed that they would not be allowed to travel outside of Kon Tum Town to any of the districts except for Kon Ray, where the KRBS is located. The Team was only allowed to visit two ECCEs in Kon Tum Town, which were pre-selected by the provincial authorities. Another noteworthy limitation to the evaluation was the planned overlap of the USAID-commissioned evaluation of the KRBS Project, led by the SI Evaluation Team, with EMW’s internally-commissioned four-member evaluation team (MoE Team), led by four members of the National Ministry of Education. Upon arrival in Vietnam, the SI Evaluation Team was informed during their in-briefing with USAID on September 25, that USAID had recommended EMW to schedule their internal evaluation to take place during the same time as USAID’s evaluation to minimize the imposition on the Provincial and District Authorities.52 The Team was limited by the number of key informants from the major provincial ministries and authorities who participated in the official meeting on the first day of data collection in KTP. Rather than meeting in consecutive small groups with each office and ministry as the Team had prepared to do (applying the various data collection instruments that had been developed for each group of informants), the Team was restricted to engaging with key individuals from the Education Promotion Association (EPA), Department of Education and Training (DOET), the Provincial Women’s Union (WU), Department of Planning and Investment (DPI), and the Department of Foreign Affairs (DOFA) in a collective group setting. In addition to the challenges associated with interviewing more than 30 individuals at the same time, the Team was further limited by the presence of the MoE evaluation team that was competing with the SI Team for opportunities to ask their distinct evaluation questions to the same group of key informants. The Team’s access to reliable, primary information from key respondents was further limited and likely influenced due to the mandate that both EMW staff and officials from DOET accompany the Team on all of their visits, interviews, and focus group discussions, not to mention the presence and competing interest of the MoE Team. The original evaluation design was based on the understanding that the Team would have control over sampling and selection of informants; however, officials at the KRBS selected the majority of informants for both the interviews and group discussions including teachers, and school staff. 4.4 FINDINGS 4.4.1 Overarching The KRBS is a functioning, semi-boarding school with superior facilities and a competent staff providing access to lower secondary education for 170 lower-secondary students from ethnic minority populations. 51 See Amendment of Solicitation Modification of Contract:: AlD-486-TO-12-00002/ AlD-RAN-I-00-09-00019; dated 09/27/2012 52 (USAID, 2012) KRBS Project End-Of-Project Evaluation 23 It employs 27 staff and is fully funded by the GVN according to the appropriate prime ministerial directives and regulations pertaining to its semi-boarding school status. The school complies with GVN standards and receives technical inspections in accordance with GVN practice—at least one per year. The KRBS Project has demonstrated to GVN at district, provincial, and central levels the advantages of collaboration with INGOs that can provide capital expenditure to education projects that then are transferred to government ownership for operational costs. The GVN has assumed responsibility for the maintenance of the grounds and buildings. The EMW engineer, previously the project manager during the construction phase of the project, visited the school with the Team and accompanied Team members in an inspection of the facilities. The water supply worked well, with only minor plumbing repairs needed. The school grounds were clean and the gardens well maintained. Since the end of the KRBS Project, the EMW engineer has been under no obligation to visit or inspect the school, but has done so on an informal basis when other work has taken him to the area. The management staff did not report how or if they are planning for continued maintenance and upkeep of the school facilities. School Management reported that they did not receive management training to support the transition process between EMW and GVN, nor had they been prepared for the completion of the KRBS Project and its final handover. Two members of the school management board did not know that the KRBS Project had closed. This contradicted the information given verbally to the Evaluation Team by EMW, who stated that handover training had been provided.53 School management reported that they had received some management training from the DOET, but the Team was not able to establish the frequency or content of DOET training. The Team observed that the school’s superior facilities were not operating at full capacity. The school has two vocational classrooms, one for sewing and one for electrical studies, but the school lacks permanent teachers to support these classrooms. People with the appropriate skills are hired from the community to provide courses at various times throughout the year. These courses are for Grades 8 and 9 and the classes run four or five times a week over the course of four to five months. Inspection of the vocational training classrooms showed that the equipment had not been used in the recent past.54 The KRBS receives financial support from three sources: (1) the GVN, (2) income-generating activities and (3) parental or community support. Specifically, the school takes its statutory allocations of funds from GVN, together with special support under Program 135, as detailed below. 4.4.2 Funds received from the GVN Neither the school management nor the DOET shared accounting records of government funding for the school with the Evaluation Team. Both bodies assured the Team that the KRBS receives its full statutory funding, including funds from Program 135. (The funding of all schools in Vietnam, but particularly boarding schools, is complicated because money comes from many separately accounted sources. Being a semi-boarding school, KRBS is covered by three directives: (1) public financing for 53 EMW, 2012 54 The failure to meet target enrollment is discussed in Error! Reference source not found. 24 KRBS Project End-Of-Project Evaluation education; (2) policies on semi-boarding students and semi-boarding schools; and (3) operational regulations on semi-boarding schools.55) Student funding is provided under four main regulations: (1) tuition fees and financial support for students; (2) policies on students; (3) policies on boarding students; and (4) policies on semi-boarding students.56 Teachers are remunerated under four GVN regulations: (1) policies on salaries in the public sector; (2) financial support for teachers; (3) policy on teachers working in remote areas; and (4) policy on teachers in very remote areas.57 The GVN has been officially financing the school as a public, government-operated, semi-boarding school since it started admitting students in February 2009.58 The enrollment rates for KRBS have been 2008/09-154; 2009/10-186; 2010/11-172; 2011/12-160; 2012/13-170. This is in light of a recommendation from the GVN that the school should be built with a capacity of 250 students. 4.4.3 Income-Generating Activities In the original proposal, KRBS was to include an agricultural learning center within the school grounds. Its primary purpose was to provide a facility where students could learn basic and advanced techniques in locally appropriate agriculture. Its secondary purpose was to “produce food for the school, and perhaps income if the produce [could] be sold in local markets.”59 The success of the GVN’s poverty-alleviation policies throughout the country led to a change in the rating of the economic conditions of the three communes served by KRBS Project. In late 2008, only 40 students from the poorest commune of Dak Koi received full subsidy. As a consequence, the school had to look elsewhere for the financial resources to support the enrollment of 250 students. In order to make up for the anticipated shortfall in income, EMW changed the emphasis of the component from agricultural learning to income generation. By the time EMW’s 2011 Annual Report for the project was published, the concept of the agriculture learning center had been dropped from the progress reports altogether and the report stated, “The KRBS had conducted several income-generating activities to cover its project deficit as well as provide food for 172 students.” The KRBS farming activities no longer have any of the characteristics of an agricultural learning center. The Team was unable to obtain any evidence that the small-scale vegetable production on the school grounds was being used for any purpose other than providing minor supplements to school meals. The KRBS has pursued several income-generating activities including raising pigs, poultry, and fish; constructing a nursery garden, growing corn and dragon fruit; hosting educational trainings; and supporting voluntourism.60 Rubber production has been proposed, and a suggested bio-gas plant was rejected by EMW experts. Currently, pig farming and nursery gardening are being practiced. 55 Interview with MOET 56 Ibid 57 Ibid 58 Ibid 59 2006 EMW Proposal 60 Voluntourism is travel which includes volunteering for a charitable cause. KRBS Project End-Of-Project Evaluation 25 In discussions with the teachers, the Team was told that teachers do not participate in income-generating activities, citing long hours and distances to travel home at the end of the school day, which keeps them from these extra curricula activities. Income generation at KRBS lacks conventional bookkeeping procedures and the Team was unable to obtain information on the calculations of the profits or losses incurred by particular activities. The financial information gleaned by the Team is summarized below: 1. In 2009, EMW invested VND 419,032,00061 in dragon fruit production. It was anticipated to generate VND 72,309,000 per annum. No income was recorded. The assistant controller of EMW stated that the activity had been ‘ineffective’.62 2. Since 2009, when VND 93,570,000 was invested in pig production, recorded income totals VND 174,604,000, with costs of VND 120,452,000—a profit of VND 54,152,000. School management stated that VND 33,220,000 was spent on New Year gifts to students and VND 18,000,000 on computer screens (total expenditure VND 51,220,000), leaving a balance of VND 2,932,000 over three years. 3. The school garden contributed the equivalent of VND 4,000,000 per annum in vegetables for feeding the students. 4. No initial expenditure was recorded for the chickens, but an income of VND 10,850,000, from which VND 9,200,000 costs was subtracted, left a balance of VND 1,650,000 unspent over an unspecified period. 5. Nursery plants showed no initial expenditure recorded, but an income of VND14,500,00 was recorded and spent as follows: VND 9,000,000 on the Students’ Association, VND 2,000,000 on water for the football team, VND1,000,000 for “Support staff go along with football team,” and VND 2,000,000 for “Gifts for heroes and poor families,” leaving a surplus of VND500,000. 6. “Voluntourism,63” produced VND101, 529,605. This money was paid as a donation to EMW from Road monkey.64 7. Hosting of teacher-training sessions generated VND 135,200,000 in 2010; VND 135,200,000 in 2011; and VND 142,900,000 in 2012. There is no record of how this was spent. According to these figures, total income (not profit) over the last three years has been VND 714,783,605. Under GVN regulations, this amount can be spent at the school management’s discretion. The school principal plans to continue with the pig farming and nursery garden programs and wants to develop a rubber plantation on one or two hectares of the school’s property. School management stated that they have no plans to continue with the teacher-training program now that KRBS Project funding has ceased.65 Despite the fact that the KRBS no longer incorporates the learning activities advanced in the original proposals into the income generating activities, the students are still contributing to the farming. The Team could not ascertain the extent of students’ input, as no records of students’ working hours in support of these activities were supplied. 61 In August 2009 USD 1.00 = VND 17,367: VND 419,032,000 = USD 2,4128.05 62 Discussions between assistant controller of EMW and Evaluation Team members in Danang 10/10/12 63 Voluntourism is travel which includes volunteering for a charitable cause. 64 Roadmonkey is a U.S.-based, for-profit organization with a social mission. It partners with nonprofit organizations on adventure expeditions to ensure that its volunteer projects are meaningful and sustainable to local community in need. 65 KRBS, 2012 26 KRBS Project End-Of-Project Evaluation Due to the opaque nature of the income-generating-activity accounts, the Evaluation Team was unable to establish what, if anything, this component of the project has contributed to school finances. It can state, however, that any contributions (profit over investment) made so far, appear to be insignificant. The two activities that have generated substantial income with high profitability are voluntourism and the hosting of teacher trainings. Voluntourism has not been repeated since the one-time episode in November 2009 and according to EMW, who arranged the activity, it is unlikely to be repeated due to the global economic downturn. Without EMW as an intermediary, the Provincial DOET explained at a meeting on 1 October 2012, any contact with NGOs working in KTP must be through the Provincial Department of Foreign Affairs (DOFA), and it is unlikely that the activity could be arranged by the school in the future without EMW’s support. The best income generator has been the hosting of teacher training, but this too is unlikely to continue. Despite the presence of sleeping, eating, and conference facilities, seminars at the KRBS were unpopular with the teachers who participated, as the school lacks communication facilities and is too remote. More significantly, payments for the trainings were made by EMW, using KRBS Project money under the training component that ceased with the completion of the KRBS Project. The DOET has not held training courses at the school and, given the costs, is unlikely to do so in the future. 4.4.4 Parental and Community Support During the initial stages of the KRBS Project, substantial local contributions were made to the school, including $129,000 (USD) in cash for leveling the school grounds and construction. In addition, seven households in Dak To Lung Commune voluntarily donated 19,000m2 of land to build the school. By definition,66 the parents of students at KRBS have limited resources. They are, however, required to make contributions in rice for feeding the students. This is currently levied at 10 kg of rice per child per term. The majority of parents with children at KRBS contribute rice for feeding the students; the remaining small minority, not quantified by the school management committee, pay cash in lieu thereof. 4.5 CONCLUSIONS During its brief time at the school, the Evaluation Team was given no cause to doubt that the school was providing a positive education to the students. There appeared to be good staff/pupil relationships and the atmosphere was conducive to learning with appropriate classroom infrastructure and resources. The school is functioning according to the standards set by the GVN, but the complimentary activities at the school, for which special facilities were built and investments made under the KRBS Project, are not functioning on a regular basis. This is partly because they fall outside the GVN statutory remit for funding, but also because the opportunity to motivate the community to support the school through good management practices has been missed. The school is therefore successful in providing a normal lower secondary education for a GVN semi￾boarding school but in the original concept the project envisaged that this was to have been augmented 66 Exact numbers of parents of children at KRBS defined as poor are not available. For an explanation of GVN see § Ethnic Minority Policies and Practices page 4 KRBS Project End-Of-Project Evaluation 27 by vocational training activities and specialized tuition for disabled students given by teachers specifically trained in these skills. Kon Ray Boarding School was to be a center of excellence serving as a model in addressing the particular disadvantages of the Central Highlands ethnic minorities. The funding for these activities that would supplement normal lower secondary education would come from vocational production, farming, and parental contribution. These income generating activities have underperformed or ceased and the education given by the school is therefore not augmented above that of other GVN semi-boarding schools. As well as being an underutilization of an expensive asset, the protracted, reduced enrollment has the effect of also reducing the income from the GVN that was projected when the school was conceived. The activities that fall outside the GVN statutory remit for funding have not been sustained and the estimated future earnings from income generating activities has been consistently over optimistic. Much of the GVN funding of semi-boarding schools is related to the numbers of students in the school and the substantial shortfall in the actual enrollment over the original estimated enrollment has affected the school’s income. To design a school for an enrollment of 250 and only achieve 170 has had an impact on the education that the school can provide. Notwithstanding the school’s income-generating activities and parental contributions, the funds available from the GVN will only allow the KRBS to function as a semi￾boarding school for 180 students – 70 students below the target enrollment. 5 SUSTAINABILITY OF OTHER KRBS PROJECT ACTIVITIES All components of the KRBS Project are now closed. Apart from KRBS itself, all the activities, including training, micro-credit, agricultural extension, and ECCE rehabilitation were designed and scheduled to be completed in July 2012. 5.1 MICRO-CREDIT LOANS 5.1.1 Findings In the first phase of KRBS Project, EMW fully funded loans of 10 million VND over a period of 36 months,67 administrated by the People’s Committee of Dak To Lung to parents connected to the school. There were 28 households under the micro-credit subcomponent; each household borrowed 10,000,000 VND. All households invested in cows and two also invested in goats. However, the loans were too large and did not perform. The GVN, which closely monitors and regulates microcredit mechanisms, suggested that placing the WU in charge of the loan process would help to create more trust and would be more acceptable to the authorities. In the second phase, therefore, EMW worked with the WU on this component, identifying target groups in accordance with KRBS Project goals. EMW played an advocacy role and provided funding, borrowed from social banks. EMW trained the micro-grant recipients in early childhood care and community development, on the supposition that women who received micro-credit would be encouraged to send their children to the ECCE while working at their own small businesses. This idea was taken up by the Education Promotion Association (EPA) as part of their campaign for preschool enrollment in kindergartens. 67 Information of micro-credit households (VIE) and a sample of Need Assessment 28 KRBS Project End-Of-Project Evaluation 5.1.2 Conclusions Due to reductions in time allocated for field data collection, the Evaluation Team was not able to meet and interview recipients of micro-credit loans under the KRBS-Project. However, insights and statements that the Team gathered in interviews and discussions with EMW and a member of the WU, who participated in a meeting with the Kon Tum Provincial Departments, indicated that close cooperation with the government, and the selection of a trusted partner in the WU, had led to successful management of the micro-credit activity. Furthermore, using the WU to source funds through the social banks guaranteed the sustainability of this component. Through cooperation with the WU on microcredit for poor parents, the project demonstrated the benefits of working through GVN systems and organizations to improve living standards for the ethnic minorities, particularly in terms of advocacy. 5.2 AGRICULTURAL EXTENSION 5.2.1 Findings EMW subcontracted all agricultural extension and capacity-building activities, outside the income￾generation and food activities within KRBS itself, through the Improved Livelihoods for Mountainous Communities (ILMC) Project, funded by the Canadian Centre of International Development and Cooperation (CECI). Training was conducted by the Kon Tum Veterinary Department of the Ministry of Agriculture and Rural Development. 5.2.2 Conclusions Agricultural trainings took place in districts to which the Team did not have access. The sustainability of this activity could, therefore, not be evaluated. 5.3 EARLY CHILDHOOD CARE AND EDUCATION CENTERS 5.3.1 Findings In line with GVN policy of encouraging the development of ECCEs and the recommendations of the World Bank,68 the second phase of the KRBS Project advanced two strategies for improving the enrollment of children at the ECCEs. The first created teacher-training programs for primary school and ECCE teachers in KRD; the second rehabilitated 25 kindergartens within ECCEs based in nine districts across the province. Twenty-five ECCE were selected by the GVN to demonstrate to their communities what could be done to improve facilities and rehabilitate kindergartens (kindergartens are fee-paid institutions in Vietnam). The Team visited two ECCEs, Hoa Mai Kindergarten and Nang Mai Kindergarten, that had benefitted from upgrading. Both were in suburbs of Kon Tum Town. Hoa Mai Kindergarten in Doan Ket Commune consisted of three campuses with very different levels of facilities. The core campus, on the road, was immaculate, with bright, clean classrooms and a wide range 68 (World Bank, 2009) KRBS Project End-Of-Project Evaluation 29 of playground equipment, as well as a new kitchen built by EMW. The building work appeared to be of high quality. Since the addition of a kitchen, enrollment has increased from 50 to 160 students. Enrollment was almost exclusively of Kinh ethnicity. Proper kitchens do not exist at either of the two campuses most used by the ethnic minorities. The teachers at one of these campuses travel to the main campus to get food for students. At the other, teachers cook for children on a single gas ring. Each of these campuses has a fluctuating enrollment of between 20 and 25 students. The second campus visited, which had an exclusively ethnic minority enrollment, had not been selected by the Kon Tum authorities for improvement and was not, therefore, the responsibility of EMW. It was located in a compound that included a hospital and accommodation for lepers brought for treatment in Kon Tum. This campus was for the children of parents suffering from leprosy and the kindergarten was dilapidated. KRBS Project money had not been spent on renovating the facilities, as it had not been selected by the provincial authorities. The second ECCE visited, Nang Mai Kindergarten, Le Loi Ward, Kon Tum City, displayed a full range of upgraded facilities and a large collection of teaching aids and materials, created by teachers who had received EMW training. 5.3.2 Conclusions Two ECCE do not form a sample large enough to make any meaningful evaluation of the performance of this component. EMW, for many years, has supervised a range of construction and renovation activities in Vietnam and the work that the Team viewed was of a high standard. The evidence of teachers making toys and learning materials lead to the conclusion that integration of the training with the building rehabilitation is symbiotic. The Kon Tum Provincial Authorities selected the ECCEs that the Team visited. The Team was not provided with the criteria used for selection. EMW was not involved in the process of selecting the ECCEs the Team visited. The Team did have reservations, however, regarding the criteria that were used to target the renovations of ECCEs. The Team was unable to obtain these criteria from the Kon Tum Provincial Authorities. 6 MANAGEMENT OF THE KON RAY BOARDING SCHOOL PROJECT The Team evaluated two spheres of management: (1) EMW’s management of the KRBS Project and (2) USAID’s management of the CoAg and implementing partner, EMW. 6.1 PROJECT MANAGEMENT 6.1.1 Findings One staff member at EMW remained in position throughout the KRBS Project: the engineer responsible for the construction aspects of EMW projects. In the original management plan included in Modification of Agreement 3, the project was described in this way: “very simple and straightforward and uses the same format as the one used on hundreds of other projects each year.” The plan went on to say, “Once the project is completed, EMW will turn the buildings over to the Kon Tum DOET. This will happen 30 KRBS Project End-Of-Project Evaluation approximately ten months after the project is approved and funding made available. From that point forward, all management duties will be handled by the Kon Tum DOET.” EMW assigned two managers to the KRBS Project: a project supervisor (PS) and a program manager (PM). While the PS looked after the construction, the PM remained responsible for developing the vocational training program, the income-generating program, and the agricultural technical assistance program. The PS stayed with the project throughout the six years; the PM was moved to a different posting within EMW. A second PM took over for two years until 2008, a third until 2009, and a fourth until the KRBS Project closed.69 During this six-year period, the project significantly transformed as reflected in the several modifications to the CoAg.70 USAID told the Team71 that, in the initial stages of the project, EMW’s original reporting did not meet requirements, so USAID developed a template that included an annual work plan, an M&E plan, and indicators.72 USAID did limited monitoring of its own—“only informal checking”—although they did request field-monitoring reports.73 During the KRBS Project, the school reported to EMW after each activity, such as the teacher training, vocational training, and agricultural extension activities. They also provided annual reports to EMW. The Team was provided with 19 of the 25 progress reports submitted by EMW to USAID. The first is dated October–December 2006 and the last, the 25th progress report, April–June 2012. They are formulaic, often repeating entries from previous reports. The Team was unable to obtain any indication that USAID had responded to these reports in writing. Over its lifespan, substantial changes to the project implementation were made. The Team was unable to verify whether or not these changes were made in collaboration with USAID. It is not known whether matters were discussed between EMW and USAID verbally, agreed upon, and then enacted, or whether EMW made these changes unilaterally and these alterations went unnoticed through lack of USAID oversight. One change in particular that underscores the management challenges associated with this agreement is EMW’s failure to enact most of its award obligations targeted at the inclusion and support of disabled children. Phase I had included access to education for disabled students as one of the project goals. EMW interpreted this in terms of physical access and duly included it in the construction brief. Phase II enhanced this component by adding access for children with disabilities to medical and educational services as well as proposing targeted training for teachers to work with children with disabilities. The last mention of access for disabled students in the progress reports is in July–Sept 2008, where it states under Output 1: “Facilities will be made accessible for disabled students; Indicators: door sizes, corridor widths, washroom dimensions and ramp access compliance with Americans with Disabilities Act (ADA) or regional standards. Achievement date: Completed.” The Evaluation Team was not able to obtain documentation or clear information about how the project was managed, particularly regarding the project activities targeting children with disabilities. Only two of the KRBS Project’s four designers still work for EMW. They both assisted the Team with interviews. 69 (EMW, Staff Discussions, 2012) 70 See § Theory of Change in an Evolving Project: page 8 71 (USAID, 2012) 72 Reference to meeting with USAID on arrival 73 (USAID, 2012) KRBS Project End-Of-Project Evaluation 31 None of the project managers from USAID who were involved in overseeing the project from 2006 to 2012 were available for interviews. The current EMW program development director, together with the current EMW President, the recently-retired Vietnam Country Director, and the first project manager, were the designers of the KRBS Project. The Evaluation Team was only able to interview the second project manager, who succeeded the original project manager, as well as the original country director. 6.1.2 Conclusions The lack of management continuity on both sides of the cooperative partnership posed difficulties in the Evaluation Team’s efforts to ascertain a clear and cohesive history of the project. Based on the above findings, the Team concluded that project management must have also suffered from this discontinuity. Phase 1 of the KRBS project – the construction of the school - was efficiently managed. EMW was aware of the need to sub-contract management of the micro-credit and agricultural practices components but in phases 2 and 3 the project management lost site of the wider objectives that had been brought into the project through the numerous CoAg modifications. While EMW cooperated well with the University of Danang in delivering the teacher training it did not ensure that key elements of the agreement such as the special emphasis on inclusion of children with disabilities were implemented. The rehabilitation of ECCE facilities in phase 3 was capably managed and efficiently executed. In short the construction components of the KRBS project were well managed but there was a lack of direction in the management of other constituents and the overall holistic vision of the project was missing. 6.2 PARTNERSHIP MANAGEMENT 6.2.1 Findings From 2006 to 2012, the KRBS Project experienced a large number of changes in personnel. USAID moved through three contracting officer’s representatives over the six years, the last of whom had left before the Team arrived in-country.74 EMW implemented a new and more appropriate management structure to the project as it evolved from mostly construction to a training and capacity-building program. There were four project managers during this time. This lack of institutional memory, combined with incomplete documentation at both USAID and EMW, posed a challenge to the evaluation. The KRBS Project designers were John Anner (current EMW President); Nguyen Thi Minh Tu, (currently program development director for EMW); and Mark Conroy (retired country director and current senior advisor). The first project manager was Mr. Quyen who had been succeeded as project manager by Nguyen Thi Minh Tu, who had in turn handed over responsibility to the current deputy director of EMW Vietnam Vo Thi Hien. Nguyen Thi Minh Tu and Mark Conroy gave extensive interviews to the Evaluation Team. Understandably, none of the project managers from USAID was available for interview. While management of the construction of the school was transparent, adequately recorded and efficient, management of the ‘soft’ elements of the project was not. The ‘soft’ elements lacked a clear, mutual understanding of the objectives of the project between the two parties, who did not attempt to resolve the challenges through communication or documentation. 74 Ms. Brett Jones (2006–2008), Mr. Howard Handler (2008–2011) and Mr. Jay Kryck (2011–July 2012) 32 KRBS Project End-Of-Project Evaluation 6.2.2 Conclusions Given the challenges of working in KTP, pragmatic changes were inevitable as the project progressed, but the changes were not sufficiently documented and tracked to allow for appropriate project management. The reports submitted by EMW to USAID were prescriptive and did not give information on the progress towards the objectives of the project or its contribution in providing educational opportunities to ethnic minorities in Kon Tum Province. Apart from an initial criticism by USAID of EMW’s reports the Team found no evidence that USAID had reacted to the reports or referred to the CoAg to see that funds were being spent according to the conditions of the supplements. As a result, components that had been mutually agreed in the modifications to the CoAg were not implemented and USAID did not draw EMW’s attention to this deficiency. The management teams on both sides of the partnership were responsible for the complete omission of certain elements such as improving access to medical and educational services for children with disabilities. 7 ETHNIC MINORITY STUDENTS’ ACCESS TO EDUCATION 7.1 FINDINGS 7.1.1 Baseline Data The RFA required chosen grantees to collect baseline data and to develop a performance-monitoring plan to track and measure program outputs and outcomes. The Evaluation Team found that EMW did not carry out a baseline assessment to determine the existing situation for ethnic minority students’ access to education or academic performance in KRD prior to designing and implementing the project. The Team made several attempts to collect this information from provincial officials at the DOET, district officials in Kon Ray, and from the KRBS management staff; all efforts were unsuccessful. Furthermore, the Team was unable to obtain any recent statistics on the status of ethnic minority children’s access to education and academic performance in either KTP or KRD, from evaluation key informants. 7.1.2 School Enrollment KRBS enrollment for the 2012-2013 school year was recorded at 170 students. The enrollment level has remained below capacity for the three years the school has been in operation. The first project proposal document stated that the KRBS originally was designed to serve three of the six communes in KRD. None of the individuals the Team interviewed or met with were able to explain why the school does not serve children from all six district communes, particularly in light of the school’s under-enrollment. Several provincial and district authorities, as well as KRBS management and teachers, attributed the under-enrollment to the number of school-aged children in the communes.75 At the same time, the majority of current students involved in the focus group discussions (FGDs) referenced siblings, neighbors, and friends who remained at home in their villages and were not attending school. Interviews with the District People’s Committee and the District Bureau of Education revealed the existence of a full boarding school in KRD Center since 1992. The Team spoke with EMW staff as well as the country director, none of whom reported to have been aware of this boarding school. 75 (BOET, 2012) KRBS Project End-Of-Project Evaluation 33 The Team was unable to obtain verifiable information as to whether or not the KRBS has affected any change in the enrollment rate of ethnic minority children in KRD. The Team was unable to determine whether any of the current or past students at KRBS had previously attended, or tried to attend, the full boarding school in KRD Center, or whether they had previously been attending school at all. 7.1.3 Academic Performance KRBS management and teachers did not have data, reports, or even anecdotal information about student performance at KRBS. The only comment on student academic performance was made by the group of teachers the Team interviewed at KRBS, who reported that the performance of ethnic minority students was considerably lower than the performance of Kinh students. Teachers reported that the most significant outcome of ethnic minority students’ tenure at KRBS was their exposure to, and development of, modern life skills. EMW followed graduates for four years after their completion of ninth grade at KRBS. Between 2008 and 2012, a total of 172 students graduated from KRBS. Among the graduates, 25 students progressed to the 10th grade, one attended university, one went on to college, and one enrolled in a vocational training school. This information was communicated verbally to the Team and was not available in documentation. EMW reported that they could not track or monitor the subsequent academic performance of graduated KRBS students, as this could only be done through MOET channels not open and available to them. 7.1.4 Barriers to Education The majority of parents who participated in the guided group discussion agreed that education is good for their children, but stressed that, without government financial support and subsidies; they would not be able to afford to send their children to school. Teachers reported that deficient Vietnamese-language skills among ethnic minority children presented the greatest challenge to increasing their access to education. Teachers stated that the majority of students at KRBS struggle to communicate in Vietnamese and often revert to speaking in their mother tongues when not required to use Vietnamese.76 The Team’s FGDs with current students support this finding regarding language skills, as all of the students, regardless of grade level, experienced difficulty communicating with the Vietnamese Team members. All of the provincial and district officials, as well as KRBS management, cited distance to school facilities as the number one barrier to education for ethnic minority students. Many informants from these groups reported that even a distance of one kilometer would present an obstacle for children to attend school. Driving between Kon Tum Town and the KRBS, the Evaluation Team counted 20 schools along the main road, which covered a distance of approximately 35 kilometers. 7.2 CONCLUSIONS Based on the parents’ statements about barriers to education, it’s possible that the semi-boarding school model, where GVN covers the cost of students’ boarding and parents are only required to provide a rice or cash contribution, encouraged the enrollment of students at KRBS. 76 (Teachers, 2012) 34 KRBS Project End-Of-Project Evaluation To measure the effect of the KRBS Project on ethnic minority students’ access to education and academic performance, baseline data on students’ access to education prior to the construction of the KRBS is essential. Likewise, routine monitoring data on basic indicators such as attendance, promotion, drop-out, and academic performance are critical to understanding the value of the KRBS Project. In the absence of baseline data, current district-level statistics on student enrollment and progression rates, as well as data on the total number of schools in the district serving each grade-level, compared with the total number of ethnic minority children of school-going age at each age level, would have helped to inform the evaluation about possible changes in access resulting from the KRBS Project. Due to the lack of these key, secondary data sources, the Team relied heavily on interviews with provincial and district authorities as well as with administrators, teachers, students, and parents of students from the KRBS. These key informants were unable to provide the Team with this basic information. Because student academic performance and educational outcomes are not monitored, it is not possible to assess the performance of KRBS in developing educational and employment opportunities for the ethnic minorities, or whether KRBS has had any effect on the academic achievements of ethnic minority students. The absence of reliable data on the basic indicators for access to education limits understanding about the problem of ethnic minority students’ access to education, which appears to be poorly understood by the primary groups seeking to improve the problem. While each of the barriers to education cited by the key groups may be significant, the inability of respondents to support their statements with reliable information does not lend particular support to one theory over another and thus does not bode well for further investments in ethnic minority education in KRD. Moreover, it is difficult to know whether KRBS has had any impact on improving access to education by addressing these perceived barriers. Parents are required to provide payments in the form of rice or cash in order for their children to attend the school. None of the teachers themselves originate from ethnic minority populations, and thus they remain unable to communicate with students in their mother tongues. Teachers reported that they received very little training regarding the challenge of working with students from ethnic minority populations.77 Discussions with University of Danang (UD) professors also revealed that training courses were not designed to provide teachers with this skill set, indicating a major disconnect between the goals of the program and the way in which certain activities were implemented. Officials and KRBS management were unable to report with confidence how far the students’ homes are from the school, and whether students previously were attending other lower￾secondary schools before the construction of KRBS. 8 CHILDREN WITH DISABILITIES’ ACCESS TO EDUCATION The first two of the project’s three phases included a substantial focus on assisting children with disabilities. Goal 3 of the first phase was to improve access to education for disabled students in KRD through building all of the KRBS facilities to accommodate children with disabilities and through training teachers in how to instruct students with disabilities. The second phase contained the greatest emphasis on supporting children with disabilities, which included expanded support for specialized teacher training 77 (Teachers, 2012) KRBS Project End-Of-Project Evaluation 35 at the KRBS, primary schools and ECCEs in KRD as well as the direct provision of medical assistance, such as prosthetics and physical therapy, to children with disabilities. 8.1 FINDINGS KTP is among the provinces with the highest prevalence rate of people with disabilities in Vietnam;78 however, according to KRBS management and teachers, no children with disabilities have ever attended the KRBS. The KRD People’s Council reported79 that all of the children with disabilities were of primary school age and that there were no children with disabilities at the lower-secondary-school age within the district. Authorities from the District Bureau of Education informed the Team that in 2011, there were 79 students with disabilities at the primary school level in KRD and 714 students with disabilities at the primary school level across KTP. The Team was unable to obtain documentation of these figures to confirm their accuracy. During a site visit to the school, the Team observed that one of the girls involved in the student FGDs had considerably malformed hands. EMW was not able to provide the Team with information on the status of children with disabilities in KTP or KRD. Although EMW submitted a report on the state of children with disabilities in KRD in an annex to their second proposal to USAID, the Team was unable to obtain a copy of this report from either USAID or EMW. In the second proposal (2007), one of the four stated goals was to “improve access of children with disabilities to medical and educational services.” The activities to be included were to provide children who need them with (1) medical assistance; (2) prosthetics; (3) assistive devices; and (4) physical therapy. The Team could find no evidence that any of these activities had been executed and were unable to find any discussion of these activities in the progress reports submitted by EMW to USAID. Both interviews and observations indicate that significant attention was given to the infrastructural design of the KRBS to better-accommodate students with disabilities. However, none of the respondents at either KRBS or EMW were able to tell the Team anything about the prevalence of children with disabilities, the nature of the most common disabilities, or the connection between KRBS design features and facilities and access to education for children with disabilities. When questioned about the design features of the school, both teachers and management referenced the ramps and the braille signage.80 Teachers reported that educational materials in braille were not available. The boarding school was built on a steep hillside, with the buildings and facilities located on four separate terraces. An extensive ramp was constructed to allow general access to each of the four areas of the school; however, not all of the buildings were specifically wheelchair accessible. Apart from KRBS infrastructural features, there is little evidence that the project activities targeting children with disabilities were implemented. None of the respondents that the Team interviewed, including the project manager and CEO of EMW, were aware of the covenants in the CoAg on the subject of children with disabilities. 78 The Team found it possible to find overall national statistics on disabilities, but enquiries to specific localized data was unforthcoming. In an email request to UNICEF the Representative for Vietnam replied, “statistics are scant and unreliable when it comes to disabled children and especially in more hard to reach areas.” 31 December 2012. 79 (BOET, 2012) 80 (KRBS, 2012) 36 KRBS Project End-Of-Project Evaluation KRBS teachers involved in the guided group interview reported that they had not received training on working with children with disabilities. Professors from UD in charge of the teacher-training component of the project reported that training materials did not cover teaching children with disabilities. When questioned about the project activities focused on providing children with prosthetics and physical therapy, EMW project managers and directors reported that those activities were associated with a different project and that there were no activities targeting children with disabilities under this agreement. In response to further questioning about the apparent absence of these project activities, the EMW project manager reported that funding from these activities was transferred to cover increased construction costs incurred by the project. The Evaluation Team found that the absence of these activities was not flagged as a concern by USAID. 8.2 CONCLUSIONS The Evaluation Team was unable to determine why the project activities targeting children with disabilities were not carried out. If these activities were dropped in a formal manner, the changes were not documented. The Team was unable to establish where the demand for the facilities and special activities targeting children with disabilities was established. Though the EMW proposal for funding these activities cited a special study that was conducted in KTP on children with disabilities, EMW was not able to produce the report upon the Team’s request. At the same time, it is unclear what role KRBS has played in meeting the district’s alleged need to provide accessible education to disabled students. The fact that no disabled students have been registered and enrolled at KRBS during its time of operations is discussed elsewhere in this report.81 To this extent, the Team was unable to ascertain what, if any, special considerations were made in designing KRBS and its associated activities to increase enrollment of children with disabilities. 9 TEACHER TRAINING 9.1 FINDINGS According to EMW proposals, teacher training was to focus primarily on improving the capacity of teachers to work with students with disabilities. The development and delivery of specialized teacher￾training courses was a central component of each of the three project phases. EMW subcontracted with UD to implement the teacher training associated with both the second and third phases of the project. The Team found UD to be a strong partner in the training activities and a good choice for developing education in KTP, specifically. UD has a close association with DOET in Kon Tum, as UD’s primary satellite school is located in the central town of KTP. Furthermore, UD is a partner of long standing with DOET for teacher training across the country. Professors from UD, in conjunction with staff from the Kon Tum DOET, conducted a training needs assessment (TNA) among teachers in KTP to ascertain the areas which required the most training. Professors reported that DOET staff provided them with a list of training topic areas for ECCE, primary 81 See § 8, Children with Disabilities’ Access to Education KRBS Project End-Of-Project Evaluation 37 school, and KRBS teachers. DOET instructed the professors to work from the list of topics to conduct the TNA. The goal of the TNA was to present the training topic areas to a sample of teachers to determine the topics in which they most required training. UD professors reported that the suggested list of training topics did not include techniques for teaching children with disabilities. Furthermore, professors reported that they were not informed that the training should focus on working with children with disabilities and none of the training curriculums they developed and delivered included a focus on this population. At the same time, USAID reported that they were not informed of changes to the training curriculum and neither EMW project reports nor USAID agreement modifications indicated formal changes agreed upon by USAID. UD subsidized the majority of their contributions to the project by providing the curriculum design for free and conducting their training courses at subsidized rate. UD professors reported that all training materials and curriculum were provided to DOET upon completion of the project; however, they could not say whether DOET had any plans to use the curriculum in the future. The Team conducted a semi-structured, group discussion with a selection of teachers from KRBS who participated in the training course designed and delivered by professors from UD. The teachers received training from UD over the course of three summers. The training sessions covered topics such as how to apply IT skills in teaching and how to integrate positive reinforcement into teaching methods. Although the courses did not cover working with students with disabilities, teachers reported that they liked the trainings and found them to be very useful. EMW reported to the Team that they did not monitor training outcomes. 9.2 CONCLUSIONS EMW did not undertake a systematic approach to identifying training needs and did not ensure that those training topics identified in the agreement were carried out in practice. The Team was unable to determine where the disconnect occurred between training topics proposed by EMW, and the actual training UD professors provided. It was also unclear what DOET’s role in the decision-making process was regarding the pre-selection of training topic areas. Because EMW did not monitor or seek to measure the outcomes of their training courses, they were unable to determine whether or not the training was appropriate and useful, nor whether teachers retained information, applied knowledge, or developed skills as a result of the training. Furthermore, KRBS teachers and management were unable to report whether teacher training affected student learning and educational outcomes. The lack of forward thinking regarding the use and future application of training materials was a shortfall of the project. Given UD’s close association with DOET, greater focus on sustainability could have been fostered by transference of materials and methods designed under the project to the appropriate provincial authorities for further implementation after the project finished. 10 OVERARCHING RECOMMENDATIONS The autonomy of decentralized decision making in the Central Highlands allows for specific negotiation with district and provincial authorities to produce beneficial and flexible outcomes. Similar projects in the Central Highlands should leverage this flexibility in the system so that issues such as broadening the catchment area of the school are discussed and policy issues resolved to maximize the value of USAID contribution. 38 KRBS Project End-Of-Project Evaluation The management of micro-credit is strictly regulated by GVN and its implementation should be carefully discussed and agreed with all the relevant authorities before being introduced as a component of any project. There should be more clarity between CoAg partners accurately defining what USAID is funding and, more specifically, whether that funding has been used for the purpose agreed in the CoAg. Furthermore, any changes to the original project activities and scope should be well documented. Reporting systems for future agreements should be upheld and implemented according to the parameters of the agreement. While recognizing the need for flexibility to complete projects in challenging circumstance, management decisions made under CoAgs should be clearly documented and recorded. The process of repeatedly adding, by modification, disparate components to an existing agreement should be avoided. In the future, a CoAg on human trafficking should not end up rehabilitating kindergartens and subcontracting agricultural extension components. The use of modifications to substantially change the objectives of a CoAg should be discouraged in the interests of clarity and focus of purpose. The KRBS Project began as an Anti-TIP CoAg funded from Cambodia, and evolved from a construction project into a program that incorporated agricultural extension and micro-credit as well as teacher training for early learning. Amending the CoAg at such a fundamental level strained the management of the project, increased the difficulties of proper monitoring, and complicated accounting systems. It also led the implementing partner into executing sub-programs in areas where it lacked expertise and experience. While varying an existing CoAg is a less bureaucratic option than creating a new agreement, care should be taken that the gains of such a course of action definitively outweigh the disadvantages that may follow. CoAgs should only be signed with organizations that have demonstrated skills implementing the specific fields covered by the agreements. While it is clear that EMW has superior expertise in construction, several of the “soft” development components did not perform according to plan. The most successful components of the project were those that fell within the established expertise of the EMW and where EMW recognized its own inexperience and brought in other organizations to implement those components. EMW recognized its high level of competence in construction and implemented accordingly. In micro-finance and agricultural extension it recognized its lack of experience and partnered with institutions with proven track records in these fields. The KRBS Project was successful where it kept within the expertise of the implementing partner, but did not achieve its objectives when the tasks adopted by EMW fell outside their proficiency. Continuously broadening existing CoAgs leads partners outside their competencies and should be avoided. With the advances being made in broadening education opportunities and access being afforded to ethnic minorities under GVN Program 135 and the World Bank SEQAP (US$186 million, of which US$27.9 million came from the Vietnamese government and the rest financed by the WB), the findings of this evaluation lead the Team to believe that USAID’s resources would be better used outside the basic education sector. In the future, USAID and EMW should ensure that projects are designed and implemented with a comprehensive understanding of the problem they are seeking to address and a firm grasp of the local context. At the planning stage, a thorough needs assessment should always be conducted to inform the design of any development intervention. Following the needs assessment and the design of a project implementation proposal, comprehensive baseline measurements of all project indicators should be established to provide the foundation for both routine project monitoring and outcome analysis upon project completion. Performance monitoring plans (PMPs) should be required for all projects, particularly those with multiple follow-on proposals characterized by broadening scopes, objectives, and activities. KRBS Project End-Of-Project Evaluation 39 Despite the care taken to assure physical access to the KRBS, the absence of teacher training in educating those with special needs and the failure to recruit a single disabled student to date has meant that a major opportunity to make KRBS a center for excellence in educating the disabled has significantly fallen short. Broader objective of such projects should be given higher priority. Furthermore, USAID should encourage KTP and KRD officials to reach out to this population with the goal of fulfilling the original project mandate. Relying on school income generation to fund extra curricula activities is unrealistic in communes defined as ‘extremely difficult’ by GVN. Great caution should be adopted in forecasting the contribution that these activities will make to school budgets. In future projects that include income-generating activities, USAID should ensure proper documentation of receipts and work timesheets, and make sure that budget increases are properly documented and tied to specific activities and permissible expenditures. Appropriate accounting systems should be implemented and audited by EMW, which would allow for meaningful planning of income-generating activities. Routine staff turnover both within the USAID and implementing partner organizations leads to challenges with institutional memory. Key project documents should be properly filed and available, both in hard copy and digitally. Continued under-enrollment misses the opportunity for an additional 70 ethnic minority students to benefit from the superior facilities supplied at high cost. Through negotiation with GVN, USAID should encourage KTP and KRD officials to open the schools’ catchment area to include the other three communes in the district, in order to increase the enrollment rate and take advantage of the school’s full capacity. Training materials developed in the course of this project should be reviewed and institutionalized as part of the project’s greater goal to provide replicable programs for similar regions. Teachers should have specialized training in the skills of teaching children whose mother tongue is not Vietnamese. While it may be difficult to find certified teachers within ethnic minority populations, teachers’ aids and classroom assistants could provide a cultural bridge between ethnic minority students and Kinh teachers, who struggle to communicate and understand the social and cultural norms that govern learning behaviors. In the future, EMW and USAID should strive to ensure that DOET does not influence the selection of educational facilities for upgrading, such as the ECCEs, and should encourage targeting the neediest facilities and communities. Close cooperation with organizations that have long established connections with provincial authorities (such as the WU and DU) greatly increases the likelihood of positive outcomes from initiatives in Vietnam, in general, and the Central Highlands, in particular. Partnering for implementation with institutions that have strong and established connections with GVN authorities at all levels will help to ensure that projects meet their goals. 40 KRBS Project End-Of-Project Evaluation BIBLIOGRAPHY (n.d.). BOET, K. R. (2012, October 2). Canadian International Development Agency. (2012). Vietnam. Retrieved September 30, 2012, from Canadian International Development Agency: http://www.acdi-cida.gc.ca Dang, H.-A. (November 2009; revised January 2010). A Widening Poverty Gap for Ethnic Minorities. Washington DC: World Bank. Departments, K. T. (2012, October 1). DOFA, K. T. (2012, October 1). (K. E. Team, Interviewer) EMW. (2012, October 13). Program Development Director, East Meets West Foundation. EMW. (2012, September 29). Staff Discussions. Human Rights Watch. (n.d.). Montagnard Christians in Vietnam. Retrieved October 7, 2012, from Human Rights Watch: http://www.hrw.org Huyen, N. T. (2012, September 26). Director of Department of Ethnic Minority Education, Ministry of Education and Training. KRBS, M. (2012, October 2). Discussions. London, J. D. (2011). Education in Vietnam. Singapore: Institute of Southeast Asian Studies. Montagnard Human Rights Organization. (n.d.). Montagnard Human Rights Organization. Retrieved October 25, 2012, from http://www.mhro.org Oxfam Great Britain. (1998). Education for Poor Children: Research Findings from Lao Cai Province, Vietnam. Oxfam Great Britain, Vietnam Country Programme. Students, K. R. (2012, October 1-3). Teachers, K. R. (2012, October 1-3). Trading Economics. (n.d.). Vietnam Inflation Rate. Retrieved October 32, 2012, from Trading Ecomica: http://www.tradingeconomics.com/vietnam/inflation-cpi Turk, R. S. (2008). Explaining Ethnic Minority Poverty in Vietnam: a summary of recent trends and current challenges. Hanoi: World Bank. U.S. Department of State. (2012, January 5). Diplomacy in Action. Retrieved December 30, 2012, from http://www.state.gov: http://www.state.gov/outofdate/bgn/vietnam/195238.htm KRBS Project End-Of-Project Evaluation 41 U.S. Department of State. (2012, January 5). Vietnam 01/05/12. Retrieved December 30, 2012, from U.S. Department of State: http://www.state.gov/outofdate/bgn/vietnam/195238.htm USAID. (2012, September 25). Staff Discussions. Hanoi, Vietnam. VHLSS. (2006). Vietnam Household Living Standards Survey . Hanoi: Government of Vietnam. War Legacies Project. (n.d.). Health effects of Agent Orange/Dioxin. Retrieved October 22, 2012, from War Legacies Project: http://www.warlegacies.com/ World Bank. (2009). Ethnicity and Development in Vietnam. Washington: World Bank. World Bank. (n.d.). Vietnam Development Report 2012: An Overview. Retrieved November 3, 2012, from World Bank: http://web.worldbank.org 42 KRBS Project End-Of-Project Evaluation ANNEX A: KRBS PROJECT EVALUATION STATEMENT OF WORK STATEMENT OF WORK A. TASKS/QUESTIONS The Evaluation Team will be tasked with addressing the following overarching questions: Project Goals 1. To what extent has the project accomplished its goals? (See Section C.3 for goals of initial project and the extension.) 2. What have been the greatest successes as a result of the KRBS project? 3. What were the major challenges, how were they overcome, and what lessons have been learned? 4. Make recommendations for further areas of improvement in the Kon Tum educational system at the preschool and lower secondary levels and why. Sustainability 1. Evaluate the overall sustainability of the KRBS and the refurbished kindergartens. If the schools are not sustainable, provide recommendations to improve its sustainability. 2. Evaluate sustainability the other program components, teacher training, income generation, support for the early childhood centers, agriculture extension, and microcredit. B. TEAM COMPOSITION An evaluation team with a team leader will carry out the evaluation. Additionally, local support may be hired as necessary, by the implementer, for translation and logistics purposes. C. SCHEDULE & LOGISTICS The evaluation is estimated to take about 9-10 weeks, with the in-country portion of the evaluation starting on June 4, 2012 and finishing no later than June 29, 2012 (approximately four weeks). The Contractor’s in-country team must give USAID an initial briefing and a Team Planning Meeting (TPM), where both parties will exchange their ideas and expectations regarding the evaluation. At the briefing the Contractor’s team must describe in detail the proposed evaluation methodology and will provide a draft schedule of meetings and interviews (work plan). The final methodology and schedule with incorporated comments from USAID must be submitted for USAID approval within the first week of the evaluation team’s arrival in Hanoi. In line with Section H.5, “CONTRACTOR’S STAFF SUPPORT, AND ADMINISTRATIVE AND LOGISTICS ARRANGEMENTS,” the Contractor is responsible for making all logistical arrangements for the team. All expatriate team members will need a valid visa prior to entering Vietnam. That includes full responsibility for, but not limited to, transportation, computers, organizing meetings and interviews, and implementing the proposed evaluation methodology. However, USAID/Vietnam will assist, however, in arranging meetings with key stakeholders identified by USAID prior to the initiation of field work. KRBS Project End-Of-Project Evaluation 43 Additionally, USAID can also make recommendations on in-country lodging, if needed. USAID/Vietnam personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. Prior to the departure of the evaluation team on June 29, 2012, the Contractor’s team must debrief USAID on its findings, conclusions, and recommendations. A preliminary draft of the evaluation report must be presented at this time. USAID will provide oral comments at the debriefing. The evaluation team must provide USAID/Vietnam with a draft report by July 9, 2012 and USAID will provide comments by July 16, 2012. Once the Team Lead receives USAID comments s/he will have two weeks to finalize the report by July 30, 2012. D. METHODOLOGY & DATA SOURCES Since much of this evaluation will be conducted to evaluate the net impact and if the Kon Ray Boarding School project achieved its strategic and immediate objectives. The evaluation team should use various methods to assess the different aspects of the project to see all the objectives/components outlined RFTOP no. SOL-486-12-019 6 above was successfully achieved. Though the implementers have full leeway to design and use methods they feel are appropriate at different stages of the assessment, the evaluation must be participatory in its design and implementation. Literature/Documentation Review: The evaluator must review all major program background documents such as the: Annual Performance Reports, Annual Work Plans, 2009 Project Evaluation Report, Quarterly Reports, EMW Monitoring Plan, and other relevant USAID and EMW source data. Further, the following methods (not inclusive of all possible methods) for the assessment are suggested: 1. Checklists or Questionnaire: Checklists or questionnaires should be developed to ensure there is consistency in the collection of information. 2. FGDs: The evaluation should include FGDs with the target beneficiaries: People’s Committee of Kon Tum, WU, Department of Education and Training, ECCE Centers’ Teachers and Staff, KRBS Teachers and Staff, Students, Parent Organizations and Parents, University of Danang Professors (trainers), etc. 3. Individual Interviews using Checklist or Questionnaire: The evaluation should include an individual checklist or questionnaire to interview key individuals from the target beneficiaries: People’s Committee of Kon Tum, WU, DOET, ECCE Centers’ Teachers and Staff, KRBS Teachers and Staff, Students, Parent Organizations and Parents, and University of Danang Professors (trainers), etc. Data analysis: The evaluation should look at the activities and determine if they achieved desired results. E. DELIVERABLES Work Plan: The Contractor must prepare a detailed work plan for submission and review. This work plan will include the methodologies to be used in the evaluation, a schedule, and a detailed field work plan, including data collection instruments. The work plan must be submitted to the USAID/Vietnam General Development Office for approval no later than the third day of work, after arriving in Vietnam and before data collection begins. 44 KRBS Project End-Of-Project Evaluation Methodology Plan: The Contractor must prepare and discuss with USAID/Vietnam’s General Development Office a written methodology plan (the data collection and analysis process proposed by the evaluator) prior to implementation. Preparation, Presentation and Discussion of Preliminary Draft Report: Prior to departure from Vietnam, the Contractor must submit a preliminary draft of the report containing initial key findings and recommendations to the USAID/Vietnam General Development Office. The GDO will provide preliminary comments prior to final Mission debriefing to facilitate the preparation of the draft report. Draft Evaluation Report: The Contractor must submit a draft report of the findings and recommendations to the USAID/Vietnam General Development Office one week after the evaluation team’s departure from Vietnam. The written report should clearly describe findings, conclusions, and recommendations. USAID/Vietnam will provide comment on the draft report within one week of submission. Final Report: The Contractor must submit a final report that incorporates the team responses to Mission comments and suggestions no later than two weeks after USAID/Vietnam provides written comments on the team's draft evaluation report (see above). The format must include an executive summary, table of contents, acronym list, methodology, findings, and recommendations. The report must be submitted in English, electronically to USAID/Vietnam. The report must be limited to 30 pages, including an Executive Summary not exceeding 5 pages, and excluding Annexes. RFTOP no. SOL-486-12-019 7 F. SCHEDULE The Contractor must plan for the evaluation to begin o/a late May-early June and last approximately 9–10 weeks, of which four weeks will be in Vietnam. Additional time may be required for the Contractor to finalize the final report (if necessary). Below is an illustrative schedule: ACTIONS/DELIVERABLES– estimated LOEs DUE BY (o/a) Preparation& Review Literature – one week prior to arrival Methodology &Work Plans (Draft) – three days prior to arrival Begin in-country evaluation – Kick-off Meeting 06/04/2012 Team Planning Meeting (TPM) – 06/04/2012 Methodology/Work Plans (final) – two days 06/06/2012 Site/Visits/Data Collection – two weeks 06/22/2012 Initial Draft of Major Findings and Recommendations – 4.5 days 06/29/2012 Discussion of Preliminary Findings/Recommendations – .5 Day 06/29/2012 Depart Vietnam 06/29/2012 Draft of Evaluation Report – one week 07/09/2012 USAID Vietnam review of Draft Evaluation Report – one week 07/16/2012 Final Report – two weeks 07/30/2012 (End of Section C) KRBS Project End-Of-Project Evaluation 45 ANNEX B: KRBS PROJECT COOPERATIVE AGREEMENT MODIFICATIONS MOD # SIGNING DATE EMW PROJECT MODIFICATIONS PERIOD OF IMPLEMENTATION 3 9/30/2006 EMW agreement with USAID for ANTI-TIP activities is modified to incorporate a new, separate agreement between USAID and EMW using different funding, for the KRBS Project (this is the first application from EMW to USAID for the KRBS Project). Extend the completion date from September 11, 2008 to September 30, 2008 Increase both the total estimated amount and the total obligated amount by $512,906 from $200,000 to $712,906 Realign the budget to reflect these changes Expand the program description to include the additional activities for “Kon Ray Boarding School” 9/30/2006–9/30/2008 4 2/15/07 Revise the standard provisions to include “Homeland Security Presidential Directive-12 (HSPD-12) (September 2006). Unchanged 6 9/27/07 Expand the program description to include supplemental activities to the KRD Ethnic Minority Boarding School Increase the total estimated amount of the Agreement by $229,897, from “$712,906” to “$942,803” to cover the cost of the supplemental activities Increase the obligated amount to $942,803 to fully fund the Agreement Unchanged 8 7/31/08 Revise the Agreement completion date from July 31, 2008 to July 31, 2012 Revise the total estimated amount of the Agreement from $1,184,666 to $2,512,127 (includes EMW cost share) Revise the total obligated amount of the Agreement from $1,184,666 to 1,412,171 ($227,505) Incorporate Recipient’s proposal entitled “Ethnic Minority Boarding School in Kon Tum, Vietnam” as part of Attachment 2 Program Description of Agreement 7/31/2012 9 9/30/09 Increase the total estimated amount of the agreement by $241,863 Unchanged 10 6/3/2010 Provide incremental funding of $630,000 from $1,412,171 to $2,042,171 Revise the fiscal data Change financial reporting requirement Update ‘special provisions’ Unchanged 11 9/23/2010 Provide incremental funding of $360,000, changing the total obligated amount from $2,042,171 to $2,402,171 Unchanged 12 12/22/2011 Provide incremental funding of $109,956 changing the total obligated amount from $2,042,171 to $2,512,127 Unchanged 46 KRBS Project End-Of-Project Evaluation KRBS Project End-Of-Project Evaluation 47 ANNEX C: EVALUATION LIMITATIONS AND CHANGE OF EVALUATION SCOPE Social Impact Evaluation of the Kon Ray Boarding School-Project CoAg between USAID/Vietnam and the East Meets West Foundation 10.8.12 This document summarizes the changes that have been made to the original evaluation data collection plan and schedule, which have effectively altered the evaluation methodology and reduced the scope of the evaluation. The purpose of this document is to clearly outline these changes and the resulting limitations, in order to confirm with USAID the evaluation questions the team is able to effectively address. As recognized in contract No. AID-RAN-1-00-09-00019, signed on July, 17th 2012 between USAID/RDM Asia and SI, two weeks were allocated for the Evaluation Team to conduct site visits and data collection activities in KTP. Subsequent to contract signing, USAID submitted a request to the Kon Tum Provincial Authorities for the Evaluation Team to conduct site visits and data collection activities. On September 18th, SI was informed that the Kon Tum Provincial Authorities denied USAID’s request for 2 weeks of data collection, at which time SI was informed that the Evaluation Team was only approved to conduct site visits and carry out data collection activities for 6 days. The Evaluation Team arrived in Vietnam on Monday, September 23rd. On Tuesday 24th the Team was assured by USAID that verbal consent had been given for 6 day’s access but was then told on Thursday, September 27th that the Kon Tum Provincial Authorities had, without explanation, further reduced the time approved for data collection to three days. The Provincial Authorities, DOFA also controlled who the Evaluation Team was allowed to meet and which sites it was allowed to visit. At a meeting with EMW at its Danang office on Friday, September 28th, the SI Evaluation Team learned that during their 3 days in KTP, they would be accompanied by both EMW’s independently-commissioned Evaluation Team of 4 individuals and the EMW Project Manager and Construction/Engineer Specialist. The summation of these changes drastically altered the evaluation methodology and therefore, the team’s ability to answer each of the evaluation questions detailed below. 1. Boarding School Project Sustainability: a. Financial – the ability of the school administration to be financially self-sufficient through internal revenue generation and government contributions. Ability to plan and develop a multi-year budget for all revenue expenses and track and report on revenue and costs. b. Management – the ability of the school administrators to make correct decisions about strategic planning, curriculum, training, personnel, procurement, and maintenance. c. Technical – does the school have the proper tools, personnel, motivation and training, to conduct maintenance, repair the systems (water, electrical, wastewater), maintain the grounds and in general keep the facility up to a high standard? d. Environmental – does the school limit, prevent, and repair damage to the local environment with improved plantings, retaining mechanisms to prevent erosion, 48 KRBS Project End-Of-Project Evaluation ensure the purity of local water supply, appropriate waste disposal and maintenance of septic systems. 2. Sustainability of other program components including: a. Teacher training b. Income generation of the school c. Support for early childhood centers d. Agriculture extension e. Microcredit loans 3. Has phase one met stated goals and objectives including: a. Improve educational outcomes for disadvantaged ethnic minority students in KRD. b. Develop valuable skills in students to enable them to become productive members of society c. Improve access to education for disabled students in KRD d. Create opportunities for local families to improve their family incomes, develop new businesses and improve their farming techniques. 4. Has phase two (cost extension) met stated goals and objectives including: a. Ensure long-term sustainability of the school b. Refurbish 25 kindergartens in KTP c. Improve the capacity of teachers to work with disadvantaged, disabled and ethnic minority students d. Develop a replicable model to be used in areas with similar conditions in the Central Highlands e. Improve family incomes in the project area: According to SI’s proposal, and under the above-referenced contract, the Team’s original plan for data collection in Kon Tum included initial and separate meetings with both the Provincial and District Authorities, including the DOET, The People’s Committee, the Department of Education Promotion, and the WU. The major focus of data collection in the Province was designed around visits to several districts within the province where EMW upgraded kindergarten facilities, provided teacher training to teachers, worked with the WU to support Early Childhood Education Centers, and provided assistance to families through agricultural extension training and micro-credit loans. In addition to site visits, the Team planned to facilitate group discussions with program beneficiaries and stakeholders in these districts, as well as in-depth interviews with selected individuals from each group, such as the principals, teachers, and budget officers from the kindergartens. The team also planned to visit the three communes that support the KRBS to speak with parents of both past and current students, and to develop a sense of the context in which the ethnic minority students come from. While at the KRBS, the Team planned to facilitate several FGDs with both current and past students, as well as teachers, and staff, and to hold in-depth interviews with selected individuals from each of these groups, such as the school accountant, to discuss and review budgeting, and the principal, to discuss and review student testing and academic achievement. The Team also planned to have adequate time to examine all of the school facilities, including the vocational training laboratories and the income-generating projects from the pigs, chickens, and school garden. This evaluation plan and the supporting methodology were designed to facilitate the triangulation of data, with sufficient time to field-test data collection tools and to follow-up with individuals and groups in the event that additional information was needed. The design was also based on the understanding that the team would have control over sampling and selection of informants. During the three days the Evaluation Team spent in Kon Tum, from Oct. 1-3, the Team was unable to follow and implement the majority of the evaluation plan and methodology described above. The Team was restricted to large, group meetings at the Provincial level, and was unable to visit any sites KRBS Project End-Of-Project Evaluation 49 outside of Kon Tum Provincial center except for the KRBS. An overview of the Team’s activities is provided below. Monday, Oct. 1 – Meeting with Kon Tum Provincial Authorities and visit to 2 kindergartens Meeting from 8:00 a.m. to 11:00 a.m. with the following individuals in one room (approximately 20 people): - 4 members of SI Evaluation Team - 1 USAID Representative - 3 individuals on the EMW independent Evaluation Team - 2 individuals from EMW headquarters - 2 reps from Ministry of Foreign Affairs - 2 representatives from the Provincial WU - 1 rep from MOET - 1 rep from the EPA - 2 reps from The People’s Committee - 1 rep from Department of Planning and Investment (DPI) Site visits to two kindergartens in Kon Tum town center in the afternoon (not selected by SI Evaluation Team) with the following individuals participating in the visit and group meetings: - 4 members of SI Evaluation Team - 1 USAID Representative - 3 individuals on the EMW independent Evaluation Team - 2 individuals from EMW headquarters - 2 reps from Ministry of Foreign Affairs - 1 rep from MOET - 1 Kindergarten Principle - 2 kindergarten teachers Tuesday, Oct. 2nd - visit to KRBS, KRD Meeting with KRBS Management Team including the following individuals: - KRBS principal and 2 vice-principals - 4 members of SI Evaluation Team - 1 USAID Representative - 4 individuals on the EMW independent Evaluation Team - 2 individuals from EMW headquarters - 1 rep from Ministry of Foreign Affairs - 1 rep from MOET Meeting with KRD People’s Council and Bureau of Education and Training (BOET) with the following individuals: - 4 members of SI Evaluation Team - Vice chair of the People’s Council - Head of the District BOET - Head of the Provincial DOET Wednesday, Oct. 3rd - Meetings and Visit to KRBS 8 FGDs with current students, 3 boys and 3 girls from each grade (selected by SI) Meeting with 4 teachers (selected by the school) Meeting with 9 parents, 8 male and 1 female, some with children currently enrolled, others with children who already graduated (selected by the school) Meeting with 4 past students (selected by the school) 50 KRBS Project End-Of-Project Evaluation The collective, multi-stakeholder design of many of the above-documented meetings and discussions significantly hindered the Evaluation Team from gathering reliable data. Nonetheless, based on these site visits, meetings, interviews, and FGDs, and in conjunction with the discussions held with EMW and the University of Danang, the SI Evaluation Team is able to effectively address the following evaluation questions: 1. Boarding School Project Sustainability (a) Financial (b) Technical (c) Management 2. Sustainability of other components (a) Teacher Training (b) Income Generation for the school 3. Phase I (a) Access to education for ethnic minority students (b) Access to education for disabled students 4. Phase II (a) Sustainability of the Kon Ray School (b) The capacity of teachers to work with disadvantaged, disabled and ethnic minority students KRBS Project End-Of-Project Evaluation 51 ANNEX D: VIETNAM KRBS PROJECT EVALUATION PLAN 9.25.12 Four Overarching Evaluation Questions: 1. Boarding School Project Sustainability: a. Financial – the ability of the school administration to be financially self-sufficient through internal revenue generation and government contributions. Ability to plan and develop a multi-year budget for all revenue expenses and track and report on revenue and costs. i. Key Informants include: 1. EMW management 2. Provincial Government MOET 3. KRBS Management b. Management – the ability of the school administrators to make correct decisions about strategic planning, curriculum, training, personnel, procurement, and maintenance. i. Key informants include 1. EMW management 2. KRBS Management 3. KRBS Teachers c. Technical – does the school have the proper tools, personnel, motivation and training, to conduct maintenance, repair the systems (water, electrical, wastewater), maintain the grounds and in general keep the facility up to a high standard? i. Key informants include: 1. EMW Management 2. KRBS Management 3. KRBS Teachers 4. KRBS Staff (technicians, grounds staff, etc.) d. Environmental – does the school limit, prevent, and repair damage to the local environment with improved plantings, retaining mechanisms to prevent erosion, ensure the purity of local water supply, appropriate waste disposal and maintenance of septic systems. i. Key informants include: 1. EMW Management 2. KRBS Management 3. KRBS Teachers 4. KRBS Staff 2. Sustainability of other program components including: a. Teacher training i. Key informants include: 1. EMW Management 2. KRBS Management 3. University of Danang teachers and trainers 4. Academic and vocational teachers trained in pedagogical techniques (5-day course) 5. Primary school teachers in Kon Ray and surrounding districts (3-day course) 6. Teachers from ECCE Centers upgraded under cost extension (4-day course) 52 KRBS Project End-Of-Project Evaluation b. Income generation of the school i. Key informants include: 1. EMW Management 2. KRBS Management 3. KRBS Teachers c. Support for early childhood centers i. Key informants include: 1. Teachers at ECCE 2. EMW Management 3. KRBS Management 4. WU 5. University of Danang d. Agriculture extension i. Key informants include: 1. EMW Management 2. Recipients of Ag. extension training 3. KRBS Management 4. Provincial Department of Agriculture e. Microcredit loans i. Key informants include: 1. EMW Management 2. KRBS Management 3. WU 4. 300 Recipient Families 5. Central Region Livelihood Improvement Project (CRLIP) 3. Has phase one met stated goals and objectives including: a. Improve educational outcomes for disadvantaged ethnic minority students in KRD. i. Increased daily attendance rates ii. Lower drop-out rate iii. Improve the promotion rate from lower secondary (grades 6-9) to upper secondary school (grades 10-12) 1. Key Informants include: a. EMW Management b. KRBS Management c. Parents of students d. Students i. Key documents: 1. School records over life of program e. KRBS Teachers b. Develop valuable skills in students to enable them to become productive members of society: i. Key informants include: 1. Past students 2. Current students c. Improve access to education for disabled students in KRD i. Key informants include: 1. PDOET 2. EMW Management 3. KRBS Management 4. KRBS Teachers 5. Parents of students KRBS Project End-Of-Project Evaluation 53 6. Students d. Create opportunities for local families to improve their family incomes, develop new businesses and improve their farming techniques. i. Key informants include: 1. WU 2. EMW Management 3. Parents of students 4. Students 5. Central Region Livelihood Improvement Project (CRLIP) 4. Has phase two (cost extension) met stated goals and objectives including: a. Ensure long-term sustainability of the school i. Key Informants include: 1. EMW Management 2. PDOET b. Refurbish 25 kindergartens in KTP i. Key informants include: 1. EMW Management 2. PDOET 3. KRBS Management c. Improve the capacity of teachers to work with disadvantaged, disabled and ethnic minority students i. Key informants include: 1. KRBS teachers 2. University of Danang 3. EMW Management d. Develop a replicable model to be used in areas with similar conditions in the Central Highlands i. Key Informants include: 1. EMW Management 2. PDOET 3. KRBS Management e. Improve family incomes in the project area : i. Key informants include: 1. Parents 2. Statistics from Provincial Government (Finance Ministry) 54 KRBS Project End-Of-Project Evaluation KEY INFORMANT SI QUESTIONS TO INFORMANT ? # Provincial DOET Does the DOET see Kon Ray as a private or public school to understand who the financial auditor of the school is and to whom they send their audit? Does Kon Ray receive government contribution/support? Do students have to pay a fee to attend Kon Ray? Are you aware of any external funding sources that Kon Ray School receives? Are you aware of any internal funding that they school is generating on their own. What does the Department think/what is your attitude to Kon Ray raising their own internal funding to support the school? Do you think in the future that Kon Ray will be financially self-sufficient? Would you like them to be? Does DOET have any involvement in either constructing or reviewing the Kon Ray School budget? Does DOET have any thoughts about the effect of the scale of Kon Ray on their need for financial support? Is the school to big and complex in infrastructure? Do heads of public schools have a discretionary budget? Is the structure of school budgets the same for each school throughout the country in terms of allocation to each line item? Is the allocation of money to schools purely based on enrollment? Is the government capable of supporting the financial needs of Kon Ray? 1.a What is your policy on access to education for disabled students? Do you ensure that all schools in all of the districts are designed to support equal access for students with disabilities? How many schools in Kon Ray can provide access for disabled students? What are the criteria for schools to be qualified as accessible to disabled students How many teachers in the province have attended the training for how to teach disabled students – how many are qualified What constitutes qualification for teachers to instruct students with disabilities Who provides teacher training for working with disabled students? How often is it provided? Does Danang provide teacher training? 3.c What is the provincial policy for government contribution to boarding school? Does Kon Ray School qualify as a boarding school with regard to receiving financial support from the government? Are you involved in the EMW planning for the Kon Ray School to become financially independent from EMW and USAID? If the Kon Ray school cannot financially support itself with government contributions, what are your suggestions for how the school can become financially self-sufficient? What is the government policy for teacher student ratio? If a school generates income, is it allowed to keep it all? Is the school allowed to use the self￾generated income as it wishes? 4.a Tell us what you know about EMW refurbishment of the kindergartens in Kon Tum? What did the refurbishment involve? Tell us about what the government has done to refurbish kindergartens? 4.b How do you see the KRBS model? Would you use this model again in other areas of the province? Why or why not? Would you be interested in receiving additional donor funding to replicate the Kon Ray model in other areas of the province? 4.d KRBS Project End-Of-Project Evaluation 55 KEY INFORMANT SI QUESTIONS TO INFORMANT ? # EMW Management Tell us what measures are already in place to build financial sustainability for the Kon Ray School? What additional income generating measures are you considering to develop? Please provide us with some documentation on the schools’ income generation and expenditures. Please tell us about the financial support you receive from the government. Where do you see gaps in the school’s ability to be financially sustainable? Are you allowed to keep all of the income that you generate? Can you use internally-generated income as you wish? Does EMW manage the school’s budget or does the school itself manage its own budget? Does EMW see the school as a long-term commitment even if USAID funding does not continue? What is the future of EMW involvement in the school from a financial perspective? What portion of internally-generated funds is managed by the school itself versus EMW? What kind of capacity building are you providing for the Boarding school in terms of financial management? Does EMW have plans to continue funding the Kon Ray School? 1.a Please tell us about all of the positions you have at the Kon Ray School including management, teachers, and staff. How many of the people working at Kon Ray come from ethnic minorities? How many of the people working at Kon ray are disabled? What standards does the school follow for school management? Does EMW have its own standards and processes that Kon ray must adhere to for school management? Is the Kon Ray management divided between school management and project management? Please tell us about the management and their roles and responsibilities. Has EMW provided Kon Ray school management with training in strategic planning, curriculum development, personnel and procurement, and maintenance? What is the level of autonomy in management decisions between EMW and the school management board? How are disagreements between the school and EMW resolved? Please give us an example of this. What measures are you taking to build the management capacity of the school and how are you starting to remove direct management control over the school? 1.b Please tell us what measures are in place for the school to be maintained properly regarding infrastructure (water, sewage, electricity, grounds, machines, etc.) Are any of the maintenance jobs done by parental contribution? Does the school budget have a line item for maintenance? What are the maintenance requirements of school? 1.c How does the school maintain the environment? How does the school take care of waste? What are the schools’ environmental standards to maintain a healthy environment for the students? 1.d Is Danang University the only entity in charge of providing teacher training? Is this training sustainable? If so how, if not, why not? Do teachers/participants need to pay for training? Will they be required to pay in the future? Can you please tell us about the various trainings that UD provides for teachers in Kon Tum? We would like to see the training plans please. Do you have an agreement or contract with UD to provide the teacher training? What are the plans to integrate the teacher training component into a sustainable model supported by the government? 2.a This is covered by 1.a 2.b Please tell us about the ECCE component of the program. Do you work with any partners on this component of the program? Are kindergartens the same thing as ECCE? Are these ECCE paid for by parents? What plans are in place to make sure this program component is sustainable? 2.c 56 KRBS Project End-Of-Project Evaluation KEY INFORMANT SI QUESTIONS TO INFORMANT ? # EMW Management What did the ag extension component entail? When was it implemented? Have you seen any direct contributions from the ag? Extension to increased incomes for the families? Who was responsible for providing ag extension training? Who received training? Was the objective of the ag training to raise income for the school or for the families? Please tell us about the challenges and successes of this program. 2.d Please tell us about the microcredit program. Please describe your relationship with the WU to on this project component. How were there 300 families selected for loans? Were they tracked? How many of them repaid the loans? Did the loans help to improve their economic standard? If so how? Did the program experience any challenges with this component? 2.e Please explain to us the methods you use to track education outcomes of students. - Attendance rates - Dropout rates - Enrollment rates - Promotion rates from lower to upper secondary How often is the school reporting to you? What changes have you seen in educational outcomes of students over years? Please tell us about why the school has not reached its 250 capacity enrollment rate. How did you choose 250 as the enrollment rate of the school? Do you think the enrollment rate is a signal of weakness in the program? 3.a What are your criteria for determining access to education for disabled students? What is your definition of disabled student? Why did you choose to focus the program on accessibility for disabled students? Do you have demonstrated need for schools that are accessible by disabled students? What was the expectation at the project design phase, for the number of children with disabilities who would attend the school? Has your expectation been met? 3.c Tell us about how the program has contributed to increased incomes for families. 3.d This question was already answered 4.a Please tell us about the program component to refurbish the 25 kindergartens. Who was involved in this program component – did you work with any partners. How did you select the 25 schools – who chose them, based on what criteria? Have you monitored the refurbishments? 4.b How has EMW trained teachers to provide for students with disabilities? Tell us about how teachers’ capacity has been built to work with students with disabilities. 4.c Does EMW think that Kon Ray is a replicable model to use in other areas of the country? If so, why, if not, why not? Does EMW have any plans to replicate the model? Is there anything you would change in the model before replicating it? What have been the biggest challenges/flaws with this model? What are the best design elements of the Kon Ray model? How did you decide to develop the school in this way? On what criteria did you base this school model? Please tell us how you worked with the government to develop this school model. 4.d KRBS Project End-Of-Project Evaluation 57 KEY INFORMANT SI QUESTIONS TO INFORMANT ? # KRBS Management Tell us what measures are already in place to build financial sustainability for the Kon Ray School? What additional income generating measures are you considering to develop? Please provide us with some documentation on the schools’ income generation and expenditures. Please tell us how much the school generates on its own and how much the school receives from the government. Please tell us about the financial support you receive from the government. Where do you see gaps in the school’s ability to be financially sustainable? Are you allowed to keep all of the income that you generate? Can you use internally-generated income as you wish? Who manages the school’s budget? Do you think that EMW is committed to funding/supporting the school as a long-term commitment even if USAID funding does not continue? Are you and your staff/managers committed to working at the boarding school even if EMW and USAID cannot continue their support? How many people are involved in the financial management of the school? What are their roles and responsibilities? What are your plans for financial sustainability in the past? What has been successful and where do you see gaps? Do you have any stakeholders (private companies, NGOs, etc.) who are providing financial support to the school? Do you know what the future of EMW involvement in the school from a financial perspective? What portions of internally-generated funds are managed by the school itself versus EMW? What motivated each of the management members to work for the Kon Ray School? What motivates the management members to continue working at Kon Ray? 1.a What are the roles and responsibilities of the people involved in your management board? How many of the people working at Kon Ray come from ethnic minorities? How many of the people working at Kon ray are disabled? Does your management board have different staff in charge of project management and school management? What are your management standards? Are they the same as the governments? Does EMW have its own standards for project management and for school management? Has Kon Ray school management received training in strategic planning, curriculum development, personnel and procurement, and maintenance? What is the level of autonomy in management decisions between the school management board and EMW? What is the relationship between How are disagreements between the school and EMW resolved? Please give us an example of this. What measures are you taking to build the management capacity of the school and how are you starting to remove direct management control over the school? 1.b Please tell us what measures are in place for the school to be maintained properly regarding infrastructure (water, sewage, electricity, grounds, machines, etc.) How many people do you have on staff to conduct maintenance? What are the roles, responsibilities, and skills of your maintenance staff? Are there staff positions you would like to fill but cannot? What are your maintenance standards? How do you ensure that proper school maintenance is carried out? Are any of the maintenance jobs done by parental contribution? Does the school budget have a line item for maintenance? What are the maintenance requirements of school? Are there any problems with the infrastructure and maintenance of the school? What are highest priority maintenance and infrastructure needs for the school? Have you had any problems in the past and if so, how were they addressed? 1.c 58 KRBS Project End-Of-Project Evaluation KEY INFORMANT SI QUESTIONS TO INFORMANT ? # KRBS Management How does the school maintain the environment? Does the school have a specific program to maintain the environment? How does the school take care of waste and waste water? What are the schools’ environmental standards to maintain a healthy environment for the students? Does the school have an environmental code of conduct? How does the school ensure the drinking water is safe for the children? Has the school experienced any complaints from the local community about how the school environment is maintained? Are there any local representatives from the community involved in the school’s environmental management? Have any of the school’s environmental management practices also benefited the local community. 1.d Which of your teachers have received training and from whom? What training did they receive and when? Has the training been affective? Please explain. Are there any gaps or needs your teachers have for additional training? If so, what are they? Is this training sustainable? If so how, if not, why not? Does the school contribute funds to support the training? Do teachers receive per diem to attend training? If so, who pays this per diem? Is there enough training provided to your teachers to meet their demands? How do you generate demand from teachers for training? Will they be required to pay in the future? Do you have an agreement or contract with UD or any other entities to provide the teacher training in the future? What plans have you made to ensure that teachers continue to receive training once EMW and USAID funding is discontinued? What your plans to integrate the teacher training component into a sustainable model supported by the government. 2.a This question is covered by 2.a 2.b Is the school involved in the ECCE component of the EMW Program? 2.c Has the school been involved in the agriculture extension and microcredit program that EMW carried out? 2.d Has the school been involved in the agriculture extension and microcredit program that EMW carried out? 2.e Please explain to us the methods you use to track/monitor education outcomes of students. - Attendance rates - Dropout rates - Enrollment rates - Promotion rates from lower to upper secondary How often do you report on these outcomes to EMW, District, Province? What changes have you seen in educational outcomes of students over years? Please tell us about why the school has not reached its 250 capacity enrollment rate. How did you choose 250 as the enrollment size of the school? Do you think the enrollment rate is a signal of weakness in the program? What are your criteria for selecting students? What are you doing with the facilities that aren’t being used? Could this space be used in a different way? For example, for primary school students? How did you develop these criteria? How do you treat applications from disabled students? Have you ever received applicants? What do you do to mobilize students to come to the school? How do you see your schools’ enrollment rate compared with other schools in the district? 3.a KRBS Project End-Of-Project Evaluation 59 KEY INFORMANT SI QUESTIONS TO INFORMANT ? # KRBS Management What are your criteria for determining access to education for disabled students? What is your definition of disabled student? Why did you choose to focus the program on accessibility for disabled students? Do you have demonstrated need for schools that are accessible by disabled students? What was the expectation at the project design phase, for the number of children with disabilities who would attend the school? Has your expectation been met? How many disabled students do you currently have at the school? If you do not have any disabled students currently enrolled in the school, do you think the training that teachers received on how to work with these students was useful or not? Does the school have any initiatives different from the government to encourage disabled students to attend the school? What is the role of EMW in raising and promoting that kind of initiative? 3.c Do you know anything about EMW initiative to refurbish the 25 kindergartens in Kon Tum? 4.b How do you see the KRBS model? Would you use this model again in other areas of the province? Why or why not? Would you be interested in receiving additional donor funding to replicate the Kon Ray model in other areas of the province? 4.d KRBS Teachers Did you develop your own curriculum or do you follow a government-led curriculum? Do you have special guidelines for teachers about how to work with disabled students? What is the relationship between the teachers and the school management? Are teachers involved in any of the school management decisions? How often do you hold planning sessions How often do you have meetings and report with the District Education Officials? If you want to introduce something new into your teaching, do you have to obtain approval from the DOET first? How long have each of the teachers been teaching at Kon Ray? How many years of experience do the teachers have? Do the teachers intend to continue teaching at Kon Ray into the future even USAID funding is discontinued? What training have you received? What skills did you develop from the training? How have you applied these skills? What was the focus of the training? What are your needs in terms of support from the school management and EMW? Do you have any training needs? 1.b Please tell us what measures are in place for the school to be maintained properly regarding infrastructure (water, sewage, electricity, grounds, machines, etc.) Do you see a connection between the school environment and the student performance? Do any school maintenance issues get in the way of your teaching? Please tell us about the general motivation of the teachers and students. Do the school facilities meet your needs as teachers? Do they meet the needs of the students? Are you aware of the schools’ maintenance standards? Do teachers have a role in ensuring that school maintenance is carried out? Are any of the maintenance jobs done by parental contribution? Are there any problems with the infrastructure and maintenance of the school? What are highest priority maintenance and infrastructure needs for the school? Have you had any problems with the school infrastructure and facilities in the past and if so, how were they addressed? 1.c 60 KRBS Project End-Of-Project Evaluation KEY INFORMANT SI QUESTIONS TO INFORMANT ? # KRBS Teachers How does the school maintain the environment? Do the teachers have a role in environmental maintenance? Does the school have a specific program to maintain the environment? How does the school take care of waste and waste water? What are the schools’ environmental standards to maintain a healthy environment for the students? Does the school have an environmental code of conduct? How does the school ensure the drinking water is safe for the children? Has the school experienced any complaints from the local community about how the school environment is maintained? Are there any local representatives from the community involved in the school’s environmental management? Have any of the school’s environmental management practices also benefited the local community. Are the teachers satisfied with the way the school environment is maintained? Have the teachers experienced any problems with the environmental maintenance in the past, and if so, how were they addressed? 1.d Do the teachers receive a supplementary income to what the government provides them? If so, where does it come from? Are the teachers able to meet their financial needs? What benefits do the teachers receive? For example, free accommodation, free food What has motivated each of the teachers to come and work in the school? Are the teachers paid extra for supervising non-education/curriculum related activities. What internal income-generating activities does the school have? How sustainable are these activities? Are you receiving funding to attend trainings? What is the teacher involvement in the schools’ income-generating activities? Do teachers provide extra assistance to students outside of the classroom hours? Do you receive payment for extra assistance? 2.b Please explain to us the methods you use to track/monitor education outcomes of students. - Attendance rates - Dropout rates - Enrollment rates - Promotion rates from lower to upper secondary How often do you report on these outcomes to EMW, District, and Province? What changes have you seen in educational outcomes of students over years? Please tell us about why the school has not reached its 250 capacity enrollment rate. How did you choose 250 as the enrollment size of the school? Do you think the enrollment rate is a signal of weakness in the program? What are your criteria for selecting students? What are you doing with the facilities that aren’t being used? Could this space be used in a different way? For example, for primary school students? How did you develop these criteria? How do you treat applications from disabled students? Have you ever received applicants? What do you do to mobilize students to come to the school? How do you see your schools’ enrollment rate compared with other schools in the district? 3.a KRBS Project End-Of-Project Evaluation 61 KEY INFORMANT SI QUESTIONS TO INFORMANT ? # KRBS Teachers What are your criteria for determining access to education for disabled students? What is your definition of disabled student? Why did you choose to focus the program on accessibility for disabled students? Do you have demonstrated need for schools that are accessible by disabled students? What was the expectation at the project design phase, for the number of children with disabilities who would attend the school? Has your expectation been met? How many disabled students do you currently have at the school? If you don’t currently have any disabled students enrolled in the school, do you think the training that teachers received on how to work with these students was useful or not? Does the school have any initiatives different from the government to encourage disabled students to attend the school? What is the role of EMW in raising and promoting that kind of initiative? 3.c How have you been trained to provide for students with disabilities? What skills have you developed as a result of the training? How have you applied the skills you have learned? Tell us about how your capacity has been built to work with students with disabilities. Do you have any training needs that have not been met? Has it been useful for you to receive intensive training about working with students with disabilities even though you don’t work with disabled students? What do you do to maintain the training you received? 4.c 62 KRBS Project End-Of-Project Evaluation ANNEX E: INTERVIEW AND FOCUS GROUP DISCUSSION GUIDES DEPARTMENT OF PLANNING AND INVESTMENT QUESTIONS RELEVANCE Describe briefly the plans and achievements in poverty reduction of Kon Tum for the period of 2006 to now? Overall context of anti-poverty in Kon Tum Were there any changes of state budget investing in a commune if a commune is removed from the Program 135 list? Communes are phased out of Program 135 Are you familiar with EMW and its activities in promoting education in the province? What is the role of DPI in EMW’s projects? Role of DPI in EMW’s project DPI has a very important role in budgeting for KTP. Each year, Kon Ray School needs around 1.1billionn VND for operation for about 160 students? How much does it cost, per student, to provide financial support to the KRBS? How much do other boarding schools in the area receive and how is this estimated? Do you think the current state budget meets the needs of all schools? Do you think the government spends more per child on Kon Ray than other schools in the province? Do you know of any activities the school has to generate funding on its own? Do you think the school can develop these measures to obtain more money? Boarding School Project Sustainability: Financial – the ability of the school administration to be financially self-sufficient through internal revenue generation and government contributions. Ability to plan and develop a multi-year budget for all revenue expenses and track and report on revenue and costs. Please tell us what you know about the infrastructure of Kon Ray School? In your opinion, does the KRBS give value for money for the government, or is it too costly to maintain in the future? Do you think Kon Ray could be a model for building other schools in the province? KRBS Project End-Of-Project Evaluation 63 EDUCATION PROMOTION ASSOCIATION QUESTIONS RELEVANCE Please tell us about EMW activities to promote education in the province? What is the role of WU in EMW’s projects? Role of Women Union in EMW’s project Please tell us about EMW activities to support ECCE centers? Now that EMW program is finished, how will the ECCE centers continue these activities? Will these activities need support in the future or they can manage themselves in terms of human and financial resources? Sustainability of other program components including: Support for early childhood centers Please tell us about EMW activities to support agriculture extensions? Who were the beneficiaries and how were they selected? Do you know whether these activities produced any benefits? Agriculture extension Please tell us about EMW activities to provide poor families with microcredit loans? Who were the beneficiaries and how were they selected? Do you know whether these activities produced any benefits? Will these loan programs continue in the future? Microcredit loans Please tell us about EMW activities to support local families to improve their incomes. Who were the beneficiaries of these activities and how were they selected? Will these activities continue in the future? Create opportunities for local families to improve their family incomes, develop new businesses and improve their farming techniques. 64 KRBS Project End-Of-Project Evaluation PROVINCIAL WOMEN’S UNION QUESTIONS RELEVANCE Please tell us about the work you have done with EMW. What role does the WU have in EMW’s projects? Role of Women Union in EMW’s project Please tell us about the EMW activities to support the ECCE centers? What role does the WU have with this activity? Now that EMW is finished with this program, what kind of support is needed to ensure that ECCE centers will continue in the future? For example, human and financial resources. Sustainability of other program components including: Support for early childhood centers Please tell us about EMW activities in support of agriculture extension training for poor families. What was the role of WU in these activities? Agriculture extension Please tell us about EMW activities to provide poor families with microcredit loans. What was the role of WU in these activities? How were the beneficiaries selected? How were they monitored? Did they repay their loans? Were there any benefits the families received from the loans? Will the WU continue to provide poor families with loans? Microcredit loans Please tell us about EMW activities to support local families to improve family incomes. What is the role of WU in these activities? Will WU continue these activities without support from EMW? Create opportunities for local families to improve their family incomes, develop new businesses and improve their farming techniques. Please tell us about the WU program for training of trainers and the community volunteers who instruct ethnic minority women about early childhood education. Training of trainers and community volunteers KRBS Project End-Of-Project Evaluation 65 NATIONAL MINISTRY OF EDUCATION AND TRAINING 1. Please tell us about the government’s policy for general education. 2. Please tell us about the policies towards ethnic minority students and education. 3. We would like to know anything in particular about education in the highland area and Kon Tum in particular. 4. Please tell us about the government’s policy on boarding schools. 5. Please tell us about how boarding schools obtain government subsidies/funding. What are the criteria schools must meet to receive government funding? 6. Please tell us about the government’s policy on early childhood education and care 7. Please tell us how the government monitors school performance – both student performance and teacher performance. 8. Please tell us how the government ensures that there are enough skilled and qualified teachers in the remote communes and districts in rural provinces like Kon Tum. 9. Please tell us about the government policy for disabled children/students 10. How does the government ensure access to education for children with disabilities? 11. What is the government policy on teacher training? 12. Does the government instruct schools about what vocational skills should be taught/offered to students? 13. What is the government policy on teaching foreign languages in schools – which languages receive preference? 14. Please explain how salaries are figured for teachers across the country. 15. Do teachers receive an extra benefit for working in remote, poor areas of the country? 16. How does the government attract/recruit students to come to school? 17. How does the government attract/recruit ethnic minority students to come to school? 18. What is the overall state of education in Kon Tum? 19. How does the state of education in Kon Tum compare with other provinces throughout the country? 20. What is the government policy on teaching in ethnic languages? 21. What is the role for international donors in supporting education development in Vietnam? 22. How do you see international donors helping to improve the quality of education in Vietnam? 23. What is your experience working with international donors on education development programs? 24. Do you have any lessons to share with us, or any recommendations about what you would like to see in the future? 25. What would you like to know about the KRBS Program? 26. Is there anything from our evaluation that could help you with your work and planning? 66 KRBS Project End-Of-Project Evaluation CURRENT STUDENTS AT KON RAY BOARDING SCHOOL 1. How many hours does it take you to get home from school? 2. How did you learn about the Kon Ray School? 3. Does your family have to pay for or give contributions for you to attend Kon Ray? 4. How many brothers or sisters do you have? 5. Do you have any brothers or sisters with you at school? 6. Do you have brothers or sisters who are old enough to attend school at Kon Ray, but who do not attend school at Kon Ray? 7. What type of work do your parents do? 8. Do you have difficulties when you attend school to buy clothes? 9. Are you taking any of the vocational courses? If yes, which ones? 10. Are there vocational courses you would like to take but Kon Ray does not offer? 11. What are your plans for after you graduate from Kon Ray? a. Will you go to higher education b. Will you work with your parents c. Will you go into a vocational job 12. Do you like to study at Kon Ray? 13. What are your suggestions to improve Kon Ray? 14. What do you think about your teachers? a. What are things you like about your teachers? b. Would you like to change about your teachers? 15. What language do you speak at home? 16. Was it challenging for you to learn how to speak Vietnamese at Kon Ray? 17. Do you spend time working on the school garden or helping to maintain the environment? 18. Do you ever feel hungry when you’re at school? 19. How many times a day do you eat? 20. What kind of food do you eat? 21. Do you miss your family when you are at Kon Ray? 22. How often do you see your family when you are at Kon Ray? KRBS Project End-Of-Project Evaluation 67 KON TUM PROVINCIAL DEPARTMENT OF EDUCATION AND TRAINING 1. How did KTP choose to build the boarding school in Kon Ray? a. Did the Province first ask EMW for assistance or did EMW ask the Province if they could implement the project? Please describe the sequence of events. 2. How did the Province choose KRD for the boarding school? 3. Please tell us about what happened when the Kon Ray School opened and the 3 target communes were no longer classified as poor and therefore no longer eligible to receive government subsidy to attend the boarding school. a. Please explain to us how the government decided to provide subsidy to the students even though they were not classified as poor. 4. Please tell us about the current constraints that the education system of KTP is facing? 1.a Kon Ray School financial sustainability. 1. Is Kon Ray a public or private school? 2. What kind of financial support does Kon Ray currently receive from the government? 3. How much do students have to pay to attend Kon Ray? 4. Does Kon Ray receive funding from external sources? 5. Does Kon Ray generate any internal funding? a. If yes, what are the activities? b. How much of their current costs does this cover? 6. Does DOET have any involvement in planning or reviewing the Kon Ray School Budget? 7. Does the scale of Kon Ray present challenges for DOET to provide sufficient financial support? 8. How do you see the scale and infrastructure of the Kon Ray School? 9. What are the criteria for the amount of financial support the government provides to Kon Ray? a. For example, is it based on enrollment rate, number of teachers etc.? 10. Please explain DOET’s plans for providing financial support to the Kon Ray Boarding School now that EMW is finished with the program. 3.a Educational outcomes for disadvantaged, ethnic minority children 1. Please explain to us the situation for ethnic minority students accessing education in Kon Tum. a. Have you seen any change in this situation in the last 5 years? b. If yes, what are the changes? 2. Please tell us about the educational performance of students at the Kon Ray School. a. Have you seen any changes in student performance? b. If yes, what are the changes? 3.c Access to education for children with disabilities 1. What is your policy to provide access to education for children with disabilities? a. What are the criteria for schools to be classified as accessible to children with disabilities? b. What are the criteria to ensure that teachers are qualified to instruct children with disabilities? c. How many schools in Kon Tum are designed to accommodate children with disabilities? d. How do you ensure that children with disabilities can come to school? 2. Please explain to us the situation for disabled children accessing education in Kon Tum. a. Have you seen any change in this situation in the last 5 years? b. If yes, what are the changes? c. What is the population of disabled children in Kon Tum? d. Do you have a recent assessment we can review for our records? 3. In your opinion, has the EMW program improved access to education for disabled children? a. Please explain why or why not. 68 KRBS Project End-Of-Project Evaluation b. How many children with disabilities currently attend the Kon Ray School? 4. Regarding the “Strengthening and consolidation of the network to assist the rehabilitation of People with Disabilities for the period 2008-2012 and Orientation towards 2015”. Please share with us the progress to date in reaching the goals of this project. 4.b Refurbishment of kindergartens 1. Tell us what you know about EMW refurbishment of the kindergartens in Kon Tum? 2. What kind of activities did EMW do refurbish the kindergartens? 3. Was the refurbishment useful? a. Please explain 4. Does the government have plans to do is own refurbishment to schools in the Province? 5. In your opinion, has the EMW program improved access to education for ethnic minority children? a. Please explain why or why not. 6. In your opinion, is the Kon Ray Boarding School a good model for improving the education of ethnic minorities? a. Please explain why or why not. b. Is there anything you would change about the Kon Ray School? 7. Would you like to replicate the model in other districts in Kon Tum? a. If yes, where? 8. What suggestions do you have for USAID for future support to the province? 9. What suggestions do you have for EMW for future support to the province? KRBS Project End-Of-Project Evaluation 69 ANNEX F: EVALUATION KEY INFORMANTS AND STAKEHOLDERS EVALUATION KEY INFORMANTS ORGANIZATION NAME POSITION TITLE Department of Ethnic Minority Education Ministry of Education of Vietnam (MOET) Ms. Nguyen Thi Thu Huyen Director East Meets West Foundation (EMW) Ms. Minh Chau Nguyen Country Director East Meets West Foundation (EMW) Ms. Vo Thi Hien Deputy Director, Community Development East Meets West Foundation (EMW) Mr. Mark Conroy Senior Advisor (Former Director) East Meets West Foundation (EMW) Ms. Pham Thi Dieu Loan Project Assistant East Meets West Foundation (EMW) Mr. Nguyen Hiep Assistant Financial Controller East Meets West Foundation (EMW) Mr. Vo Le Phu An Construction Project Officer East Meets West Foundation (EMW) Ms. Nguyen Thi Minh Thu Program Development Director East Meets West Foundation (EMW) Ms. Ho Thi Y Nhi Sr. Deputy Director for Admin.& Finance Kon Tum DOFA Mr. Nguyen Thanh Duoc Chairman Kon Tum DOFA Ms. Y Ly Trang Vice Chairman Kon Tum DOFA Mr. Vien Official Kon Tum DOFA Ms. Lieu Official Kon Tum DOET Mr. Cam Official Kon Tum DPI Ms. Nhan Official Kon Tum WU Ms. Lien Vice Chairman Kon Tum Women’s Union Ms. Linh Director of Development and Foreign Cooperation Kon Tum EPA Mr. Thai Chairman Kon Ray People’s Council Mr. A Pa Pin Chairman Kon Ray People’s Council Mr. Nguyen Xuan Trung Chief Administrative Office Kon Ray BOET Mr. Nguyen Thiem Chairman University of Danang Prof. Le Minh Son Vice Head University of Danang Prof. Nguyen Tan Le Professor University of Danang Prof. Le Tu Tin Professor University of Danang Prof. Le Thi Tan Professor University of Danang Prof. Nguyen Thi Minh Tam Professor Danang Orthopedic & Rehabilitation Hospital Hoang Van Cuc Director Danang Orthopedic & Rehabilitation Hospital Minh Administrative Chief EMW Evaluation Team Ms. Lieu Evaluation Team Member USAID Mr. Randolph B. Flay Program Development Office Director 70 KRBS Project End-Of-Project Evaluation EVALUATION KEY INFORMANTS ORGANIZATION NAME POSITION TITLE USAID Ms. Kyung (KC) Choe General Development Office Director USAID Ms. Le Ha Van Development Assistance Specialist Nang Mai Kindergarten School, Le Loi Ward, Kon Tum City Ms. Mai Head of School Hoa Mai Kindergarten School, Doan Ket Commune, Kon Tum City Ms. Nga Head of School KRBS Mr. Doan Van Thoai Head of KRBS KRBS Ms. Hoa Thi Hong Vice Head of KRBS KRBS Mr. Vuong The Thanh Vice Head of KRBS EVALUATION STAKEHOLDERS STAKEHOLDER TOPIC INDIVIDUAL OR GROUP MEETING CECI program ILMC Improved Livelihoods for Mountainous Communities Agricultural Individual CRLIP (Central Region Livelihood Improvement Project). Micro-credit Individuals Da Nang Rehabilitation Hospital Disabled Group Dak Koi commune Agricultural Extension Group Dak Ruong commune Agricultural Extension Group Dak To Lung commune’s Women Union Micro-credit Group Dak Tolung Agricultural Extension Individual DOET in KRD Individuals Duc Pho Health Center Disabled Individual Dak To Lung Commune Management Unit Individuals East Meets West (EMWF) Foundation staff Individuals ECCE Centers’ Teachers and Staff Group Headmaster of Kon Ray school, Mr. Dang Van Thoai Individual KRBS current students Group KRBS past students Group KRBS Teachers Group KRBS Management and Staff Group Kon Ray People’s Committee Individuals Kon Tum DOET Individuals Kon Tum Dept. of Planning and Investment Individual Kon Tum Study Promotion Association (SPA) Literacy Individual Kon Tum Vet Department (Department of Agriculture) Husbandry training Individual Parent Organizations Group KRBS Project End-Of-Project Evaluation 71 EVALUATION STAKEHOLDERS STAKEHOLDER TOPIC INDIVIDUAL OR GROUP MEETING Parents with children currently enrolled in the school Group Parents and Community Members from Early Childhood Activity’s benefiting communes Group Parents with children who have completed their education at Kon Ray Group Recipients of microenterprise assistance Group Recipients of agriculture extension training Group Teachers who received training Group People's Inspection Team (PIT) Construction Group Quang Ngai Orthopedics Station Disabled Individual Roadmonkey Voluntourism On Line Student Promotion Association (SPA) members (volunteers) Group University of Da Nang (UD) Professors (trainers), etc. Teacher Training (Dr. Duong Mong Ha) Individuals VN Help Fundraising On Line WU Individuals Youth Union Individuals 72 KRBS Project End-Of-Project Evaluation ANNEX G: DOCUMENTS REQUESTED FROM EMW 1. EMW original budget application for first proposal to USAID (2005) 2. EMW second budget application for second supplemental application to USAID (2009) 3. Original cooperative agreement between USAID and EMW (2005) 4. EMW scoping statement and environmental assessment (submitted to USAID) 5. Modifications 1, 2, 5, 7 (these are missing from our documents) 6. Training curriculum for all University of Danang trainings provided to EMW project stakeholders 7. EMW Performance Monitoring Plan (submitted to USAID) 8. EMW benchmark/baseline data 9. People with disabilities survey for Kon Tum 10. Indicator monitoring reports on student achievement: enrollment rate, dropout rate, promotion rate. 11. Copy of the PowerPoint presentation Hein made for SI 12. Financial Management Plan (June 2009) soft copy 13. List of 27 staff at Kon Ray School- their names and positions and titles 14. MOU with service provider who delivered the agriculture extension trainings to farmer families 15. MOU with the Commune Savings and Credit Management Unit for micro-credit loans 16. Ag extension and Animal husbandry needs assessments (as cited in Oct-Dec. 07 Progress report. 17. 5 Agriculture extension models that were developed for the training 18. Baseline data ( cited in output 2 under objective 5 from the Oct-Dec. 07 Progress report) 19. Legal documents for EMW to work in Kon Tum (as cited in the Jan-March 08 progress report) 20. January 2009 agreement on acceptance and implementation of the project between EMW and Kon Tum People’s Committee 21. School maintenance plan – (as cited in the Jan-Mar 2009 progress report) 22. The annex to the secondary application to USAID for the survey of disabled students 23. The first communication from KTP to EMW requesting assistance for the school 24. The needs assessment that EMW conducted in Kon Ray to decide that Dak Tu Lung should be the site for the school. 25. Letter of invitation and support from Kon Tum Department of Education 26. Current state of enrollment of students in KRD in lower secondary school 27. The decision letter from the government Provincial People’s committee (2010) to say they will finance the school issued by the department of training and education 28. The financial report on the funding spent by EMW to build the pig pen facility 29. The letter from EMW to the EPA to cancel the payment for the volunteer kindergarten school monitors 30. MOU between EMW and the KRBS 31. MOU between EMW and KRD People’s Committee 32. MOU between EMW and the KRD Education Department 33. MOU between EMW and the Provincial People’s Committee 34. MOU between EMW and the Provincial Education and Training 35. MOUs between EMW: a. WU b. EPA KRBS Project End-Of-Project Evaluation 73 ANNEX H: KRBS PROJECT REVISED EVALUATION SCHEDULE VIETNAM SEPTEMBER 24 – OCTOBER 16, 2012 DATE ACTIVITY Monday, Sep. 24th Evaluation Team Planning Meeting at Sofitel – all day Tuesday, Sept. 25th 10:00 – 10:30 a.m. - Evaluation Team meetings with USAID 10:30 – 11:30 a.m. – Evaluation Team meetings with General Development Office Wednesday, Sept. 26th Evaluation Team meetings: The Department of Preschool and Dept. of Ethnic Minorities Education (MOET) Thursday, Sept. 27th Evaluation Team flies to Danang: Depart Hanoi 18:00, Arrive Danang 19:15 Friday, Sept. 28th Evaluation Team meets with EMW Foundation Saturday, Sept. 29th Evaluation Team work day – all day Sunday, Sept. 30 Overland travel (drive) from Danang to Kon Tum Monday, Oct. 1st Meetings in Kon Tum with the following entities: People’s Committee Dept. of Foreign Affairs Department of Education Department of Planning WU Department of Learning Promotion Visit to Kindergartens in Kon Tum Town: Hai Mai Nang Mai Tuesday, Oct. 2nd Meetings with KRBS Staff: Headmaster Management Meetings with the following entities: KRD, BOET Kon Ray People’s Council Wednesday, Oct. 3rd FGDs with the following entities: Teachers Parents Current students from each grade-level Thursday, Oct. 4th Evaluation Team returns to Danang Friday, Oct. 5th Meetings with EMW in Danang Saturday, Oct. 6th Evaluation Team work day in Danang Sunday, Oct. 7th Evaluation Team work day/rest day in Danang Monday, Oct. 8th Evaluation Team meets with EMW Tuesday, Oct. 9th Evaluation Team meets with the following entities: University of Danang teachers and trainers involved in KRBS Program Director, Danang Orthopedic & Rehabilitation Hospital Wednesday, Oct. 10th Evaluation Team final meeting with EMW and travel back to Hanoi Thursday, Oct. 11th Evaluation Team work day Hanoi Friday, Oct. 12th Evaluation Team work day Hanoi Saturday, Oct. 13th Evaluation Team work day and meeting with EMW Hanoi staff Sunday, Oct. 14th Evaluation Team work day/rest day Hanoi Monday, Oct. 15th Evaluation Team PowerPoint presentation for USAID Tuesday, Oct. 16th Evaluation Team departs Hanoi