28 January 2013 This final report was prepared for Mongolia Mission of the United States Agency for International Development by Barbara V. Braatz and Clement Tingley, PE, under Project Order Number AID-438-O-13-00001 and Consultant Project Order under AID-OAA-C-11-00154. EVALUATION OF THE ULAANBAATAR SCHOOL BUILDINGS THERMO-TECHNICAL RETROFITTING PROJECT FINAL REPORT Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project ii FINAL REPORT EVALUATION OF THE ULAANBAATAR SCHOOL BUILDINGS THERMO￾TECHNICAL RETROFITTING PROJECT Prepared by: Barbara V. Braatz, Team Leader Clement Tingley, PE 28 January 2013 DISCLAIMER The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project iii ACKNOWLEDGEMENTS This evaluation required extensive document review, numerous interviews, and multiple site visits to Ulaanbaatar School Nos. 79 (Buildings 1 and 2) and 63. We would not have been able to complete this evaluation without the generous assistance of the staff of USAID/Mongolia and GIZ, the school directors and staff of Ulaanbaatar School Nos. 79 and 63, project partners in the Government of Mongolia and Ulaanbaatar City Government, and project engineers, architects, and contractors. We would especially like to thank Ms. Mendsaihan Hasbaatar (Project Specialist/Program Officer) and Mr. Michael Foster (Program Officer) of USAID/Mongolia, for their help and guidance in designing and implementing this evaluation, and for their hospitality during our time in Ulaanbaatar. We would also especially like to thank Ms. Ruth Erlbeck (Director, Integrated Urban Development Program) and Mr. Ralph Trosse (Technical Director, Integrated Urban Development Program) of GIZ for arranging our site visits and interviews, providing us with numerous documents and data rapidly, and responding to all of our questions thoroughly and with great patience. We would also like to thank the following individuals for their insights and time in answering our questions: Mr. S. Tuvshinkhuu (Program Officer), Mr. S. Tserendash (Senior Officer), and Ms. Sabine Muller (Country Director) of GIZ; Mr. D. Bold (Director, School No.79); Mr. S. Erdenebaatar (Director, School No.63); Mr. Ts. Bayarbat (Director General, Urban Development and Land Affairs Policy, Ministry of Construction & Urban Development); Mr. Bayarmagnai (Specialist, Education Department of Ulaanbaatar City Government); Ms. N. Tuya (Director of External Cooperation Division, Ministry of Education & Science); Dr. Ya. Duinkherjav (Professor of Structural Construction and Chairman of Civil Engineering Department, Mongolian University of Science & Technology); Mr. A. Tsogt (Senior Officer, Building Energy Efficiency Project); Mr. B. Gantumur (Executive Director, Mongolian Association of Civil Engineers); Mr. Ch. Tserenmyadag (Director, CH Construction Co. Ltd), Ms B. Gerelmaa (CEO, Gikon Co. Ltd); Ms. Ralkhaasuren (Accountant & Manager, PST Co. Ltd); and Mr. P. Naranbaatar (Director, KonsKomm Co. Ltd). Finally, we would like to thank Ms. Baasankhuu Damba for her able translation service, and Mr. Sukhbat for his dependable logistical support. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project iv ACRONYMS AND ABBREVIATIONS ADS Automated Directive System BEEP Building Energy Efficiency Project CO2 carbon dioxide CO2-eq CO2-equivalent CH4 methane GASI General Agency for Specialized Inspection GHG greenhouse gas GOM Government of Mongolia GIZ Gesellschaft für Internationale Zusammenarbeit (German International Cooperation) HOB heat-only boiler IPCC Intergovernmental Panel on Climate Change MES Ministry of Education & Science MNT Mongolian tugriks MUST Mongolian University of Science & Technology N2O nitrous oxide PM particulate matter RDMA Regional Development Mission for Asia SOW Statement of Work UB Ulaanbaatar USAID United States Agency for International Development Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project v UNITS cm centimeter Gcal Gigacalories (109 calories) kcal kilocalories (103 calories) kg kilogram kW kilowatt m meter t ton (metric ton) yr year Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project vi CONTENTS Acknowledgements .................................................................................................................. iii Acronyms and Abbreviations ....................................................................................................iv Units ........................................................................................................................................... v Contents ....................................................................................................................................vi Executive Summary ................................................................................................................... 1 Introduction .................................................................................................................................................... 1 Evaluation Methodology .............................................................................................................................. 2 Findings and Conclusions............................................................................................................................. 2 Thermo-technical Retrofits to School Buildings ................................................................................ 2 Energy and GHG Reductions ............................................................................................................... 4 Knowledge and Skill Building ................................................................................................................ 4 Recommendations and Lessons Learned ................................................................................................... 5 Recommendations for the Project ........................................................................................................ 5 Lessons Learned and Recommendationsfor Future Programming ................................................. 6 Introduction ............................................................................................................................... 8 Purpose of the Evaluation ........................................................................................................10 Evaluation Methodology .........................................................................................................10 Project Description .................................................................................................................. 11 Findings and Analysis ..............................................................................................................14 Thermo-technical Retrofits to School Buildings ....................................................................................14 Energy and GHG Reductions ...................................................................................................................19 Knowledge, Skill Building, and Gender Implications ............................................................................23 Conclusions ..............................................................................................................................25 Thermo-technical Retrofits to School Buildings ....................................................................................25 Energy and GHG Reductions ...................................................................................................................27 Knowledge, Skill Building, and Gender Implications ............................................................................28 Recommendations and Lessons Learned ................................................................................29 Recommendations for the Project ............................................................................................................29 Lessons Learned and Recommendations for Future Programming ....................................................30 Appendices...............................................................................................................................32 Appendix A. Evaluation Scope of Work ................................................................................................. 33 Appendix B. Final Evaluation Design and Work Plan ..........................................................................42 Appendix C. Engineer’s Field Visit Report .............................................................................................53 Appendix D. Conflict of Interest Statements .........................................................................................60 Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 1 EXECUTIVE SUMMARY INTRODUCTION In November-December 2012, Barbara V. Braatz and Clement Tingley, PE conducted a final performance evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project for USAID/Mongolia. The purpose of the evaluation was to determine the extent to which the project achieved its energy, greenhouse gas (GHG), and learning objectives, and to develop recommendations for the project and for future programming of a similar type. The evaluation team was also asked to answer five specific questions about the outcomes of the project. The overall goal of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project was to demonstrate the heat energy savings that could be achieved through thermal insulation and heating system improvements at three school buildings. The selected buildings (School Nos. 79-1, 79-2, and 63) are located in ger districts of the Mongolian capital city Ulaanbaatar (UB), are not connected to the UB district heating system, and are heated with coal-fired heat only boilers. The buildings were determined to be structurally sound prior to the project, but they lacked adequate insulation and were in poor condition – there was frost on the insides of some classrooms during the winter months, roofs were leaking, floors lacked insulation, and there were significant cracks in exterior walls. The project was designed to achieve the following results:  increased efficiency of energy use in the three buildings, and consequent reductions in coal consumption, coal costs, and coal-related GHG emissions;  a more comfortable learning environment for children and staff at the schools; and  trained and knowledgeable local builders, engineers, and architects who are able to design and implement retrofits. In addition, the project design included the preparation of a learning module on energy efficiency, renewable energy, and climate change for distribution to the rehabilitated schools. The specific improvements that were made to each building varied. All three buildings received thermal envelope and related building improvements (insulation of walls, floor, and roof; new higher efficiency windows; new exterior doors; ventilation system cleaning at the roof; vent rain guards; parapet wall caps; improved rainwater drainage). School No. 79-2 and School No. 63 also received heating system improvements (cleaning and any necessary replacement of the hot water circulation system and radiators, installation of radiator operating valves, replacement of the old boilers with new higher efficiency boilers that can be fired with biomass or coal), and School No. 63 received instrumentation to measure net boiler heat output and a new interior roof drainage system. School No. 63, a school for handicapped children selected by the project to be an example of “best practice” of a child friendly and environmentally safe school, also received sanitary and handicapped improvements (solar hot water system, new bathrooms, classroom sinks, a new septic system, a new well, and handicapped ramps). These sanitary/handicapped retrofits are not part of this evaluation so their quality was not assessed; they are mentioned here as background information and because this evaluation noted a couple areas for follow-up action. USAID contributed a total of $1.517 million to support the project, approximately 90 percent of which was used for the thermal-technical retrofits. The remaining portion was used for the sanitary/handicapped retrofits at School No. 63. Deutsche Gesellschaft fur Internationale Zusammenarbeit GmbH (GIZ) was the lead implementing partner, and served as the program manager and technical advisor to the project. Qualified Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 2 Mongolian architectural, engineering, and construction companies prepared the final architectural plans and completed the rehabilitation of the school buildings. Project activities began in November 2011 and ended in December 2012. The retrofitted buildings were officially approved by the Ulaanbaatar General Agency for Specialized Inspection (GASI) on 18 October 2012. EVALUATION METHODOLOGY The evaluation began with a desk review of project documents provided by USAID/Mongolia to: 1) assess project objectives and accomplishments; 2) determine and evaluate information collected to date; 3) determine what additional data are needed to complete the evaluation; and 4) establish a procedure for obtaining these data. The desk review was followed by eight days of field work between 3 and 12 December 2012, which coincided with the last month of the project. The field work consisted of reviews of the architecture plans, several visits to all three school buildings, and interviews of key informants. The key informants included management staff and engineers at GIZ; the directors of Schools 79 and 63; Mongolian engineering, architectural, and construction partners; and officials at partner Ministries and UB City government departments. The site visits and interviews were used to assess the extent and quality of the building and heating system improvements at each site, clarify information provided to the evaluation team, and obtain additional data for analysis. FINDINGS AND CONCLUSIONS THERMO-TECHNICAL RETROFITS TO SCHOOL BUILDINGS The thermal-technical retrofits to School Nos. 79-1, 79-2, and 63 have transformed exceedingly cold, drafty, and leaky buildings into warm and comfortable environments conducive to learning. The resultant comfort levels in the school buildings have exceeded the expectations of the school staff and students, as well as some project engineers and contractors. School staff and students are pleased with the new environment, more proud of their schools, and expect learning to improve as a result of the retrofits. Two unplanned benefits of this project are that the teachers and some parents at School No. 63 were so enthusiastic about the reincarnation of the building that they repainted the interior of the facility themselves, and people who live near School No. 63 are using the building as a type of informal community center for sports and other activities during non-school hours. Overall, the evaluation team found that the quality of workmanship in both the construction plans and the construction was high, that building envelop improvements were consistent with the plans, and that high quality materials were used. Project partner architects, engineers, and contractors, as well as school directors, noted the unusually thorough and effective construction site supervision and project management, which ensured that construction procedures adhered to plans, that installation errors were fixed immediately, and that project partners were well coordinated. The evaluation team observed some problems with the heating system improvements and operation of the new building environments. First, the new boilers at School Nos. 79-2 and 63 were not in service when the evaluation team visited. The evaluation team was told that the boilers did not function properly when they were installed, and that GIZ had implemented several manufacturer-suggested solutions, but they were not successful. The old boilers were re-installed at School Nos. 79-2 and 63 at the start of the heating season. When the evaluation team left Mongolia on 16 December 2012, GIZ was working to retrofit the new boilers so they would operate. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 3 Several individual rooms in the three buildings, especially classrooms full of children, were found to be uncomfortably hot. One classroom was measured at 30° Celsius (86° Fahrenheit). Several windows were open at all three buildings when evaluation site visits were made. In response to questions about room temperature control, teachers indicated that they open or close the windows. The project did not include installation of valves on the radiators at School No. 79-1 (or any other heating system upgrades at School No. 79-1), so windows provide the only available means of regulating room temperature when classes are in session. The project also did not include instructing teaching staff at School Nos. 79-2 and 63 on the use of radiator valves to moderate classroom temperatures. There was a high amount of condensation on the windows at School No. 79-1, and less significant condensation on windows at School No. 63. This could be due to moisture release from the masonry walls, over-crowding, inadequate ventilation systems, and/or obstructed ventilation systems. Moisture release from the masonry walls is believed to be a significant contributor due to rainwater infiltration prior to and during construction. Condensation should lesson as the winter heating season proceeds and the walls dry out. The project included cleaning ventilation system outlets at the roof, but not retrofits of ventilation systems. The ventilation systems may be undersized for the number of people in the buildings, so overcrowding may continue to cause excess condensation during winter months. However, the evaluation team inspected the roof vents of the ventilation system at School No. 79-1, and found that a number of the vents were partially or fully obstructed by frost, and one vent rain guard was missing. Clearing the vent openings and replacing missing vent guards may reduce condensation levels, and alleviate overheating of rooms. The glass in one window School No. 79-1 was cracked. All windows were intact when the building was inspected by GASI on 20 September 2012, so the break occurred after the retrofits were completed and officially approved. The broken window is not the responsibility of the project, but it negatively affects the energy efficiency of the building. The project installed instrumentation that measures net boiler heat output over time in the boiler room at School No. 63. Monitoring this heat meter, in conjunction with monitoring interior temperatures and moisture conditions School No. 63, can facilitate the efficient operation of the heating system to maximize efficiency gains and building comfort levels. The project did not include such post-retrofit monitoring of heat output and building conditions at School No. 63, or installation of a heat meter and post-retrofit monitoring at School Nos. 79-1 and 79-2. The heating system operations at School No. 79 are outsourced to a state￾owned company under the UB City Government, which means that installation of heat meters at School Nos. 79-1 and 79-2 would require complex and time-consuming negotiations among the state owned company, the UB City Government, and the school director. The director of School No. 63 retains control of heating system operations at his building. An un-insulated brick building to be used for teacher housing is being connected to the heating system of School No. 79-2. This is outside of the control of the project, but GIZ has advised School No. 79 not to make the connection because this will negatively affect the functioning of the heating system. Two issues regarding the sanitary improvements at School No. 63 were noted by the evaluation team. Prior to this project, pit latrines were built at School No. 63 because the interior toilets could not be used during the winter months. (The old septic lines were too shallow, and froze in the winter. This problem was solved by the project’s installation of an entirely new septic system). One latrine remains at the site. The Director of School No. 63, who has official “ownership” of the school now that it has been approved and handed over, has promised to remove the pit latrine once the ground has thawed after the winter. Until that happens, the latrine is unattractive and a potential hazard. Also prior to this project, the water in the old well was found to Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 4 have bacteriological contamination. A 13 December 2012 water quality test of the new well that was installed by this project found that the water met the Mongolian standard with regard to bacteriology, but did not meet Mongolian standards for color, smell, turbidity, and metals. A second test will take place between 7-8 January 2013, and the results will be reported to USAID/Mongolia. Adequate water quality should be verified. ENERGY AND GHG REDUCTIONS The project did not allow for the collection of actual pre-retrofit heat demand and coal consumption data, and not enough time has elapsed for collection of comprehensive post-retrofit data, so GIZ derived pre￾retrofit (baseline) heat demand, fuel consumption, and GHG emission estimates through modeling. This evaluation uses the results of GIZ’s heat demand model, and revised input data and methods for the modeled fuel and GHG estimates. Based on the heat demand model, it is estimated that the thermal-technical retrofits to School Nos. 79-1, 79- 2, and 63 will reduce annual heat loss by an average of approximately 40 percent per year. Actual reductions will vary, especially depending on how the heating systems and buildings are operated. With the old boilers in operation and coal as the fuel, estimated annual coal consumption, coal costs, and fuel-related GHG emissions will also decline by an average of 40 percent per year. If the new boilers at Schools Nos. 79-2 and 63 are installed successfully by January 2013, and they operate at their rated efficiency and are fired with coal through the end of the heating season, the estimated percentage reduction in coal demand, fuel costs, and GHG reductions at those two buildings would be greater by about half. It is unlikely that biofuels would be used in the new boilers because of the high cost and limited supply of biofuels in UB. In answer to the first three specific evaluation questions about project outcomes, the evaluation team estimates that, based on the heat demand modeling and the boilers and fuel currently in operation at the schools, the thermal-technical retrofits to Schools Nos. 79-1, 79-2, and 63 will: 1. Reduce annual heat loss per square meter by an average of approximately 46 percent per year. 2. Reduce GHG emissions per unit of heat demand by an average of approximately 0.9 metric tons of carbon-dioxide-equivalent per gigacalorie of heat demand. 3. Reduce coal-related GHG emissions by an average of approximately 40 percent per year. The heat meter that was installed at School No. 63 can be used to monitor actual net boiler heat output, from which post-retrofit fuel consumption and GHG emissions at School 63 could be recalculated. The readings from this meter would also facilitate future assessment of the accuracy of the heat demand model. The project did not include such post-retrofit monitoring of heat output at School No. 63, or re-assessment of the heat demand modeling and fuel consumption and GHG estimates. Installation of similar instruments at School Nos. 79-1 and 79-2 would have enabled more thorough evaluation of the energy modeling, and more accurate calculations of post-retrofit fuel consumption and GHG emissions, but such installation would have required complex and time-consuming negotiations (described above). KNOWLEDGE AND SKILL BUILDING The final two of the five specific evaluation questions about project outcomes asked: 1) how much local capacity to design and implement retrofitting technology was built; and 2) if knowledge about energy efficiency increased among school staff. Knowledge increase among school staff was not an explicit part of the project design, although it would be an anticipated outcome of the project if the learning module, which is an output of this project, is effective and is utilized. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 5 Based on interviews of GIZ staff and engineers, and Mongolian engineering, architectural, and construction partners, the evaluation team found that the project resulted in an increase in local knowledge and skills in retrofitting technologies (materials and installation techniques), and in construction site and project management. This resulted from effective on-the-job training, construction site supervision, and overall project management. The amount, or level, of increase in knowledge and skills for individual construction workers cannot be assessed because the evaluation team did not interview individual workers due to time constraints and unavailability (many were temporary workers). However, directors of Mongolian partner companies noted that their experience in this project has provided them with valuable skills and experience that will give them a competitive advantage for future work of this kind. One construction company stated that they had already been contacted by the UB government about a similar project that will be open for bids next year. It is clear from our interviews that the UB government is paying greater attention to building energy efficiency in both new construction and existing buildings, and there is a growing demand for engineers and architects who have expertise in building energy efficiency. However, cost remains a significant hurdle to retrofits of public buildings. Based on interviews of the school directors and a few brief conversations with school teachers, the evaluation team found that this project has increased the awareness and understanding of the benefits of improved building energy efficiency among school staff. This has resulted largely through demonstration – the transformation of their school buildings and the comfort in which they now operate has exceeded their expectations, and learning is expected to improve as a result of the project. Both directors anticipate significant coal savings this year as a result of the retrofits; one director commented that he had already noticed significant reductions in fuel use. However, teachers in School Nos. 79-2 and 63 do not appear to be aware of the availability of radiator operating valves for temperature control, and there may be measures that the heating system operators can implement to maximize both comfort and fuel savings. This project has contributed to awareness of the benefits of building retrofits beyond the staff at School Nos. 79 and 63. One school director has already received visits from other school directors and private business owners who want to learn more about the retrofits and find out how they “can get this done” at their building. In addition, this project has contributed to the awareness among UB and national government entities of the benefits of building retrofit technologies, and should contribute to ongoing efforts in the national government to develop new energy efficiency standards and codes for the construction sector, including retrofits to existing buildings. GIZ prepared a learning module, in the form of a booklet in the Mongolian language, on energy and climate change for secondary school teachers and students, and distributed copies of the booklet to School Nos. 79 and 63 in late November. It is too early to determine how the learning module will be used and whether it will increase learning for either teachers or students. However, it combines a useful mix of conceptual explanations, simple experiments, and questions to facilitate learning, and the school directors have already asked their secondary school teachers to develop suggestions for incorporating the material into the curriculum. RECOMMENDATIONS AND LESSONS LEARNED RECOMMENDATIONS FOR THE PROJECT The evaluation team recommends that the project undertake the following actions:  Repair and reinstall the new boilers at School Nos. 79-2 and 63. If this is unsuccessful, the new boilers should be replaced with new coal-fired boilers with similar or higher efficiency ratings and Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 6 that are appropriate for the quality of coal available in UB. The new boilers should be monitored throughout the 2012-2013 winter months to ensure continued operation.  Inspect the roof vents on all three buildings, clear all the vents of frost, and replace all missing vent rain guards. Instruct building operations staff to check and clear roof vents regularly throughout the winter.  Review the results of the second well water quality test at School No. 63. If the results still do not meet Mongolian standards, find and eliminate the source of contamination and/or implement treatment measures as needed.  Verify that the pit latrine at School No. 63 is locked so that it cannot be used. Once the ground has thawed at the end of the winter, verify that the latrine has been removed. If the latrine is not removed, remind the school director that it should be removed. LESSONS LEARNED AND RECOMMENDATIONS FOR FUTURE PROGRAMMING Based on the lessons learned from this project, the evaluation team recommends that future programming of this type consider including activities that would improve operations of retrofitted buildings, and improve project learning. These recommendations have both cost and project lifetime implications, but their implementation would help ensure that improvements in building comfort and energy efficiency are maximized, would reduce uncertainties in the estimated energy and GHG benefits of the retrofits, and would increase the effectiveness of educational tools. Building Operations Given the high temperatures and condensation levels in some classrooms, the presence of open windows at all buildings, the cultural inclination to use the windows to regulate indoor temperatures even when outdoor temperatures are tens of degrees below freezing, and the availability of heat system instrumentation to monitor heat demand over time, the evaluation team recommends that future programming of this type include:  Cleaning and any necessary replacement of hot water piping system and radiators, and installation of radiator valves, at all buildings to ensure that all hot water systems operate efficiently and that end￾users are able to control radiator heat. This clearly has cost implications but would improve comfort levels and increase energy efficiency gains.  Installation of a heat meter at all buildings, monitoring of net boiler heat output and temperature and moisture conditions in the retrofitted buildings for a year following construction, collaboration with the heating system operators to design and implement optimal heat management procedures, and training of teaching staff and other end-users in optimal end-user heat management procedures (managing radiator valves, windows, doors, classroom vent openings). Also, the glass in one window at School No. 79-1 was broken within two and a half months of retrofit completion. The evaluation team recommends that for projects of this type, building maintenance staff be trained in window repair, and the availability of replacement units be verified or an appropriate number of replacement units be provided to the building managers at project end to ensure that repairs can be made in a timely fashion. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 7 Project Learning To increase the accuracy of estimated energy and GHG benefits of projects of this kind, the evaluation team recommends that future similar programming include:  Collection of annual coal consumption data for the buildings prior to the retrofits and after the retrofits, and collection of information on coal characteristics (calorific values, moisture contents), sources (mines and mine characteristics), transport conditions (distance, mode), and prices (variability by quality and time of purchase).  Use of heat meters to monitor net heat output (heat demand) at all buildings over the entire heating season after the retrofits, and ideally prior to retrofits over an entire heating season as well. If a meter is not present at all buildings and/or pre-retrofit monitoring is not possible, the results of monitoring should be used to calibrate the model used to estimate heat demand at the unmonitored buildings. Monitored heat demand values in combination with data on coal consumption data and calorific values would enable actual boiler efficiencies to be derived, and compared against rated efficiencies. To raise awareness of the quantitative costs and benefits of thermo-technical retrofits, the evaluation team recommends that future similar programming include:  Development of briefing materials (brochures and presentations) that contain quantitative information about the costs and energy and GHG benefits of thermal-technical retrofitting of buildings. These materials should incorporate quantitative information obtained through implementing the previous two recommendations. To improve USAID’s abilities to provide effective educational tools, the evaluation team recommends that future programming that involves the development of tools similar to the learning module prepared by this project include monitoring and evaluation of outcomes associated with the tool, including whether and how the tool was used, what aspects of the tool were more beneficial and why, and what aspects of the tool were less beneficial and why. In addition, the evaluation team recommends that project lifetimes be long enough to allow for preparation of a draft version of the educational tool and field testing prior to publication of the final version. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 8 INTRODUCTION Fossil fuel combustion is the largest source of greenhouse gas (GHG) emissions in Mongolia, accounting for almost 90 percent of national gross emissions from all sectors.1 About 80 percent of the emissions from fossil fuel combustion come from stationary sources (e.g., heat plants, power plants, stand-alone boilers for heating public and commercial space, and stoves for residential heating and cooking), and carbon dioxide (CO2) is by far the most important GHG emitted, accounting for approximately 95 percent of combustion emissions. Annual per capita CO2 emissions from fossil fuel use in Mongolia are high compared to other countries with similar levels of economic output and income, averaging 4.31 metric tons carbon dioxide per capita (4.31 t CO2/capita) in 2010. This is due to several factors.2 First, the length and intensity of Mongolia’s winters result in high per capita energy use for heating. Wintertime temperatures in the capital city Ulaanbaatar (UB) regularly fall below minus 30° Celsius3 at night, making it the coldest capital city in the world, and heating is needed for nine months. Second, Mongolia’s almost complete reliance on coal as an energy source for heating, as well as electricity generation, results in a high carbon intensity of energy use. And third, energy efficiencies on both the supply side (power and heat production, transmission, and distribution) and the end￾use side (stand-alone heating systems, building envelopes, and lighting) are low. Ulaanbaatar has experienced rapid population growth over the last decade, primarily due to in-migration from rural areas. Over one million people, or over 40 percent of Mongolia’s population, live in UB, and UB’s population is growing at almost 3 percent per year.4 Many migrants settle in peri-urban “ger” settlements, which currently comprise about 60 percent of the city’s population. The ger settlements consist of nomadic felt and canvas tents (gers) and self-constructed, poorly-insulated wood or brick detached houses. Most of UB’s ger settlements are not connected to the city’s district heating system, so households rely on simple coal-fired stoves or simple coal-fired, heat-only boilers (HOBs) for heating. These stand-alone heating systems are not only energy inefficient, they are also the primary reason why UB is one of the world’s most polluted cities with respect to particulate matter during the winter months.5 Exposure to high concentrations of particulate matter (PM) increases the risk of cardiopulmonary conditions, acute and chronic respiratory infections, and lung cancer. A recent World Bank study, which monitored 2009 PM concentrations in UB, found that the calculated exposure of the city’s population to PM2.5 was, on average throughout the year, 10 times higher than the Mongolian Air Quality Standards and 6-7 times higher than the most lenient World Health Organization targets.6 1 Mongolia Second National Communication under the United Nations Framework Convention on Climate Change, 2010, http://unfccc.int/essential_background/library/items/3599.php?such=j&symbol=MNG/COM/2%20E#beg 2 International Energy Agency, 2012, Key World Energy Statistics, http://www.iea.org/publications/freepublications/publication/kwes.pdf For comparison, the 2010 per capita CO2 emissions from fossil fuel use in Kyrgyzstan and the Republic of Moldova, which had gross national product (GDP) and purchasing power parity (PPP) levels similar to Mongolia, were 1.3 and 1.72 t CO2/capita, respectively. 3 This is equivalent to minus 22° Fahrenheit. 4 United Nations Statistics Division, 2011, Country Profiles 2000-2010, http://unstats.un.org/unsd/pocketbook/country profiles.pdf 5 The World Bank, 2012, Ulaanbaatar Clean Air Project, http://documents.worldbank.org/curated/en/2012/02/15940671/mongolia-ulaanbaatar-clean-air-project 6 The World Bank News and Views, 25 April 2012, http://www.worldbank.org/en/news/2012/04/25/curbing-air￾pollution-in-mongolia-capital PM2.5 is particulate matter with an aerodynamic diameter of 2.5 microns or less. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 9 The public school buildings that serve the ger settlements typically lack proper thermal insulation, and usually are not connected to heat distribution systems. These schools rely on inefficient, coal-fired, HOBs for heating, and the pipes and radiators attached to the boilers are often inadequately maintained and lack individual heat control. This has not only GHG, fuel cost, and air quality implications, but also affects the comfort of the learning environment for students. Temperatures inside classrooms during the coldest winter months can be as low as 8° Celsius (46° Fahrenheit),7 especially in classrooms on the northern sides of school buildings. Floors without insulation and poorly insulated and leaky roofs further degrade an already sub￾optimal situation. The Government of Mongolia (GOM) gives a high priority to education, and is determined to improve the quality of and access to education in Mongolia, especially for young children.8 However, with the growth of ger settlements, many schools are overcrowded and there are not sufficient funds for new construction. At the same time, it is becoming increasing apparent that the demand for heat in UB will soon exceed the installed capacity, so there is a growing interest in retrofitting existing buildings to improve the efficiency of building energy use. However, there is little experience in retrofitting technologies in Mongolia, especially in public buildings. To demonstrate the potential heat energy savings of building retrofits, especially insulation and improved heating systems, USAID/Mongolia has partnered with Deutsche Gesellschaft fur Internationale Zusammenarbeit GmbH (GIZ) to implement the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project. This $1.517 million project has thermo-technically retrofitted three public school buildings that serve ger settlements in UB.9 Thermal rehabilitation of buildings is a process of retrofitting existing buildings with insulated walls, floors, and roofs, higher efficiency windows, and tighter external doors to improve heat retention and minimize air leakage. Rehabilitation may also include cleaning and upgrades to the heating systems (i.e., cleaning piping, replacing radiators, and installing higher efficiency boilers). The project chose to retrofit public school buildings in ger settlements because these schools serve low income populations, are in great need of building condition improvements, and have a high visibility. Also, a target population composed of public school children and staff in the areas of UB that are growing most rapidly provides an effective avenue for widespread demonstration and learning about the benefits of thermal rehabilitation. The project was designed to achieve the following results:  increased efficiency of energy use in the three buildings, and consequent reductions in coal consumption, coal costs, and coal-related GHG emissions;  a more comfortable learning environment for children and staff at the schools; and  trained and knowledgeable local builders, engineers, and architects who are able to design and implement retrofits. 7 The target room temperature in Mongolian schools is 20° Celsius, or 68° Fahrenheit. 8 GIZ, October 2011, Project Design Document appended to USAID Grant Agreement between the Deutsche Gesellschaft Fur Internationale Zusammenarbeit (GIZ) GmbH and the United States Agency for International Development for the Ulaanbaatar School Buildings Thermo-Technical Retrofitting Project, USAID Grant Number 438- OUGA-12-001, signed 23/11/2011. 9 Ibid. This project also made sanitary and handicapped improvements at one of the school buildings (School No. 63). Approximately 90 percent of the project funds were for the thermo-technical rehabilitation of the three school buildings; the remaining 10 percent were for the sanitary and handicapped improvements at School No. 63. Only the thermo￾technical portion of the project is the subject of this evaluation. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 10 In addition, the project design included the preparation of a learning module on energy and climate change for secondary school teachers and students, and distribution of the module to the retrofitted schools. GIZ served as the program manager and technical advisor to the project, while the actual rehabilitation of school buildings was completed by qualified Mongolian engineering and construction companies. The thermo-technical improvements that were made to each school building varied among the three buildings, and are described below in the Project Description. PURPOSE OF THE EVALUATION The purpose of this final performance evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project is to determine the extent to which the project achieved its energy and GHG objectives, to evaluate the strengths and weaknesses of the project design and implementation, and to develop immediate recommendations for the project and longer-term recommendations to inform the design and development of future projects by USAID. In addition, USAID/Mongolia specifically directed the evaluation team to answer the following questions: 1. What is the percentage change in heat loss/square meter after the retrofits were completed? 2. What is the emission reduction per unit of energy saving? 3. What is the estimated percentage decrease in GHG emissions as a result of the retrofits? 4. How much local capacity was built to design and implement retrofitting technology as a result of the project? 5. Did the level of knowledge about energy efficiency increase among school staff as a result of the project? EVALUATION METHODOLOGY This evaluation was conducted by a team of two international consultants with expertise in performance evaluation, clean energy and GHG analysis, building construction, and building energy management. Barbara V. Braatz served as Team Leader and Evaluation and Clean Energy Specialist, and Clement Tingley, PE served as Civil Engineer. Both team members participated in all phases of the evaluation. The Evaluation Statement of Work (SOW) is presented in Appendix A, and the Final Evaluation Design and Work Plan is presented in Appendix B. Table 1 of the Final Evaluation Design and Work Plan contains the work plan schedule, including deliverables and their due dates, and the roles and responsibilities for each Evaluation Team member. The evaluation started with a desk review of project documents provided by USAID/Mongolia to: 1) assess project objectives and accomplishments; 2) determine and evaluate information collected to date; 3) determine what additional data are needed to complete the evaluation; and 4) establish a procedure for obtaining these data. Based on this desk review, the evaluation team compiled a brief summary of information collected from the desk review and key information gaps, and prepared a Draft Evaluation Design and Work Plan. The Draft Evaluation Design and Work Plan included a description of the evaluation approach and methods, a preliminary interview schedule, a preliminary informant interview guide for each Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 11 type of informant (architects and engineers, contractors, school personnel, and government officials), and a direct observation checklist to facilitate record keeping during site visits. The desk review was followed by eight days of field work between 3 and 12 December 2012, which coincided with the last month of the project. The field work consisted of reviews of the architecture plans, several visits to all three school buildings, and interviews of key informants (see Table 2 of the Final Evaluation Design and Work Plan [Appendix B] for the final interview schedule). The key informants included management staff and engineers at GIZ; the directors of School Nos. 79 and 63; Mongolian engineering, architectural, and construction partners; and officials at partner Ministries and UB city government departments. The site visits and interviews were used to assess the extent and quality of the building and heating system improvements at each site, clarify information provided to the evaluation team, and obtain additional data for analysis. These data included technical and engineering information about the retrofitted buildings, heating technologies, heating fuels; technical information about the energy and GHG modeling; and qualitative information with which to evaluate whether the project increased local knowledge and skills in the design and installation of building retrofit technologies, and whether the project increased knowledge about energy efficiency among school staff. The interview guides that were prepared before the site visits began were revised as the interview process proceeded and the specific roles of the informants were clarified. At the completion of the interviews, the evaluation team revised the Draft Evaluation Design and Work Plan to include the final interview schedule and the final interview guides (the final interview guides are contained in the Final Evaluation Design and Work Plan [Appendix B]). There are several limitations of the evaluation. First, no actual heat output or fuel use data were collected prior to the retrofits or have been collected since the retrofits were completed, so the heat output and fuel use benefits of the project are modeled estimates, rather than actual values. Second, the evaluation team visited the school buildings after all construction was completed, so many retrofit materials and installation methods could not be observed directly. Also, the evaluation team’s visits to each school building were short (only about a half-day in length each), and occurred only a few months after the retrofits were completed. It is likely that heating system operators and building occupants are still adjusting to their new environment, so conditions in the buildings may continue to change over the coming months. Note that about ten percent of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project funds were devoted to improving the sanitary and handicapped conditions at one of the schools (School No. 63, which serves handicapped children). These sanitary/handicapped retrofits are not part of this evaluation so their quality was not assessed; they are mentioned here as background information and because this evaluation noted a couple areas for follow-up action. PROJECT DESCRIPTION The Ulaanbaatar School Buildings Thermo-technical Retrofitting Project began in November 2011 and ended in December 2012. The overall project goal was to demonstrate the heat energy savings that could be achieved through thermal insulation and heating system improvements at the three selected school buildings, and to provide a model for rehabilitation of other schools in UB. The schools were selected based on the following criteria:  High visibility of buildings;  Located in ger (low income) districts;  Pupils from low income groups of the society; Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 12  Structural stability of the buildings; and  Urgent need for thermo-technical rehabilitation.10 The three selected school buildings are Buildings 1 and 2 at School No. 79 (referred to as School No. 79-1 and School No. 79-2 in this report and in project documents) and School No. 63, which has only one building. School No. 79 is a three-building complex located in the Bayanzurkh district east of the city center. Building 1 (School No. 79-1) was designed as a secondary school with a capacity for 320 children, and Building 2 (School No. 79-2) was designed as a kindergarten with a capacity for 380 children. At present, these buildings accommodate more than twice the number of students for which they were designed, and operate on two shifts. School No. 79 (all three buildings) accommodates 2178 children from grades 1 through 12.11 School No. 63 is a primary and secondary school designed for approximately 320 handicapped children and located in the Khan-Uul district near the airport southwest of the city center. At present, 250 children are enrolled at the school.12 School No. 79-1 was built in 1980. It is heated by a new heat only boiler that was installed prior to this project. The boiler has an efficiency rating of fifty percent. Prior to the project, the walls, roof, and floors of the school building were un-insulated. The windows were un-insulated and in poor condition. Winter temperatures in the classrooms varied between 8° to 9° Celsius (46° to 48° Fahrenheit) on the north side of the building, and between 16° to 18° Celsius (61° to 64° Fahrenheit) on the south side, so coats were required inside.13 Project improvements to the building included:  Thermal envelope improvements including walls, roof, and ground floor  New windows  Three new exterior doors and one interior door  Cleaning of the ventilation system at the roof and installation of rain guards and wall caps  Improved building perimeter substrate and grading for drainage from downspouts Project improvements to School No. 79-1 did not include upgrades to the heating system because it had a relatively new boiler and because of funding limitations. School No. 79-2 was built in 1986. It is heated by a single, older boiler that has an efficiency rating of forty percent. Prior to the project, winter temperatures in the classrooms did not exceed 18° Celsius (64° Fahrenheit),14 requiring the use of coats. The walls, roof, and floors were un-insulated. The windows were un￾insulated and in poor condition. Project improvements to the building and heating system included:  Thermal envelope improvements including walls, roof, and ground floor  New windows 10 Ibid. 11 Evaluation team interview of Mr.D. Bold, director of School 79, on 5 December 2012. 12 Evaluation team interview of Mr.S. Erdenebaatar, director of School 63, on 5 December 2012. 13 GIZ, October 2011, Project Design Document appended to USAID Grant Agreement between the Deutsche Gesellschaft Fur Internationale Zusammenarbeit (GIZ) GmbH and the United States Agency for International Development for the Ulaanbaatar School Buildings Thermo-Technical Retrofitting Project, USAID Grant Number 438- OUGA-12-001, signed 23 November 2011. 14 Ibid. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 13  Cleaning of the ventilation system at the roof and installation of rain guards and wall caps  Two new 60 kW high efficiency boilers  Cleaning and any necessary replacement of the hot water circulation system  Radiator cleaning and replacement, as necessary, and the provision of operating valves for the radiators  Improved building perimeter substrate and grading for drainage from downspouts School No. 63 was built in 1975 and was in the worst condition of the three schools prior to the project. The bathrooms did not function and sanitary facilities were located in an outdoor privy. The sports hall was barred from use because it could not be heated. Winter interior temperatures required the use of coats. Ice was present on some interior walls of the building. School No. 63 had an aging unregulated heat circulation system, windows in poor condition, no insulation of the thermal envelope, and problems with roof leakage. The old HOB in the school has an efficiency rating of forty percent. School No. 63 was selected to be an example of “best practice” of a child friendly and environmentally safe school, so it received the greatest level of thermal-technical improvements, and also received sanitary and handicapped improvements. Project improvements to the school included:  Thermo-technical retrofits: – Thermal envelope improvements including walls, roof, and ground floor – New windows, and window size and location adjustments in the sports hall – Two new high efficiency 80 kW boilers – Instrumentation to measure net boiler heat output – Cleaning and any necessary replacement of the hot water circulation system – Radiator cleaning and replacement, as necessary, and provision of operating valves for the radiators – New doors at school entrance – New interior roof drainage system – Cleaning of the ventilation system at the roof and installation of rain guards and wall caps  Sanitary/Handicapped retrofits (not part of this assessment): – Solar hot water heating for washing and cooking – Bathroom improvements including showers at the sports hall, sinks in the classrooms, and a disabled toilet – New sanitary waste disposal facilities – New well – New handicapped ramps GIZ served as the program manager and technical advisor to the project. The rehabilitation of the school buildings was completed by qualified Mongolian engineering and construction companies. GIZ led in: (i) design and implementation of the technical specifications for retrofitting, conducted in tandem with Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 14 Mongolian architects and securing approvals from the Administration of Land Affairs, Construction, Geodesy and Cartography; (ii) cost calculation (priced bill of quantities) in tandem with the selected local engineering company; (iii) tendering the construction/rehabilitation work, including: a design tender, tender of contractors, bid-response requirements including the program description, specifications for applicants and standards for measuring achievements and progress towards finalization, and selection of contractors for retrofitting; (iv) technical guidance, advice, and supervision during the design process and construction; and (v) on-the-job training for contractors, architects, and engineers of consulting offices. GIZ also developed the energy and GHG estimations, and developed and published a learning module, in the form of a booklet, for secondary schools on energy and climate change. At the start of this project and before retrofitting commenced, all three buildings were inspected by a Mongolian team of independent structural engineers, and were found to be structurally sound.15 On 14-16 August 2012, Sam Nassif, USAID’s Regional Engineering Team Leader for the Regional Development Mission for Asia (RDMA), with H. Mendsaihan (Program Management Specialist, USAID/Mongolia) and Nora Pinzon (acting USAID/Mongolia Representative), inspected the construction sites of the three school buildings. The USAID inspection team “identified a few areas where … some changes or adjustments might be made to improve the project delivery process, [but] in general…found that the quality of the work carried out, both in design and construction work, … superior to that carried out in many developing countries and often comparable to the work carried out in developed countries.” The inspection team “also concluded that the Program is well managed from both the USAID and implementing partner sides.”16 The retrofitting the three school buildings was completed in September 2012, and official building inspections by the Ulaanbaatar General Agency for Specialized Inspection (GASI) and the UB Education Department occurred in late September and early October. GASI’s official approvals of the retrofitted school buildings are dated 18 October 2012.17 All interior retrofitting work was completed by the end of August, and the schools reopened for the 2012-2013 school year on 3 September 2012. FINDINGS AND ANALYSIS Our findings are presented below in three parts: 1) Thermo-technical Retrofits to School Buildings; 2) Energy and GHG Reductions; and 3) Knowledge and Skill Building. THERMO-TECHNICAL RETROFITS TO SCHOOL BUILDINGS The evaluation team reviewed the architecture plans to assess conformance with the analysis assumptions and the terms of the grant agreement, and found that the plans were generally of excellent quality, and that both international and GOM standards were applied. The thermo-technical details shown on the plans were consistent with the inputs to the “Heat energy consumption” analysis performed by GIZ on each of the schools. To the extent that field visits could confirm correspondence between the architecture plans and proper installation in the buildings, correspondence was found. In general, the quality of workmanship in 15 Evaluation team interview of Dr. Ya.Duinkherjav (Professor and Chairman of Civil Engineering Department, Head of Concrete and Steel Structure Testing Laboratory, Mongolian University of Science & Technology), on 5 December 2012. Dr. Ya.Duinkherjav was part of the team that surveyed the three buildings prior to construction. 16 Trip Report: Ulaanbaatar School Buildings Thermo-Technical Retrofitting Project, August 14-16, 2012 (filename: Final Mongolia_RDMA-EngineerSamNassipTripReport.pdf) 17 Copies of GASI’s official building approvals, in the Mongolian language, were provided to the evaluation team. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 15 both the construction plans and the construction was of high quality. This is consistent with the findings of USAID’s August 2012 inspection.18 We met with B. Gantumur, Director of Tsats Suvarga International Co, Ltd. head of the architecture and engineering firm contracted to prepare the architectural design for the project and perform the “Author’s Control” or field inspections. He was proud of his firm’s contributions to the project, and was highly satisfied with the construction completed at the three buildings. In addition, conversations with the engineer responsible for designing the heating systems for the three buildings, and with the four project contractors, confirmed active engagement not only by the Mongolian partners but also by representatives of GIZ. The detailed engineer’s site visit reports are located in Appendix C of this report. The findings regarding retrofits at each school building are as follows: School No. 79-1 We visited the boiler room and noted that the building’s new HOB that was installed prior to this project is in use. The exterior of the building was covered with a cheerful orange and gold coating. The thickness of the wall overlap at the windows appeared to be consistent with 15 cm of insulation. The school was identified with lettering at the front entrance. The building has exterior downspouts. Prior to commencement of this project, the building was surrounded by a concrete perimeter that was cracked and uneven, which could have contributed to moisture build-up in the masonry walls. This was replaced with a graded gravel perimeter to facilitate drainage from the downspouts away from building. Inside the building, the overall temperature was comfortable but varied from room to room. Comfort levels and radiator temperatures were sampled with a FLUKE 61 infrared thermometer.19 Temperatures in the hallways and classrooms generally were adequate but several classrooms and one office were uncomfortably hot. One classroom was measured at 30° Celsius (86° Fahrenheit). Students appeared overheated in several classrooms. These rooms were clearly receiving too much heat given their size, ventilation, and the number of people occupying the rooms. Overheating of individual rooms may be due in part to an undersized or obstructed ventilation system, and/or overcrowding. The project design did not include retrofits to building ventilation systems, the building is overcrowded, and the ventilation system may have been designed for a smaller population. Also, the project did not include improvements to the hot water circulation system at School No. 79-1, and there were no valves on the radiators, so radiator heat cannot be controlled in individual rooms. Since there are no valves present on the radiators, the only viable means of controlling classroom temperature when class is in session is opening and closing windows. Interviews with teachers and others indicated that this is the cultural norm. Not only is this inefficient, it also reduces comfort levels due to hot spots near the radiators and cold spots near the windows. 18 Trip Report: Ulaanbaatar School Buildings Thermo-Technical Retrofitting Project, August 14-16, 2012 (filename: Final Mongolia_RDMA-EngineerSamNassipTripReport.pdf) 19 See Appensix C for accuracy information. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 16 All windows were new insulated windows. Several windows were open. The glass in one window was cracked, but we were informed by GIZ that all windows were intact when the building was inspected by GASI on 20 September 2012. A sports hall addition to this building was underway prior to commencement of this project. The sports hall is not insulated to the same standards as this project. With the radiators on and heating in the sports hall, the Fluke tester indicated a room temperature of 10° Celsius (50° Fahrenheit). Double interior doors separated the sports hall from the rest of the building. We were informed that the sports hall has not yet been approved by GASI for use. There was significant condensation present on a large number of the windows. There are a number of possible causes: (i) During construction a significant amount of rainwater entered the building during roof replacement and was absorbed by the masonry walls. Moreover, historically, the top of the masonry walls was not sealed to prevent rainwater infiltration. Also, as discussed above, prior to the retrofits, a concrete building perimeter may have directed rainwater from the downspouts toward the building. Now that the building interior is warmer and the building walls are insulated and sealed on the outside, it is likely that the moisture is being released from the masonry walls into the building. (ii) The school is over-crowded. Through the processes of respiration and perspiration, moisture for the human body is regularly released into the air. (iii) The ventilation system for the building is inadequate and/or obstructed. On a second visit to the site, the ventilation vents on the roof were inspected for proper functioning. A number of the vents were partially or completely blocked with what appeared to be frost at the discharge openings. Although this indicates that the ventilation conduits are clear (so warm moist air can pass through them), the blocked vent openings do not allow the vents to operate as designed. During the roof vent inspection, it was noted that rain guards were present over all but one of the roof vents. It appeared that the missing guard had been installed but removed. The caps for the masonry walls were present. The roof could not be inspected because of the snow present on the roof. Conversations with school personnel indicate that prior to the retrofits, the school students and teachers found it necessary to wear coats. During our visit, when the outside temperature was approximately -30° to -25° Celsius (-22° to -13° Fahrenheit), none of the students, teachers, or staff was wearing a coat. School No. 79-2 We visited the boiler room and noted that the building’s pre-retrofit boiler is in use. New boilers were present in the boiler building, but were not in service because they did not function properly when they were installed. Because of GHG considerations, the new boilers chosen for this project are capable of burning either biomass or coal. The project intended to use biomass in the new boilers, and initially fired them with wood logs. However, it was not possible to obtain optimally sized logs (i.e., large enough), which may have been a factor in malfunctioning of the boilers. Sufficient and affordable quantities of other forms of biomass, e.g. wood waste, were not readily available. The project also tried to use coal in the new boilers when they were installed. However, the type of coal consumed at the schools appears to be lignite rather than hard coal, which may have contributed to new boiler malfunction. Several solutions were offered by the manufacturer of the new boilers and have been implemented, but they were not successful. The school’s old boiler was reinstalled in September (the start of the heating season). The current hypothesis regarding the new boilers is that there is insufficient air to properly maintain combustion. The proposed solution is the installation of fans to increase air flow. This work is underway. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 17 According to GIZ, custom-designed and -built fans are in Ulaanbaatar awaiting installation. GIZ is following up on this and is committed to seeing it through. The exterior of the school building was covered with a cheerful orange and gold coating. The thickness of the wall overlap at the windows appeared to be consistent with 15 cm of insulation. The school was identified with lettering at the front entrance. The building has exterior downspouts. Prior to commencement of this project, the building was surrounded by a concrete perimeter that was cracked and uneven, which could have contributed to moisture build-up in the masonry walls. This was replaced with a graded gravel perimeter to facilitate drainage from the downspouts away from building. Inside the building, the temperature was generally comfortable but variable. Comfort levels and radiator temperatures were sampled with a FLUKE 61 infrared thermometer. In most cases temperatures seemed adequate and appropriate; however, we found some of the classrooms uncomfortably warm. The project included improvements to the hot water circulation system at School 79-2, including new valves on the radiators, so radiator heat can be controlled in individual rooms. Overheating may be due to too much radiator heat, overcrowding, and/or an inadequate or obstructed ventilation system. The project did not include retrofits to the building ventilation system. All windows were new insulated windows. Several windows were open to regulate room temperature. In response to questions about room temperature control, teachers indicated that they open or close the windows. Every radiator tested with the infrared thermometer was providing heat. New valves were present on each radiator. Conversations with school personnel indicate that prior to the retrofits, the students and teachers found it necessary to wear coats. During our visit, when the outside temperature was approximately -30° to -25° Celsius (-22° to -13° Fahrenheit), none of the students, teachers, or staff was wearing a coat. At the rear of the building there was an open trench extending to an adjacent brick building that is under construction. This new building will provide apartments for school teachers. Piping installed in the trench is to extend the hot water circulation system from School No. 79-2 to this new brick apartment building. According to GIZ, the school had been advised not to do this. School No. 63 We visited the boiler room and noted that the building’s pre-retrofit boiler is in use. The new boilers were present in the boiler building, but were not in service because when they were installed, they did not function properly. The school’s old boiler was reinstalled in September (the start of the heating season). (See the new boiler discussion for School No. 79-1 above for elaboration of problems with the new boilers and the proposed solution). An instrument for measuring net boiler heat output (out-going and in-coming water temperatures and flow rates) was mounted on the piping in the boiler building. The exterior of the school building was covered with a cheerful orange and gold coating. The thickness of the wall overlap at the windows appeared to be consistent with 15 cm of insulation. The front door appeared to be new. The walkway up to the front door was paved with precast concrete bricks. The school was identified with lettering at the front entrance. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 18 A new interior roof drainage system has been installed. Interior downspouts are recommended in cold climates because they are largely protected from freezing and mechanical damage, and are less susceptible to clogging and ice dams than exterior systems. Clogging and ice dams can result in water damage to the building. Inside the building, the temperature was generally comfortable but variable. Comfort levels and radiator temperatures were sampled with a FLUKE 61 infrared thermometer. In every case the temperature of the radiators indicated that they were heating. There were new valves on the radiators. We found some of the classrooms uncomfortably hot. The project included improvements to the hot water circulation system at School 63, including new valves on the radiators, so radiator heat can be controlled in individual rooms. Overheating may be due to too much radiator heat, overcrowding, and/or an inadequate or obstructed ventilation system. The project did not include retrofits to the building ventilation system. We examined the controls for the solar hot water heating system. Temperatures on the controls indicated that the water was sufficiently hot for bathroom and kitchen use. Solar panels were present on the roof. The sports hall was in use. Temperatures were comfortable. The windows on the north had been reduced in size; windows on the south had been installed where there had been no windows before. The new bathrooms and showers were present and appeared to be functioning. There were sinks in the classrooms. Prior to this project, pit latrines were built at School No. 63 because the interior toilets could not be used during the winter months. (The old septic lines were too shallow and froze in the winter. This problem was solved by the project’s installation of an entirely now septic system). One latrine remains at the site. It is unattractive and a potential hazard. We were informed by GIZ that the Director of School No. 63 has promised to remove the pit latrine once the ground has thawed after the winter. Removal of the latrine is outside of GIZ’s control. A new well has been installed, but the water quality test report for the new well had not been received by the time the evaluation team left UB. Prior to this project, the water in the old well had been tested and found to have bacteriological contamination, so it had been recommended that water from the old well be boiled before drinking. In early January 2013, the evaluation team received the UB Water Authority’s water quality test report for the well water samples taken 13 December 2012. The water met the Mongolian standard with regard to bacteriology, but did not meet Mongolian standards for color, smell, turbidity, and metals, so a second test is required. We were informed by GIZ that the second test will take place between 7-8 January 2013, and the results will be reported to USAID/Mongolia. All windows were new insulated windows. There was condensation on the window sill of several windows. Several windows were open. In response to questions about room temperature control, teachers indicated that they open or close the windows. Conversations with school personnel indicate that prior to the retrofits, the school students and teachers found it necessary to wear coats. During our visit, when the outside temperature was approximately -30° to -25° Celsius (-22° to -13° Fahrenheit), none of the students, teachers, or staff was wearing a coat. The project budget did not include funds for interior redecorating. The teachers and some parents were so enthusiastic about the reincarnation of the building that they repainted the facility themselves. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 19 ENERGY AND GHG REDUCTIONS The project was not designed to allow for the collection of actual pre-retrofit heat demand and coal consumption data at the three schools, and the retrofitted buildings have only been in operation for a little over three months, so not enough time has elapsed for collection of a year of post-retrofit data. Therefore, GIZ has estimated annual pre- and post-retrofit heat demand based on a building energy model,20 fuel consumption based on estimated heat demand and fuel calorific values and boiler efficiencies, and pre- and post-retrofit CO2 emissions with a carbon accounting model for buildings.21 The fuel costs were calculated from the fuel consumption estimates and average market rates for fuel in UB. The pre- and post-retrofit heat demand, fuel consumption, fuel cost, and CO2 emission estimates derived by GIZ are contained in tables in GIZ’s November 2012 “Heat Energy Consumption” reports.22 During the construction phase of the project, a heat meter was installed at one of the three school buildings (School No. 63).23 This meter will enable actual boiler heat output to be tracked at School 63, from which post-retrofit fuel consumption and GHG emissions could be recalculated. The readings from this meter would also facilitate future assessment of the accuracy of the heat demand modeling. The project did not include such post-retrofit monitoring of heat output at School No. 63, or re-assessment of the heat demand modeling and fuel consumption and GHG estimates based on observed heat demand. The project did not include installation of similar meters at School Nos. 79-1 and 79-2, which would have allowed for more thorough assessment of the accuracy of the heat demand modeling, and recalculation of fuel consumption and GHG emissions at those buildings. The heating system operations at School No. 79 are outsourced to a state owned company under the UB City Government, which means that installation of heat meters at School Nos. 79-1 and 79-2 would require complex and time-consuming negotiations among the state owned company, the UB City Government, and the school director. The director of School No. 63 retains control of heating system operations at his building. The new boilers that were purchased for School Nos. 79-2 and 63 are designed to burn either biomass or coal. GIZ derived two sets of post-retrofit fuel consumption and GHG emission estimates for these two school buildings. One set assumes coal is used throughout the year, but the old boiler operates from September to mid-January and the new boilers operate from mid-January through the end of the heating season. The other set assumes that the new boilers operate throughout the year and are fired with wood. The evaluation team was informed by GIZ that the price of coal in UB does not vary significantly in UB, even if the quality varies. No other information on the variability of coal quality and cost in UB was available to the evaluation team. Several issues arose during our review of GIZ’s calculations of heat and fuel demand, and GHG emissions: The new boilers and use of wood as a fuel 20 The evaluation team’s interview of Mr. Tuvshinkhuu indicated that GIZ based their estimates on a German building energy model (http://www.zub-kassel.de/software/produkte/helena). The model uses inputs on weather, building characteristics (e.g., building dimensions, construction materials, window characteristics), and standard indoor temperature, and produces outputs of ventilation and transmission heat loss. 21 The carbon accounting model is a German spreadsheet model that was designed for individuals in the Oberharz region of Germany to calculate their CO2 emissions from heating fuels and electricity used in residential and small commercial buildings. 22 Tuvshinkhuu, S., 30 Nov 2012, Heat energy consumption School No. 79-1; Tuvshinkhuu, S., 30 Nov 2012, Heat energy consumption School No. 79-2; Tuvshinkhuu, S., 22 Nov 2012, Heat energy consumption School No. 63. 23 This meter measures outgoing and incoming temperatures of the circulating water, and flow rates, from which heat demand per unit of time is estimated. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 20 Although GIZ is working to repair and re-install the new boilers at School Nos. 79-2 and 63, this had not yet been accomplished by the end of the evaluation team’s time in Mongolia. Information collected during the interviews of key informants indicates that it is highly unlikely that wood or other forms of biomass will be used in the new boilers once they are operational. GIZ reports that the market rate for wood is double that of coal (on an energy basis),24 and that there is not a readily available supply of wood in the size needed to properly fuel the boilers. Other informants indicated that the costs, availability, and transport distances for wood and other forms of biomass in UB preclude viable biomass use for these boilers, especially given the volumes of biomass that would be needed for the boilers at each of the three buildings. Calorific value of coal GIZ assumed a calorific value for coal of 7000 kilocalories/kilogram (7000 kcal/kg). This value is representative of hard coal. However, international energy statistics indicate that all of the coal consumed in the commercial (building) sector in Mongolia is lignite (brown coal),25 which has a calorific value of 4165 kcal/kg or lower. GHG accounting GIZ estimated pre- and post-retrofit CO2 emissions using a “carbon footprint” calculator for residential and small commercial buildings in a region of Germany. The calculator includes two components: 1) direct CO2 emissions from fuel combustion; and 2) indirect CO2 emissions from energy used to extract and transport fuel. There are several weaknesses in these estimates. First, the emission factor used for the combustion component was for hard coal rather than lignite, and does not include GHGs other than CO2. Second, extraction and transport assumptions in the calculator for the second component (fuel extraction conditions, distance fuel is transported, and mode of transport) are unlikely to be valid for coal (or biomass) used in schools in UB. And third, these calculations do not include fugitive methane (CH4) emissions from the mining and handling of coal.26 We have re-estimated the fuel consumption and GHG emission estimates using the following data inputs and assumptions:  Monthly heat demand estimates derived by GIZ for each school building  The use of the old boilers in all three buildings for the entire post-retrofit year  The use of coal (lignite) as the post-retrofit fuel  A coal (lignite) calorific value of 4165 kcal/kg27  GIZ’s estimated coal price of 85,000 Mongolian tugriks per metric ton (85,000 MNT/t) 24 See Tuvshinkhuu, S., 30 Nov 2012, Heat energy consumption School No. 79-2; and Tuvshinkhuu, S. 22 Nov 2012, Heat energy consumption School No. 63. 25 International Energy Agency, 2009 Energy Balance for Mongolia, http://www.iea.org/stats/balancetable.asp?COUNTRY CODE=MN 26 Fugitive emissions from coal are the intentional or unintentional releases of GHGs that may occur during extraction, processing, and delivery. They do not include emissions from the combustion of fuels used to transport coal. 27 This is a high value for lignite. The actual value may be lower. If the actual calorific value of the lignite were lower, estimated absolute reductions in fuel consumption, fuel cost, and GHG emissions would increase proportionately. This would not affect the estimated percentage reductions in fuel consumption, fuel cost, or GHG emissions. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 21  IPCC default emission factors for direct CO2, CH4, and N2O emissions from stationary combustion of lignite in the commercial/institutional sector, and for indirect CH4 emissions from surface mining and post-mining operations.28 The results of this re-analysis are presented in Table 1. These estimates are for the period September 2012 through August 2013, relative to post-retrofit conditions. Table 1: Estimated Annual Reductions in Heat Demand, Coal Demand, Fuel Costs, and GHG Emissions School No. 79-1 School No. 79-2 School No. 63 Total1 Reduction in heat demand (Gcal/yr) 571 340 174 1086 Reduction in coal demand (t/yr) 274 204 105 583 Fuel cost savings (US$/yr) $16,771 $12,489 $6,403 $35,662 GHG emissions avoided (t CO2-eq/yr) 478 356 183 834 Percentage change (%) (all values)2 57 43 25 1 Numbers may not sum precisely due to rounding 2 These percentages are for all values above (reduction in heat demand, reduction in coal demand, fuel cost savings, and GHG emissions avoided). The estimated annual reductions in heat demand due to thermal retrofits to the school buildings range from 174 to 571 gigacalories (Gcal) per year, or 25 to 57 percent. The significantly smaller percentage change for School No. 63 than for School Nos. 79-1 and 79-2 is due primarily to the almost 40 percent increase in heated volume at School No. 63 from the sports hall rehabilitation. Also, the modeled reduction in transmission heat loss for School No. 63 was lower than for School Nos. 79-1 and 79-2. The difference in the percent reductions between the other two school buildings is due to differences in modeled ventilation and transmission heat losses.29 The average percentage reduction is approximately 40 percent. There are a number of sources of uncertainty in the modeled heat demand estimates. The estimated reductions in heat demand may be too high for two reasons: 1) the model does not appear to include heat loss due to infiltration (e.g., heat loss due to opening windows and doors), which is likely to have increased since the retrofits were completed; and 2) inside air temperatures measured by the evaluation team were variable, and in some rooms, significantly higher than the modeled standard inside post-retrofit temperature of 20° Celsius. Sources of uncertainty cannot be elaborated further without more extensive review and evaluation of the model and input assumptions. If the new sports hall at School No. 79-1, which has not yet been approved for use but is currently heated by the building’s heating system, is included in the calculations, the estimated reduction in heat loss at School No. 79-1 would be smaller. If the hot water circulation system for School No. 79-2 is connected to the new apartment building and this additional building volume was included in the calculations, the estimated 28 IPCC, 2006, 2006 IPCC Guidelines for National Greenhouse Gas Inventories, Volume 2, Energy, http://www.ipcc￾nggip.iges.or.jp/public/2006gl/vol2.html 29 The spreadsheets that contain key data inputs and assumptions in the energy modeling are in the Mongolian language. More detailed explanations of the differences among the three buildings would require significantly more analysis. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 22 reduction in heat loss at School No. 79-2 would be smaller. However, these are factors outside of the project’s control. The estimated absolute reductions in coal demand, fuel costs, and coal-related GHG emissions at each building (Table 1) are proportional to the absolute heat reductions because our estimates assume that the pre￾retrofit boilers and fuel type are used throughout the post-retrofit year. This assumption was made because the new boilers in School Nos. 79-2 and 63 are not yet operational. Therefore, the percentage reductions in coal demand, fuel costs, and coal-related GHG emissions are equal to those of the heat reductions (25 to 57 percent). If the new boilers at School Nos. 79-2 and 63, which have an efficiency rating with coal that is a little over twice that of the old boilers, are operational from January 2013 to May 2013 and are fired with coal during that time, the estimated annual reductions in coal demand, fuel costs, and GHG reductions would increase by about half (i.e., to 60 percent for School No. 79-2 and to 50 percent for School No. 63). (Note: Estimated heat demand is not affected). However, the actual efficiencies achieved by the new boilers may be less due to boiler operation conditions, fuel characteristics, and the condition of the hot water circulation systems. The primary sources of uncertainty in the estimated absolute reductions in coal demand, fuel costs, and coal￾related GHG emissions are: 1) uncertainties in the heat demand estimates; 2) the difference between the estimated and actual calorific value of the coal; 3) the difference between the manufacturer-rated and actual boiler efficiencies; and 4) uncertainties in the GHG emission factors. These emission factors are internationally-accepted and representative of the fuel type and combustion conditions at the schools, but actual emission factors may vary due to specific fuel characteristics and boiler conditions. In addition, the estimated absolute reductions in GHG emissions do not include avoided indirect GHG emissions from fuels used to mine and transport coal to the schools. The answers to the first three of the five questions the evaluation team was directed to answer are presented in Table 2. Table 2: The First Three Evaluation Questions: Estimated Reductions in Heat Loss and GHG Emissions Due to Retrofits School No. 79-1 School No. 79-2 School No. 63 Average Percent decrease in heat loss/sq meter 57 43 38 46 t CO2-eq reduced per Gcal of heat demand saved 0.84 1.0 1.0 0.94 Percent decrease in GHG emissions 57 43 25 42 The estimated percent decrease in heat loss per square meter for School No. 63 is lower than the percentage in Table 1 because the post-retrofit floor area is larger than the pre-retrofit area (see discussion above). Uncertainties in these values are due to uncertainties in the energy modeling, described above. The mass of GHG emissions (metric tons [t]) reduced per Gcal of heat demand saved is higher for School Nos. 79-2 and 63 than for School No. 79-1 because the boilers in School Nos. 79-2 and 63 have a lower efficiency rating than the boiler in School No. 79-1 (40 percent versus 50 percent). Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 23 As discussed above, the percentage decreases in GHG emissions are solely a result of improvements to the buildings. The primary sources of uncertainty in the estimates of GHG emissions reduced per Gcal of heat demand saved, and in the estimates of the percentage decrease in GHG emissions, are uncertainties in the heat demand estimates, the coal calorific value, the actual boiler efficiencies, and the GHG emission factors. KNOWLEDGE, SKILL BUILDING, AND GENDER IMPLICATIONS The evaluation team was asked to answer two questions regarding knowledge and skill building (Questions 4 and 5 in the list presented above in the section Purpose of the Evaluation). Also, the SOW directed that evaluation data be disaggregated by gender as appropriate. Question 4: How much local capacity was built to design and implement retrofitting technology as a result of the project? We interviewed management staff and engineers at GIZ, and Mongolian engineering, architectural, and construction partners, and found that the project resulted in an increase in knowledge and skills in retrofitting technologies, particularly among the construction companies. This resulted from on-the-job training that GIZ provided to contractors during construction, as well as effective supervision. Construction workers were trained on-site in proper techniques for installation of retrofit technologies, especially exterior insulation, high-efficiency windows, sub-floors, roofs, and solar panels. Of particular note was learning the proper techniques for shaping, applying glue to, and installing sheets of exterior insulation so the seams are tight and heat loss is minimized. Several contractors commented that they had not had prior experience with such high quality materials or with the specific installation techniques used for this project. One contractor commented that the subfloor materials and construction techniques were new; another contractor noted that the solar panel installation was a new process for them; another contractor noted that although they had experience with high-efficiency insulation in new construction, this project was their first experience in thermal retrofits to existing buildings. Several contractors and engineers stated that the project results exceeded their expectations. Throughout our interviews, informants remarked on the quality and thoroughness of the construction site supervision as well as the effective overall management of the project. A site supervisor was on-site at all times during construction at the three sites, and a GIZ supervisor visited each site at least once a week and more frequently during busy construction phases. Several individuals commented that this extent of supervision is unusual in Mongolia, and that it ensured that their employees did their jobs properly and that construction procedures adhered to plans. Many contractors and engineers noted how quickly construction problems were addressed and improper installations, such as window installations, were fixed. Several individuals also noted how effective overall project management ensured that the project progressed properly and that partners were well coordinated. It is clear that the contractors, engineers, and architects learned about effective construction site supervision and project management from participating in this project. While it appears likely that contractors, engineers, and architects will be able to use their new knowledge and skills in the future, to what extent and how soon this happens is uncertain. It is clear from our interviews that the UB government is paying greater attention to building energy efficiency in both new construction and existing buildings, especially since heating demand in UB is expected to exceed installed capacity by 2014, and shortages are already anticipated for this winter. However, cost is a significant hurdle to retrofits of public buildings, especially those that are not connected to the district heating system since coal is readily available. We received mixed responses from the construction companies when asked about the likelihood of more thermal retrofitting jobs in the near future. One construction company stated that they had already been Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 24 contacted by the UB government about a similar project that will be tendered next year. They plan to bid on the project, and expect to be well positioned among the competition given their experience with this project. Two construction companies said that they were not aware of any similar opportunities. However, all the construction companies noted that their experience in this project has provided them with valuable skills and experience that will give them a competitive advantage for future work of this kind. It is not clear what portion of the construction workers will be able to use the knowledge and skills acquired from this project because many were temporary employees who may not return to construction. Also, construction workers were not interviewed. Because the construction season is so short in Mongolia, many construction workers are temporary employees (often students who need summer jobs) who disappear at the end of the construction season. During peak construction of this project, one third to three quarters of the construction workers were temporary. However, even if these workers do not use new retrofit installation skills again, their awareness of building energy efficiency is likely to have increased as a result of the project. Also, it is clear that the demand for skilled building energy engineers and architects who can advise on energy efficient design, construction, and retrofitting is growing in UB, and that this project has contributed to the development of local expertise in this area and to the demand for this expertise, given the visibility and success of this project. Question 5: Did the level of knowledge about energy efficiency increase among school staff as a result of the project? This project has demonstrated to the directors, teachers, and other staff of School Nos. 79 and 63 how thermal-technical retrofitting can transform an exceedingly cold, drafty, and leaky building into a warm and comfortable environment, and has thereby increased their awareness and understanding of building energy efficiency. Both school directors informed us that the transformation of their school buildings has exceeded their expectations as well the expectations of their staff, and expect learning to improve as a result of the project. Everyone is pleased with the outcome, and school pride has increased at both schools. Both directors anticipate significant coal savings this year as a result of the retrofits; one director commented that he had already noticed significant reductions in fuel use. The director of School No. 63 remarked on his school’s efforts to be an environmentally-friendly school and the high level of awareness of energy efficiency among staff and students. He also said that he is committed to ensuring that the retrofitted building operates at its design efficiency. Also, awareness of this project and its benefits has spread beyond the school staff since the director has already received visits from other school directors, as well as private business owners, who want to learn more about the retrofits and find out how they “can get this done” at their building. A specific output of this project is a learning module on energy and climate change that GIZ prepared for secondary school teachers and students. This module is a booklet in the Mongolian language that contains a chapter each on energy, climate change, solar energy, wind power, hydroelectricity, and energy conservation and efficiency. At the end of each chapter are simple experiments to demonstrate key concepts and facilitate learning, and a set of questions to test knowledge. The booklets were distributed to the retrofitted schools in late November 2012. Both school directors have distributed the booklets to their secondary school teachers, and asked the teachers to develop suggestions for incorporating the material into the curriculum. It is too early to determine how the learning module will be used and whether it increases learning for teachers or students. Although not directly relevant to the Question 5, we want to note that this project has also contributed to the awareness among government entities of the energy efficiency benefits of building retrofit technologies. We interviewed representatives of the Ministry of Construction and Urban Development, Ministry of Education Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 25 and Science, and the Ulaanbaatar Education Department. The project has effectively demonstrated to government entities what can be accomplished through retrofitting, and raised the visibility of the potential benefits of end-use energy efficiency measures. The results of this project should also contribute to ongoing efforts in the national government to develop new energy efficiency standards and codes for the construction sector, including retrofits to existing buildings. Gender Implications The primary purpose of this evaluation is to determine whether, and to what degree, the project was successful in saving heat energy, reducing energy costs, and reducing GHG emissions. While gender is not relevant to the data needed to evaluate these topics, the evaluation team did question GIZ and the four Mongolian construction companies about gender considerations in implementing the project. Both GIZ and the contractors noted that explicit efforts were made to hire female construction workers for the more detailed work (e.g., installation of insulation on facades, painting, laying of ceramic tile, installation of window sills) because in general, compared to men, their work was higher quality, they followed directions more carefully, and they were more reliable employees. The percentage of the permanent construction workers who were women varied among contractors. These percentages were zero percent, 11 percent, 30 percent, and 50 percent. The percentage of temporary construction workers who were women ranged from 30 to 60 percent. Also, of the four Mongolian construction companies engaged in this project, two are owned and directed by women, and the site manager for one of the construction companies was a woman. CONCLUSIONS THERMO-TECHNICAL RETROFITS TO SCHOOL BUILDINGS 1. The thermal-technical retrofits to School Nos. 79-1, 79-2, and 63 have transformed exceedingly cold, drafty, and leaky buildings into warm and comfortable environments conducive to learning. The resultant comfort levels in the school buildings have exceeded the expectations of the school staff and students, as well as some project engineers and contractors. School staff and students are pleased with the new environment, more proud of their schools, and expect learning to improve. Two unplanned benefits of this project are that the teachers and some parents at School No. 63 were so enthusiastic about the reincarnation of the building that they repainted the interior of the facility themselves, and people who live near School No. 63 are using the building as a type of informal community center for sports and other activities during non-school hours. 2. The performance of the buildings is consistent with the proper installation of the new thermal envelope. To the extent that field visits could confirm correspondence between the architecture plans and proper installation in the buildings, correspondence was found. In general, the quality of workmanship in both the construction plans and the construction was of high quality. Project partner architects, engineers, and contractors noted the high quality of the project materials, the installation procedures, and the construction supervision. 3. Room temperatures to School Nos. 79-1, 79-2, and 63 are sufficiently warm, but are highly variable. Some classrooms are uncomfortably hot. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 26 Overall, the three buildings were comfortably warm, and all radiators were operating. However, the evaluation team found several individual classrooms in all three buildings uncomfortably hot, and students in those rooms appeared uncomfortable. One classroom in School No. 79-1 was measured at 30° Celsius (86° Fahrenheit). 4. The project included heating system upgrades at School Nos. 79-2 and 63, but not at School No. 79-1, so the thermal￾technical retrofits included installation of radiator valves at School Nos. 79-2 and 63, but not at School No. 79-1. This leaves window operation as the only available method of room temperature regulation at School No. 79-1 when classes are in session. The project did not include instructing teaching staff at School Nos. 79-2 and 63 on the use of valves to moderate classroom temperatures, and the cultural norm is to open and close windows to regulate room temperatures. Several windows were open at all three buildings when site visits were made. In response to questions about room temperature control, teachers indicated that they open or close the windows. 5. There was a high amount of condensation on the windows at School No. 79-1. Less significant condensation was observed at School No. 63. The condensation could be due to moisture release from the masonry walls, over-crowding, an inadequate ventilation system, and/or obstructed ventilation system. Given rainwater infiltration prior to and during retrofit construction, and the fact that moisture cannot escape through the exterior of the masonry walls now that they are insulated and sealed, moisture release from the masonry walls is believed to be a significant contributor to interior condensation. This condensation should lesson as the winter heating season proceeds. However, the ventilation systems, which are original, may be undersized for the number of people in the buildings, so overcrowding may continue to cause excess condensation during winter months. 6. A number of ventilation vents on the roof of School No. 79-1 are partially or fully obstructed by frost, and one vent rain guard is missing. The ventilation vents on the roof of School No. 79-1 were found to be partially or fully obstructed by frost. This indicates that the ventilation system has been working (warm, moist air is rising through the conduits and exiting at the roof), but air escape is now partially or fully blocked. In addition, one of the vent rain guards was missing, which allows rainwater infiltration at that vent. Clearing the vent openings and replacing missing vent guards may reduce condensation levels, and alleviate overheating of rooms. 7. The glass in one window at School No. 79-1 is cracked, but was intact when retrofitted building was inspected by the UB City government. The glass in one window at School No. 79-1 is cracked, but GIZ confirmed that all windows were intact when the building was inspected by GASI on 20 September 2012, so the break occurred after the retrofits were completed and the work was officially inspected. The broken window negatively affects building energy efficiency. 8. The new boilers for School Nos. 79-2 and 63 did not operate properly when they were installed. The old boilers were re￾installed and are in operation. The project included improvements to the heating systems at School Nos. 79-2 and 63, including two new boilers at each building, and cleaning and replacement (as needed) of the hot water circulation systems and radiators. The new boilers at School Nos. 79-2 and 63 did not operate properly when they were installed, and GIZ implemented several manufacturer-suggested solutions, but they were not successful. The old boilers were re-installed at both buildings. The current hypothesis regarding the new boilers is that there is Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 27 insufficient air to properly maintain combustion. GIZ is working to retrofit the new boilers with custom￾designed and -built fans. 9. The instrument for measuring net boiler heat output that was installed at School No. 63 can facilitate the efficient operation of the heating system. The project installed an instrument in the boiler room at School No. 63 that measures net boiler heat output. Monitoring this heat meter, in conjunction with monitoring interior temperatures and moisture conditions at School No. 63, can facilitate the efficient operation of the heating system to maximize efficiency gains and building comfort levels. The project did not include such post-retrofit monitoring of heat output and building conditions at School No. 63. 10. An un-insulated brick building to be used for teacher housing is being connected to the heating system of School No. 79-2. GIZ has advised School No. 79 to not make the connection because this will negatively affect the proper and efficient functioning of the heating system. 11. A pit latrine remains at School No. 63, but it is expected to be removed once winter is over. The pit latrine that remains at School No. 63 is unattractive and a potential hazard, but it is outside of GIZ’s control. The Director of School No. 63 has promised to remove the pit latrine once the ground has thawed after the winter. 12. The quality of the water from the new well at School No. 63 has been tested, and met the Mongolian standard with regard to bacteriology, but did not meet Mongolian standards for color, smell, turbidity, and metals. A second test will take place between 7-8 January 2013. The results of the second test will be reported to USAID. ENERGY AND GHG REDUCTIONS 13. The project did not allow for the collection of actual pre-retrofit heat demand and coal consumption data, and not enough time has elapsed for collection of comprehensive post-retrofit data, so energy and GHG values for both the baseline (pre￾retrofit conditions) and the post-retrofit conditions are estimates derived using models. Heat demand estimates are based on a building energy model and characteristics of the thermal improvements to the building envelopes. Fuel consumption estimates are based on estimated heat demand and fuel calorific values and boiler efficiency ratings. GHG emission estimates are based on a GHG accounting model. 14. It is estimated that the thermal-technical retrofits to School Nos. 79-1, 79-2, and 63 will reduce annual heat loss by an average of approximately 40 percent per year. Estimated heat demand reductions at each building varied from 25 to 57 percent. Variability in the model results is due an increase in heated volume at building (School No. 63) that did not occur at the other two buildings, and differences in modeled ventilation and transmission heat losses. The reductions in heat demand may be overestimates due to higher actual post-retrofit values for infiltration heat loss and inside air temperatures than modeled values. 15. With the old boilers still in operation with coal as the fuel, annual coal consumption, coal costs, and fuel-related GHG emissions will also decline by an average of 40 percent per year. Total annual savings due to reduced coal consumption at all three buildings are estimated at about US$35,600. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 28 The new boilers in School Nos. 79-2 and 63 are not yet operational; therefore, the estimated absolute reductions in coal consumption, fuel costs, and coal-related GHG emissions at each building are proportional to the absolute heat reductions. Sources of uncertainty in the absolute estimates stem from uncertainties in heat demand, coal calorific content, boiler efficiencies, and GHG emission factors. 16. If the new boilers are installed successfully, and operate with coal from January 2013 through the end of the heating season, the estimated annual percentage reductions in coal demand, fuel costs, and GHG reductions at School Nos. 79-2 and 63 for this school year would be greater by about half. The new boilers have an efficiency rating with coal that is a little over twice that of the old boilers, and the heating season is about half over. The new boilers can fire with either biomass or coal, but the high cost, limited supply, and long transport distances for biofuels in UB precludes viable use in the boilers. The actual efficiencies achieved by the new boilers may be less due to boiler operation conditions, fuel characteristics, and the condition of the hot water circulation systems. 17. It is estimated that the thermal-technical retrofits to School Nos. 79-1, 79-2, and 63 will reduce annual heat loss per square meter by an average of approximately 46 percent per year. Estimated heat loss per square meter at each building varied from 38 to 57 percent. Actual reductions may be lower due to actual building operation that varies from modeled operation. 18. It is estimated that the thermal-technical retrofits to School Nos. 79-1, 79-2, and 63 will reduce GHG emissions per unit of heat demand by an average of approximately 0.94 tons CO2-equivalent per gigacalorie of heat demand. Estimated GHG reductions per unit of heat demand at each building varied from 0.84 to 1.0 tons CO2- equivalent per gigacalorie. 19. The heat meter was installed at School No. 63 will enable actual boiler heat output to be tracked, from which the accuracy of the hear demand modeling could be assessed and post-retrofit fuel consumption and GHG emissions at School No. 63 could be recalculated; however, the project did not include such post-retrofit monitoring. The project did not include such post-retrofit monitoring of heat output at School No. 63, or re-analysis of the heat demand modeling and fuel consumption and GHG estimates. Installation of similar instruments at School Nos. 79-1 and 79-2 would have enabled more robust evaluation of the energy modeling, and more accurate calculations of post-retrofit fuel consumption and GHG emissions, if the project had also included post-retrofit monitoring and evaluation of heat demand. However, installation of heat meters at School No. 79 would have required complex and time-consuming negotiations. KNOWLEDGE, SKILL BUILDING, AND GENDER IMPLICATIONS 20. This project, through on-the-job training and effective project and construction site management, increased knowledge and skills in retrofitting technology materials and installation, and in project and construction site management, among the local engineering, architectural, and construction partners. On-site training in installation of exterior insulation, high-efficiency windows, insulated sub-floors and roofs, and solar panels, and experience with higher quality materials and construction than is typical, all contributed to local learning. For some partners, this was their first experience with retrofit construction. The quality and thoroughness of the construction site supervision as well as the effective overall project management provided informative models for partners, and ensured that construction procedures adhered to plans. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 29 21. The extent to which, and how soon, contractors, engineers, and architects will be able to use their new knowledge and skills is uncertain, but demand for these skills and knowledge appears to be growing. Many construction workers are temporary employees; those employed by this project may not return to construction work. However, one construction company has already been contacted about a similar project that will open for bids next year, and all the construction companies noted that this project has provided them with skills and experience that will give them a competitive advantage for future work of this kind. Also, demand for engineers and architects who are proficient in building energy efficiency is growing in UB. 22. This project has increased the awareness and understanding of building energy efficiency among schools staff, and students by demonstrating how thermal-technical retrofitting can transform comfort levels in a building and reduce fuel consumption. The results of this project have exceeded expectations, and school staff expect learning to improve as a result of the project. Awareness of the benefits of thermal retrofits has already spread beyond the schools to local communities and other schools. This project has also contributed to the awareness among UB government and GOM entities of the energy efficiency benefits of building retrofit technologies. 23. The learning module on energy and climate change has been distributed to secondary school teachers at the retrofitted schools, but it is too soon to evaluate its effects on knowledge about energy efficiency. School directors received the booklets in late November 2012, and have asked their teachers to develop suggestions for incorporating the material into the curriculum. 24. The four Mongolian construction companies engaged in this project made an explicit effort to hire female construction workers for the more detailed work because in general, compared to men, their work was higher quality, they followed directions more carefully, and they were more reliable employees. Two of the four companies are owned and directed by women, and the site manager for one of the construction companies was a woman. The percentage of permanent construction workers who were women varied from zero 50 percent. The percentage of temporary construction workers who were women ranged from 30 to 60 percent. RECOMMENDATIONS AND LESSONS LEARNED RECOMMENDATIONS FOR THE PROJECT The evaluation team recommends that the project undertake the following actions: 1. At School Nos. 79-2 & 63: Repair and reinstall the new boilers. If this is unsuccessful, replace the new boilers with new coal-fired boilers with similar or higher efficiency ratings and that are appropriate for the quality of coal available in UB. Monitor the new boilers throughout the 2012-2013 winter months to ensure continued operation. 2. At School Nos. 79-1, 79-2, and 63: Inspect all roofs to determine whether all vent openings are clear and all vent rain guards are in place. Clear obstructed vent openings of frost, and replace all missing rain guards. Instruct building operations staff to check and clear roof vents regularly throughout the winter. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 30 3. At School No. 63: Review the results of the second well water quality test. If the results still do not meet Mongolian standards, find and eliminate the source of contamination and/or implement treatment measures as needed. 4. At School No. 63: Verify that the pit latrine is locked so that it cannot be used. Once the ground has thawed at the end of the winter, verify that the latrine has been removed. If the latrine is not removed, remind the school director that it should be removed. LESSONS LEARNED AND RECOMMENDATIONS FOR FUTURE PROGRAMMING There are a number of lessons learned from this project that can be taken into consideration when designing projects of a similar scope in the future. These lessons learned suggest two types of recommendations for future programming: recommendations to improve operations of retrofitted buildings; and recommendations to improve learning from such projects. Building Operations The evaluation team observed uncomfortably high temperatures in several individual classrooms at all three buildings, open windows at all three buildings, and significant condensation on the windows in School No. 79-1. Condensation is expected to lesson as the masonry walls dry out over the winter months, and control of room temperatures may improve as heating system operators and end-users (teaching and other school staff) become more familiar with the retrofits. However, the cultural norm is to open windows when room temperatures are too high, which negates some of the efficiency gains, and the project did not include the provision of radiator valves at School No. 79-1, so windows provide the only means of controlling individual room temperatures at School No. 79-1 when classes are in session. The heat meter that was installed in the boiler room at School No. 63 can facilitate the efficient operation of that school’s heating system by tracking heat demand over time, but the project did not include working with the heating system operators to monitor post-retrofit conditions and to design and implement optimal heat management procedures. Given these observations, the evaluation team recommends that future programming of this type include:  Cleaning and any necessary replacement of hot water piping system and radiators, and installation of radiator valves, at all buildings to ensure that all hot water systems operate efficiently and that end￾users are able to control radiator heat. This clearly has cost implications but would improve comfort levels and increase energy efficiency gains.  Installation of a heat meter at all buildings, monitoring of net boiler heat output and temperature and moisture conditions in the retrofitted buildings for a year following construction, collaboration with the heating system operators to design and implement optimal heat management procedures, and training of teaching staff and other end-users in optimal end-user heat management procedures (managing radiator valves, windows, doors, classroom vent openings). This has both cost and project lifetime implications, but would help ensure that improvements in comfort and energy efficiency are maximized. Also, the glass in one window at School No. 79-1 was broken sometime between GASI inspection on 20 September 2012 and the evaluation team’s visit on 5 December 2012. It is not clear how this happened, but the fact that a window was broken so quickly suggests that this may not be a rare occurrence. While not the responsibility of the project, the broken glass is a source of heat leakage. The evaluation team recommends that for projects of this type, building maintenance staff be trained in window repair, and the availability of Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 31 replacement units be verified or an appropriate number of replacement units be provided to the building managers at project end to ensure that repairs can be made in a timely fashion. Project Learning The estimated energy and GHG benefits of the project are uncertain due to reliance on modeled heat demand estimates and uncertainties about coal characteristics and combustion conditions. These uncertainties can be reduced by collecting pre- and post-retrofit coal consumption data, obtaining information about coal characteristics and costs, and by monitoring heat demand with instrumentation that measures net boiler heat output (e.g., the heat meter that the project installed in the boiler room at School No. 63). To increase the accuracy of estimated energy and GHG benefits of projects of this kind, the evaluation team recommends that future similar programming include:  Collection of monthly coal consumption data for the buildings prior to the retrofits and after the retrofits, and collection of information on coal characteristics (calorific values, moisture contents), sources (mines and mine characteristics), transport conditions (distance, mode), and prices (variability by quality and time of purchase). This data collection has both cost and project lifetime implications, but would improve the accuracy of the fuel and GHG estimates.  Use of heat meters to monitor net heat output (heat demand) at buildings over the entire heating season after the retrofits, and ideally prior to retrofits over an entire heating season as well. If a meter is not present at all buildings and/or pre-retrofit monitoring is not possible, the results of monitoring should be used to calibrate the model used to estimate heat demand at the unmonitored buildings. Monitored heat demand values in combination with data on coal consumption data and calorific values would enable actual boiler efficiencies to be derived, and compared against rated efficiencies. This data collection has both cost and project lifetime implications, but would improve the accuracy of the heat demand, fuel demand, and GHG estimates. Although this project raised awareness of the building-comfort and energy benefits of thermal-technical retrofitting among school staff, the communities surrounding the schools, and UB and national government entities, there are additional actions that could be taken to increase awareness of the benefits of such projects. The evaluation team recommends that future similar programming include:  Development of briefing materials (brochures and presentations) that contain quantitative information about the costs and energy and GHG benefits of thermal-technical retrofitting of buildings. These materials should incorporate quantitative information obtained through implementing the previous two recommendations. One of the outputs of this project was a learning module on energy and climate change targeted to secondary school students and teachers. The module has been distributed to the retrofitted schools but the project did not include evaluation of whether and how it was used, and its strengths and weaknesses. The evaluation team recommends that future programming involving development of similar educational tools include monitoring and evaluation of outcomes, including whether and how the tool was used, what aspects of the tool were more beneficial and why, and what aspects of the tool were less beneficial and why. The evaluation team also recommends that project lifetimes be long enough to allow for preparation and field testing of a draft version of the educational tool prior to publication of the final version. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 32 APPENDICES Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 33 APPENDIX A. EVALUATION SCOPE OF WORK STATEMENT OF WORK A Final Evaluation of Ulaanbaatar School Buildings Thermo-technical Retrofitting Project Implemented by Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) I. OBJECTIVE USAID/Mongolia seeks to recruit a highly qualified Evaluation Specialist with experience in Global Climate Change/Clean Energy to serve as a team leader for the final evaluation of Ulaanbaatar School Buildings Thermo-technical Retrofitting Project. The consultant will lead the project evaluation, and will prepare deliverables prior to and following the evaluation. In addition to the Evaluation Specialist, the evaluation team will be composed of a consultant specialized in Civil Engineering and a staff from USAID/Mongolia Program Office. A detailed description of the Evaluation Specialist’s tasks and deliverables can be found below in the Evaluation Methodology/Team composition and leadership and the Evaluation Deliverables sections. II. PROJECT IDENTIFYIG INFORMATION Program: Global Climate Change/Clean Energy Project Title: Ulaanbaatar School Buildings Thermo-technical Retrofitting Award Number: 438-OUGA-12-001 Award Effective Dates: 23 November 2011 – 22 December 2013 (project activities will be completed by December 2012) Funding: US$1,517,000 Implementing Partner: Deutsche Gesellschaft fur Internationale Zusammenarbeit GmbH (GIZ) Program Manager: Mendsaihan Hasbaatar Previous Evaluations: None III. PROJECT BACKGROUND Fossil fuel combustion is the largest source of CO2 emissions in Mongolia, accounting for about 60 percent of all emissions. The annual per capita emission of greenhouse gas emission (GHG) in CO2- equivalent is relatively high compared to other countries. This is explained by the fact that although the country has a low population (2.75 million), heating is used for nearly 9 months out of the year due to the severe climate. As a result of emissions, Ulaanbaatar is the world’s most polluted capital with regard to particulate matter during the winter months. The costs of pollution-related respiratory and cardiovascular ailments are estimated at $74 million a year. With aging and poorly insulated and maintained infrastructure, energy use in Mongolian public and private buildings is inefficient and contributes significantly to emissions. Public school buildings regularly lack proper thermal insulation and rely heavily on coal burning heat only boilers (HOBs) resulting in extreme heat energy inefficiency (up to 20 times) and emissions. USAID partnered with GIZ, in implementing a Global Climate Change/Clean Energy project focused on thermo-technical retrofitting for three selected school buildings in Ulaanbaatar City. The Ulaanbaatar School Buildings Thermo-technical Retrofitting project was designed to introduce energy efficient technologies in the construction sector to reduce high rates of GHG produced by buildings utilizing coal for heating. The new technologies also help reduce costs to heat the schools and improve the learning environment for students. The design and retrofitting were conducted by local Mongolian companies with technical direction from GIZ, resulting in local knowledge and capacity increase to construct and manage energy efficient buildings. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 34 Thermal rehabilitation of buildings is a process of retrofitting existing buildings with insulated floors, windows, external doors, roofs and walls to improve the heat retention and minimize air leakage of buildings that are otherwise inadequately designed to retain heat. Such buildings are common in Mongolia, described as ‘pre-cast panel’ buildings, largely aging Soviet structures that lack proper heat insulation and any metering or individual heat control. This results in condensation of walls and roofs and other compromising damages to the structures. GIZ previously implemented a pilot thermo technical rehabilitation project of a precast panel building in Ulaanbaatar; thermo technical rehabilitation from that pilot project added up to 50 percent in heat energy savings. The Ulaanbaatar School Buildings Thermo￾technical Retrofitting project includes baseline monitoring and post-project evaluations of heat consumption, savings, and emissions reductions. The development hypothesis of the Ulaanbaatar School Buildings Thermo-technical Retrofitting project follows that thermo-technical retrofitting of existing buildings with standalone coal-powered HOB heating system will decrease coal consumption and therefore GHG emissions. It also follows that GIZ’s technical advisory and on-the-job training of local construction companies through project activities will contribute to building local capacity to use this specific thermo-technical retrofitting technology in the future in Mongolia. GIZ served as a program manager and technical advisor to the project, while the actual rehabilitation of school buildings were completed by qualified Mongolian engineering and construction companies. GIZ led in: (i) design and implementation of the technical specifications for retrofitting, conducted in tandem with Mongolian architects and securing approvals from the Administration of Land Affairs, Construction, Geodesy and Cartography;(ii) cost calculation (priced bill of quantities) in tandem with the selected local engineering company; (iii) tendering the construction/rehabilitation work including; a design tender, tender of contractors, bid-response requirements, including the program description, specifications for applicants and standards for measuring achievements and progress towards finalization, and selection of contractors for retrofitting; (iv) technical guidance, advice and supervision during the design process and construction and; (v) on the job training for engineers, contractors, architects and engineers of consulting offices. The three schools selected for the project are located in low-income areas and heated by standalone (not part of the central heating grid) coal powered HOB. The schools have been identified by local inspection agencies as structurally sound, but in need of repair which qualified them as good candidates for thermo technical rehabilitation. The project expects to see immediate results. Highest level results within five years include;  Reduction in GHG emissions;  Improved thermal efficiency;  Reduced costs/heat energy savings to schools;  A cadre of trained and knowledgeable local builders, engineers and architects able to design and implement retrofits;  Government of Mongolia relevant entities can inspect and approve energy efficient structures, monitor and evaluate GHG emissions; and  Handicap accessibility and sanitary conditions improved in School No. 63 (a special school for mentally disabled children). IV. EVALUATION RATIONALE This evaluation is to assess whether the thermo-technical retrofitting technology introduced and applied in the three selected schools have been successful in saving heat energy (and, subsequently, costs) and reducing GHG emissions, keeping in mind the aim is to use the project as a model for the Government of Mongolia (GoM)’s planned long term rehabilitation of the 160 existing schools in Ulaanbaatar city. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 35 In addition, the evaluation will meet the final GIZ-USAID joint evaluation requirement stated under the grant agreement. The evaluation will also serve as USAID/Mongolia’s High Quality Evaluation and will meet its standards. The evaluation will be a performance evaluation as defined in the USAID Evaluation Policy (http://www.usaid.gov/evaluation). The purpose is to assess performance and whether the project achieved its objectives. The primary user of the evaluation will be the GoM, including the Ministry of Education and Science, the Ministry of Construction and Urban Development, and Ulaanbaatar City Government. The secondary users will be GIZ, USAID, and other donors operating in Mongolia. The final evaluation will be distributed to GoM, GIZ, and the donor community. The evaluation is intended to answer the following questions: 1. What is the percentage change in heat loss/square meter after the retrofits were completed? 2. What is the emission reduction per unit of saving? 3. What is the estimated percentage of decrease in GHG emissions as a result of the retrofits? 4. How much local capacity was built to design and implement retrofitting technology as a result of the project? 5. Did the level of knowledge about energy efficiency increase among school staff as a result of the project? The evaluation will include a desk review of project documents, other relevant documentations pertaining to thermo-technical retrofitting technology introduced and implemented in Mongolia, and consultations/interviews with project participants and beneficiaries. USAID/Mongolia will provide to the Evaluator all project related documents, including project design documents, quarterly reports, baselines for heat energy and coal consumptions, etc. V. EVALUATION METHODOLOGY Team composition and leadership: The Evaluation Team will consist of two international consultants and a staff from USAID/Mongolia Program Office. The consultants will include a Civil Engineer and an Evaluation Specialist. The Civil Engineer will have an expertise in conducting evaluations. The evaluation team members will be required to provide a written disclosure of any conflicts of interest, to include, but not limited to, personal, financial, material, or professional conflicts. The Evaluation Specialist will serve as a Team Leader of the Evaluation Team. In his or her role as Team Leader, the consultant will coordinate this broader team, identifying essential information sources, detailing individual responsibilities, and planning the overall team schedule. Each team member will prepare assigned sections of the final report, but the Team Leader will consolidate and finalize the report and prepare and present the debriefing presentation to the USAID/Mongolia office. The Team Leader will lead and coordinate team members and ensure the quality and timeliness of the deliverables described under section 7. As Team Leader, the consultant will;  Work with the USAID/Mongolia Program Office, before the evaluation team members assemble, to refine a plan of action for information gathering, including document review, key informant interviews, other methods as proposed, and country travel; and  Provide and outline the final evaluation report, discuss with team members, and assign writing responsibilities; ensure timely and quality team contributions. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 36 Time period: The Evaluation will cover the entire period of project performance: November 2011 – December 2012. Suggested methodology: STEP 1: Desk Review, Evaluation Methods and Work Plan  Conduct a desk review of key project and related documents including the Project Design, baseline and on-going data monitoring, gender disaggregated data, periodic progress reports, and other documents in which project activities and findings are discussed. All relevant project documents will be provided to the selected consultants.  At the completion of this phase and prior to the Field Review, the Civil Engineer and Team Leader will submit to USAID/Mongolia the agreed evaluation design, specifying in detail the evaluation methods and their strengths and limitations, data collection instruments, data analysis plan, and work plan (both narrative and timeframe). The written evaluation design will be shared with GIZ and other stakeholders prior to finalization.  Itineraries to visit project sites will be developed in conjunction with USAID/Mongolia and GIZ. Before this phase is complete, the team should have developed any data collection instruments necessary for the methods being used, such as interview protocols, sampling plans, direct observation checklists, survey questions, etc.  A presentation of the design and methodology will be made to USAID/Mongolia, GIZ and project stakeholders following the submission of the desk review, evaluation methods and work plan. STEP 2: Field Review  The Evaluation Team will discuss the Ulaanbaatar School Buildings Thermo-technical Rehabilitation project with key informants as set out in the evaluation methods and work plan. The selection of key informants should be done in a way to reduce selection bias and ensure a broad range of perspectives.  Project locations: The Evaluation Team will spend approximately four days visiting project sites to meet and discuss the project with, and collect other relevant information from target beneficiaries and local counterpart government agencies, in accordance with the proposed evaluation methods. Target beneficiaries and local government counterparts will include directors of project selected schools, representatives from the Ministries of Education and Science, and Construction and Urban Development, Ulaanbaatar City Government and Ulaanbaatar City Office of the General Agency for Specialized Inspection. STEP 3: Data analysis During field work and for up to 3 days after the completion of the Field Review, the Evaluation Team will perform sufficient data analysis enabling the compilation of preliminary findings to be presented to USAID/Mongolia. Data will be disaggregated by gender where appropriate and as required by USAID Evaluation Policy. STEP 4: De-Briefing Session Approximately one day after the completion of the Field Review and data analysis, the Evaluation Team will hold de-briefing sessions for USAID/Mongolia and GIZ. The team will present an Aide Memoire, and seek feedback from GIZ as well as relevant USAID/Mongolia staff on the preliminary findings. The team will make a formal PowerPoint presentation open to all USAID/Mongolia and GIZ staff. STEP 5: Report writing and finalization Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 37 A Draft Report will be prepared by the Evaluation Team and submitted within four days after the de￾briefing Session. USAID/Mongolia and GIZ will review and provide comments within two days. The Evaluation Team will prepare a Revised Draft Report after receiving comments from USAID/Mongolia. The Evaluation Team will incorporate changes and comments, as the Team sees fit. Where there is strong disagreement in views between the Evaluation Team and USAID, these should be noted in an appendix of the Revised Draft Report, known as a Statement of Differences. USAID/Mongolia will review the revised draft report and provide comments within 2 working days. The Evaluation Team will consider all comments in preparing the Final Report, which will be submitted within 2 working days of receiving final comments from USAID/Mongolia. The Final Report should meet the criteria outlined in USAID’s Evaluation Policy and will be uploaded by the USAID/Mongolia/Program Office to the Development Experience Clearinghouse within three months of being finalized. VI. EVALUATION DELIVERABLES The Evaluation Specialist/Team Leader will be responsible for coordinating the drafting of deliverables, consolidating and submitting the final report. The Civil Engineer will be responsible for contributing to all deliverables and drafting relevant sections of all documents and presentations based on his/her expertise and the tasks assigned by the Team Leader. In particular, he/she will ensure that the issues of energy saving, emission reduction and unit cost of retrofitting technology are captured, analyzed and represented in the evaluation process and products. The required deliverables as a joint output for the Evaluation Team are listed below: 1) After the award, the team contracted to carry out the evaluation will submit to USAID/Mongolia a detailed design including evaluation methods and an evaluation work plan of no more than 5-10 pages as a first deliverable prior to field work. This should include: a) Summary of information collected to date, identifying information gaps; b) Evaluation methods to be used to collect required information, and information sources; c) Plans for key informant interviews, site visits, and any other data collection methods; d) List of key informants to be contacted in country, and questions to be asked of them; e) Data collection and analysis plan, including data collection instruments for interviews, site visits, and other data collection methods. The data used in the evaluation should meet the following five data quality standards in accordance to USAID’s Automated Directive System (ADS) 203.3.5.2: 1) Validity; 2) Integrity; 3) Precision; 4) Reliability; and 5) Timeliness (see Appendix 2). f) Roles and responsibilities for each Evaluation Team member; and g) Draft itinerary of the site visit in consultation with USAID/Mongolia Program Office. 2) Revised summary of findings from the desk review prior to field work (no more than 10 pages). 3) Presentation of, a) evaluation methods and work plan and, b) summary of desk review findings to USAID/Mongolia and stakeholders. 4) Aide Memoire summarizing the Evaluation Team’s preliminary findings to be presented at a de￾briefing meeting with USAID/Mongolia and GIZ after the field review. 5) Draft evaluation report will be submitted within 4-5 working days after receipt of feedback from the debriefing. The format for the evaluation report shall be as follows, modified as necessary: a) Executive Summary (3-5 pages) summarizing key points related to the project, evaluation purpose and design, and findings b) Table of Contents c) Introduction: purpose, audience, and task synopsis d) Background: overview of project strategy and components and of evaluation purpose e) Methodology: description of methods and limitations Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 38 f) Findings/Conclusions/Recommendations g) Issues: technical, administrative, and other h) Future Directions i) References (including bibliographical documentation, meetings, interviews and focus group discussions); j) Annexes which will include at minimum Evaluation SOW, Final evaluation design and work plan, any statements of differences, all data collection tools, and identify sources of information 6) Revised Draft Evaluation Report will be submitted within 3 working days of receiving comments from USAID/Mongolia. 7) Final Evaluation Report will be submitted within 2 working days of receiving comments from USAID/Mongolia. The final report will include: a) Executive Summary of the Evaluation b) A main report c) Annexes 8) Any data and records collected by the Evaluation Team will be provided to USAID/Mongolia in an organized and readable format. 9) The main findings of the evaluation will be presented via PowerPoint at a briefing session at USAID/Mongolia. VII. PERIOD OF PERFORMANCE (TENTATIVE) The overall period of performance of this entire consultant services is expected to require approximately 22 working days (6 working days per week) over an elapsed 4-week period from approximately November 27, 2012 to December 21, 2012. An illustrative schedule and time requirement is as follows: Days During Desk Review, Submission of Evaluation Methods and Work Plan (Prior to arrival in Ulaanbaatar) beginning November 27, 2012 4 Week 1 Arrival on December 2, 2012 1 Week 1 Planning meeting with USAID/Mongolia, GIZ and external meetings and interviews in Ulaanbaatar/Field Review with key stakeholders 1 Week 1 Field work – include data analysis 5 Week 1-2 Debriefing with USAID/Mongolia 1 Week 3 Provide draft report December 15, 2012 5 Week 3 Depart on December 16, 2012 - Week 3 Respond to USAID/Mongolia comments & revise draft report and resubmit to USAID/Mongolia December 19, 2012 3 Week 4 Finalization and submission of final report by December 21, 2012 2 Week 4 Total 22 VIII. REQUIRED QUALIFICATIONS AND APPLICATION PROCESS Labor: A consultant bidding on the evaluation will submit a written description of the proposed methodology that would apply for carrying out the Evaluation. He or she will also submit a CV, biodata sheet, and a statement of interest. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 39 USAID/Mongolia will provide administrative, logistics and translation support for the evaluation team. USAID/Mongolia will closely work with the Team Leader to facilitate logistics for the Evaluation Team (e.g. helping to contact Mongolia-based potential interviewees and set up interview appointments, providing translation, booking hotels and transportation, providing overall secretariat work, etc.). While USAID/Mongolia will be responsible for all in-country logistical support, the Team Leader will have the primary responsibility as the Point of Contact between the team and USAID/Mongolia. USAID/Mongolia requires two international consultants for this Evaluation; 1) an Evaluation Specialist functions as the Team Leader; and 2) a Civil Engineer. Desired characteristics for both include:  Field experience working in civil engineering projects, energy efficiency technology; and  Excellent analytical skills focusing on sustainability and feasibility within a non-project context; excellent writing skills. 1. Evaluation Specialist and Team Leader Specific requirements for the consultant include; - Evaluation Specialist;  At least a Master’s degree in business, economics or a related development field;  At least 7 years of strong and substantial experience in monitoring and evaluation; and  Demonstrated ability to work with a broad range of counterparts at all levels; contribute in a team￾oriented workplace; employ strong inter-personal and inter-cultural skills; display good judgment and strong analytical skills; and communicate effectively in both written and oral form. In addition, as Team Leader, he or she will serve as the primary point of contact between USAID/Mongolia, GIZ, and the Evaluation Team. The incumbent must;  Have a proven track record in terms of being highly qualified to lead, coordinate and deliver the evaluation; and  Have excellent writing and organizational skills and proven ability to deliver a quality written product. 2. Civil Engineer: The Civil Engineer needs practical experience in building technologies and energy management. He/she should be acquainted with the construction sector, construction technologies, construction practices, contractors’ habits and problems, site management in developing countries characterized by deficient construction rules and regulations which moreover are often not enforced. He/she should dispose of experiences in school building requirements/standards, capacity building of human resource with focus on short-term skills upgrading training in the construction sector. Special knowledge is required with regard to energy management and the elaboration of energy audits/balances with particular regard to heat energy. Relations/calculations between heat energy and GHG emissions according to the primary energy carrier used should exist. The global discussion on GHG and climate change should be common knowledge. Specific requirement include;  Master’s degree in civil engineering;  Specialization on building technology and energy management;  Experience in cost- and energy efficient retrofitting of buildings, specifically in heat envelopes/thermal retrofits;  Experience with regard to construction and retrofitting in developing countries; and Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 40  Knowledgeable about GHG emissions and mitigation strategies. Estimated Level of Effort The international consultants must be available for 22 working days within a period of 4 weeks (same as Section VI above). This LOE will be broken out as follows; Team preparation, desk review, writing of a summary of desk review findings, developing evaluation methods and Work Plan: 4 days Meetings in Ulaanbaatar, Mongolia with USAID/Mongolia, GIZ and GoM counterpart: 1 day Project site visit, meeting with project stakeholders, including beneficiaries (field sites): 3 days Data analysis: 2 days Debrief at USAID/Mongolia: 0.5 day Preparation of draft, revised draft and final report: 10 days Note: The team is authorized to work on Saturdays. USAID Management of Evaluation USAID/Mongolia will designate the appropriate staff person to serve as the point of contact and a source of technical information about the Evaluation project activities. Logistical Arrangements As stated above, USAID/Mongolia will coordinate administrative, logistics and translation support for the evaluation team. USAID/Mongolia will work closely with the Team Leader to facilitate logistics for the Evaluation Team (e.g. arranging meetings with local construction firms, facilitating contact with Mongolia￾based potential interviewees, providing translation, hotel arrangements and necessary transportation, etc.). Estimated Total Budget for the Consultant Please find attached the total estimated budget. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 41 Supporting Document for Preparation Work Necessary supporting document will be supplied to the evaluation team prior to arrival to USAID/Mongolia. APPENDIX 1 USAID EVALUATION POLICY (20 pages total) APPENDIX 2 DATA QUALITY STANDARDS: USAID’s Automated Directive System (ADS) 203.3.5.1 203.3.5.1 Data Quality Standards To be useful for performance management and credible for reporting, USAID Mission/Offices and Missions should ensure that the performance data in the PMP for each DO meet five data quality standards (abbreviated VIPRT). When this is not the case, the known data limitations and plans to address them should be documented in the indicator reference sheet in the PMP. Note that the same data quality standards apply to quantitative and qualitative performance data. a) Validity. Data should clearly and adequately represent the intended result. While proxy data may be used, the DO Team must consider how well the data measure the intended result. Another key issue is whether data reflect a bias such as interviewer bias, unrepresentative sampling, or transcription bias. b) Integrity. Data that are collected, analyzed, and reported should have established mechanisms in place to reduce the possibility that they are intentionally manipulated for political or personal reasons. Data integrity is at greatest risk of being compromised during data collection and analysis. c) Precision. Data should be sufficiently precise to present a fair picture of performance and enable management decision-making at the appropriate levels. One key issue is whether data are at an appropriate level of detail to inform management decisions. A second key issue is what margin of error (the amount of variation normally expected from a given data collection process) is acceptable given the management and resource decisions likely to be affected. In all cases, the margin of error should be less than the intended change, For example, if the margin of error is 10 percent and the data show a change of 5 percent, the USAID Mission/Office will have difficulty determining whether the change was can be attributed to USAID activity or is a function of lack of precision in the data collection and tabulation process. USAID Missions/Offices should be aware that improving the precision of data often has time and financial resource implications. d) Reliability. Data should reflect stable and consistent data collection processes and analysis methods from over time. The key issue is whether different analysts would come to the same conclusions if the data collection and analysis processes were repeated. USAID Missions/Offices should be confident that progress toward performance targets reflects real changes rather than variations in data collection methods. When data collection and analysis methods change, the PMP should be updated. e) Timeliness. Data should be timely enough to influence management decision-making at the appropriate levels. One key issue is whether the data are available frequently enough to influence the appropriate level of management decisions. A second key issue is whether data are current enough when they become available. For further discussion, see USAID Information Quality Guidelines and related material on the Information Quality Act in ADS 578 and at http://www.usaid.gov/about usaid/. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 42 APPENDIX B. FINAL EVALUATION DESIGN AND WORK PLAN FINAL EVALUATION DESIGN AND WORK PLAN: THE ULAANBAATAR SCHOOL BUILDINGS THERMO￾TECHNICAL RETROFITTING PROJECT PROJECT PURPOSE The purpose of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project is to retrofit three school buildings in Ulaanbaatar City to improve heat retention, minimize air leakage, and improve heating systems, and thereby reduce the energy used for heating and associated fuel costs and greenhouse gas (GHG) emissions. Additional building improvements were made to one of the schools (No. 63, a special school for mentally disabled children) to improve handicap accessibility and sanitary conditions. The project also seeks to produce a cadre of trained and knowledgeable local builders, engineers, and architects able to design and implement retrofits, to increase the abilities of Government of Mongolia (GOM) entities to inspect and approve energy efficient structures and to monitor and evaluate GHG emissions, and to increase the level of knowledge about energy efficiency among school staff. In addition to improving building energy efficiency and reducing associated fuel costs and GHG emissions, the project also aims to facilitate the development of initial (draft) national policy guidelines for creating energy and cost efficient school facilities, and create a leaning module on energy efficiency, renewable energy, and GHG emissions for secondary school teachers and students. EVALUATION PURPOSE The objective of this performance evaluation is to assess whether the thermo-technical retrofitting technology introduced and applied in the three selected schools has achieved its objectives of saving heat energy, reducing energy costs, and reducing GHG emissions. The evaluation will also assess how well the project has increased: 1) the energy retrofit knowledge and abilities of local builders, engineers, and architects; and 2) the awareness and knowledge about building energy efficiency among school staff. As directed in the statement of work, the evaluation is designed to answer the following specific questions: 1. What is the percentage change in heat loss/square meter after the retrofits were completed? 2. What is the GHG emission reduction per unit of energy saving? 3. What is the estimated percentage of decrease in GHG emissions as a result of the retrofits? 4. How much local capacity was built to design and implement retrofitting technology as a result of the project? 5. Did the level of knowledge about energy efficiency increase among school staff as a result of the project? EVALUATION APPROACH The evaluation will be conducted by a team of two international consultants with expertise in performance evaluation, clean energy and GHG analysis, building construction, and building energy management. Barbara V. Braatz will serve as Team Leader and Evaluation Specialist and Clement Tingley will serve as Civil Engineer. Both team members will participate in all phases of the evaluation. While in Mongolia, the Evaluation Team will be provided with logistical and translation support by USAID/Mongolia. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 43 The performance evaluation will start with a desk review of project documents provided by USAID/Mongolia to: 1) assess project objectives and accomplishments; 2) determine and evaluate information collected to date; 3) determine what additional data are needed to complete the evaluation; and 4) establish a procedure for obtaining these data. Based on this desk review, the evaluation team will compile a brief summary of information collected from the desk review and key information gaps. (This summary and list of information gaps is included in this document, as directed in the evaluation SOW.) The desk review will be followed by a field review of the project sites (the three school buildings in Ulaanbaatar) and interviews of key informants, which will include implementing partners, target beneficiaries, and representatives from the GoM and Ulaanbaatar City government. The site visits and interviews will be used to assess the actual building improvements at each site, clarify information provided to the evaluation team, and obtain additional data for analysis. Data will include technical and engineering information about the retrofitted buildings, heating technologies, and heating fuels; and quantitative and qualitative information about capacity in design and installation of building retrofit technologies and knowledge of energy efficiency. The data used in the evaluation will meet the following five data quality standards in accordance to USAID’s Automated Directive System (ADS) 203.3.5.2: 1) Validity; 2) Integrity; 3) Precision; 4) Reliability; and 5) Timeliness, as is relevant to the evaluation and given data limitations of the project. In cases in which project data provided to the team do not meet these standards, this will be noted in the final report.30 The evaluation will be completed over the period 27 November to 22 December. Table 1 presents the work plan schedule, including deliverables and their due dates, and the roles and responsibilities for each Evaluation Team member. Table 2 presents an itinerary of site visits and informant interviews. Table 1: Work Plan Schedule, Team Member Responsibilities, and Deliverables Date Barbara V. Braatz Clement Tingley Deliverable & Deadline Week 1 Wed, 27 Nov – Sat, 1 Dec Desk review, logistics planning, preparation of draft evaluation design and work plan Desk review, logistics planning, preparation of draft evaluation design and work plan Draft evaluation design and work plan by 1 Dec 2012 Sun, 2 Dec Arrive Ulaanbaatar Arrive Ulaanbaatar Week 2 Mon, 3 Dec Prepare presentation Evaluation planning meeting with USAID/Mongolia and GIZ Evaluation planning meeting with USAID/Mongolia And GIZ PowerPoint presentation on draft evaluation design and work plan on 3 Nov 2012 Tues, 4 Nov - Sat, 8 Dec Revise draft evaluation design and work plan if Revise draft evaluation design and work plan if 30 At this stage, the evaluation team foresees that data precision will be the only issue of note. Precision is an issue because the project is not long enough to include post-retrofit monitoring, nor are proxy sites included from which proxy data could be derived. Therefore, energy savings and GHG reductions will be derived from estimates of future fuel consumption and cost, rather than actual values. However, the evaluation team will work to derive estimates that are as precise as possible, given available project data. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 44 Date Barbara V. Braatz Clement Tingley Deliverable & Deadline needed Field work and data analysis needed Field work and data analysis Week 3 Mon, 10 Dec – Wed 12 Dec Field work and data analysis; development of preliminary findings & conclusions; prepare for USAID debriefing Field work and data analysis; development of preliminary findings & conclusions; prepare for USAID debriefing Final evaluation design and work plan by 12 Dec 2012 Thurs, 13 Dec USAID debriefing Work on draft report USAID debriefing Work on draft report PowerPoint presentation on preliminary findings and conclusions on 13 Dec 2012 Fri, 14 Dec Work on draft report Work on draft report Sat, 15 Dec Work on draft report Work on draft report Draft evaluation report by 15 Dec 2012 Sun, 16 Dec Depart Ulaanbaatar Depart Ulaanbaatar Wed, 19 Dec – Fri, 22 Dec Revise draft evaluation report Revise draft evaluation report Final evaluation report by 22 Dec 2012, or within three working days of receipt of all comments from USAID/Mongolia Table 2: Site Visits and Informant Interviews Date Time Location Informant Mon, 3 Dec 2012 15:00-17:00 USAID/Mongolia Ms. Mendsaihan Hasbaatar/Project Management Specialist &, Program Officer, USAID Mr. Michael Foster/Program Officer, USAID Ms. Ruth Erlbeck/Director, GIZ Mr. Ralph Trosse/Technical Director, GIZ Tues, 4 Dec 2012 08:30-18:00 GIZ – Integrated Urban Development Project Office Ms. Ruth Erlbeck/Director Mr. Ralph Trosse/Technical Director Mr. Tuvshinkhuu Samdan/Program Officer, GIZ Mr. Tserendash Sugarragchaa/Senior Officer, GIZ 14:00-18:00 School No. 63 Mr.S.Erdenebaatar/Director of school Nr.63 Wed, 5 Dec 2012 09:00-12:00 School No. 79-1/79-2 Mr.D.Bold/Director of school No.79-1,79-2 Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 45 Date Time Location Informant 13:30-17:00 School Nr.63 Mr.S.Erdenebaatar/Director of school No. 63 13:30-17:00 GIZ – Integrated Urban Development Project Office Mr. Tuvshinkhuu Samdan/Program Officer, GIZ Thurs, 6 Dec 2012 09:00-10:00 Education Department of Ulaanbaatar City Government Mr.Bayarmagnai/Specialist 11:00-12:00 BEEP (Building Energy Efficiency Project) Mr.A.Tsogt/Senior Officer 14:00-15:00 Mongolian Association of Civil Engineers Mr.B.Gantumur/Executive Director Fri, 7 Dec 2012 09:00-10:00 Mongolian University of Science & Technology (MUST) Dr.(Ph.D) Ya.Duinkherjav/Professor of Structural Construction and Chairman of Civil Engineering Department, Head of Concrete and Steel Structure Testing Laboratory 11:00-12:00 Ministry of Construction & Urban Development Mr.Ts. Bayarbat/Director General, Urban Development and Land Affairs Policy, Implementation and Coordination Department Sat, 8 Dec 2012 09:00-10:00 CH Construction Co.Ltd Mr. Ch. Tserenmyadag/Director 11:00-12:00 Gikon Co. Ltd Ms B. Gerelmaa/CEO 14:00-15:30 PST Co. Ltd Ms. Ralkhaasuren/Accountant & Manager 16:00-17:00 KonsKomm Co. Ltd Mr.P.Naranbaatar/Director 17:00-18:30 GIZ – Integrated Urban Development Project Office Ms. Ruth Erlbeck Mr. Ralph Trosse Mon, 10 Dec 2012 10:00-13:30 GIZ – Integrated Urban Development Project Office Mr. Tuvshinkhuu Samdan School No. 79-1 Additional field inspection Tues, 11 Dec 2012 10:00-11:00 GIZ – Country Office Ms Sabine Mueller/Country Representative and Country Director Wed, 12 Dec 2012 14:30-15:30 MES/ Ministry of Education & Science Ms. N. Tuya/Director of External Cooperation Division Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 46 BRIEF SUMMARY OF INFORMATION COLLECTED FROM THE DESK REVIEW AND KEY INFORMATION GAPS The evaluation team was provided with six documents during week 1: Project Design Document (October 2011) Quarterly Report Number 1 (January-March 2012) Quarterly Report Number 2 (April-June 2012) Quarterly Report Number 3 (Jan-March 2012) Grant Agreement between GIZ and USAID (23 November 2011) Heat energy consumption School No. 63 (22 November 2012) These documents provide the following information relevant to the evaluation: project context and purpose; a summary of project goals, intended results, and performance indicators; a discussion of partner organizations and their roles; project schedule; project financial data; qualitative descriptions of the schools, and quantitative information about the buildings. The quantitative building data include relevant physical specifications, heat loss data, and fuel consumption and cost data, for each building, before and after the retrofits. Presumably, the “after retrofit” data are projections. Carbon dioxide emission calculations are provided in the form of a table for each building. The data for school No. 63 that were provided in the first quarterly report were updated in “Heat energy consumption School No. 63 (22 November 2012).” Key information gaps determined to date include:  Precise roles and responsibilities of all the partner organizations in the project, and their expected benefits of participation.  The data sources, methodologies, and assumptions used to prepare the quantitative building, fuel, and GHG data that are presented in tabular format in the first quarterly report and in the file “Heat energy consumption School No. 63.” Also, why did these data change for School No. 63, and are there corresponding revised tables for the other two buildings.  Engineering specifications and drawings and any other studies or analyses associated with the project.  Information on project execution by contractors, including quality of materials used, installation techniques followed and building performance subsequent to rehabilitation.  Information about how the project measured capacity building for different target groups, and what was determined.  Information about the process that has been underway to develop initial (draft) national policy guidelines for creating energy and cost efficient school facilities, achievements to date, the roles of key participants in this process, and expected achievements by the close of the project.  Information about the learning module on energy efficiency, renewable energy, and GHG emissions, especially scope, process, key participants, achievements to date, and expected achievements by the close of the project. EVALUATION METHODS For the first three evaluation questions, the quantitative analysis will be reviewed and input parameters will be validated. This review and validation will be accomplished by:  Reviewing project documents; Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 47  Reviewing construction plans;  Reviewing the quantitative analysis;  Evaluating the input parameters for accuracy and/or appropriateness;  Interviewing the contractors, architects, and engineers; and  Performing field inspections to confirm compliance with construction plans and specifications. For the fourth evaluation question, there will be interviews of architects, contractors, and construction workers to determine the number of people trained, the gender of those receiving training and the extent of the training received. For the final question, school employees and officials will be interviewed to determine the extent of their knowledge about the methods, benefits and operating cost implications of energy efficiency improvements. Illustrative interview guides and a direct observation checklist are presented at the end of this document. The strength of this approach is that it will provide a snapshot of the status of the project, ensure the validity of the approach taken, and provide reasonable estimates of energy and GHG benefits of the project. It does not, however, provide actual data that would enable the accuracy of the quantitative analysis to be validated. For this, the initial project design would have required the collection of a variety of data including actual daily fuel consumption, external and internal temperatures, and quantities and characteristics of the fuel being used, both before and after the retrofits. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 48 ILLUSTRATIVE INTERVIEW GUIDES & DIRECT OBSERVATION CHECKLIST Interview Questions – Architects and Engineers Name:_________________________. Title:_____________________________ Company:_____________________________. Date:____________________: Participation in project:_______________________________________________ 1. How was the U value of the building determined? 2. How was the ventilation heat loss determined? 3. What was the source of the materials for the retrofit, and how was it determined that the materials met specifications? 4. What difficulties did you encounter in sourcing materials and equipment? 5. How many people worked on this project and what was their gender? What level of knowledge did they develop in energy efficiency retrofit methods for buildings? 6. How has this project increased your knowledge of energy efficiency (technologies, benefits, costs) 7. How has this project increased your ability to design/install/operate building envelope retrofit technologies/space heating technologies/water heating technologies? 8. To what extent has this project met your expectations? 9. What were the main achievements of the project? 10. How well do the retrofitted buildings function? How well do they operate? 11. Is there adequate ventilation in the buildings? 12. How well do the people who run the heating system equipment (boilers, dampers, etc), and in No. 63, the solar hot water system, maintain and operate those systems? 13. Schools 79-1 and 63 received new boilers that did not operate successfully. Please explain what the difficulties were. 14. The new boilers can operate on coal or woodfuel. We understand there were problems with the wood fuel. Please explain what the difficulties were. 15. If you were to undertake this work again, would you do anything differently? Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 49 16. Is there additional information or other types of inputs that would have made your work more efficient or more effective? 17. Are there any retrofit design/installation issues that, given what you know, could have been addressed better? 18. How? 19. How was the calorific value of the coal and wood briquettes determined? 20. How were the CO2 and non-CO2 emissions from coal combustion, wood combustion, coal mining, and coal and wood transport derived? 21. What actual “before” data are available to use as a baseline? 22. Are there any actual “after” data yet, and do these support the “after” estimates? 23. Do the coal and wood “after” quantities represent two different fuel supply alternatives, or something else? 24. What are the sources of uncertainty in heat loss values, coal carbon contents, etc. Interview questions – Contractors: Name:_________________________. Title:_____________________________ Company:_____________________________. Date:____________________: Participation in project:_______________________________________________ 1. What difficulties did you encounter in sourcing materials and equipment? 2. How many employees worked on this project and what was their gender? How many were permanent and how many were temporary? 3. What knowledge and skills about energy efficiency retrofits did your employees gain from the projects? 4. How has this project increased your knowledge of energy efficiency (technologies, benefits, costs)? 5. How has this project increased your ability to design/install/operate building envelope retrofit technologies/space heating technologies/water heating technologies? 6. Will the employees who worked on this project have opportunities to apply their new knowledge and skills to other projects like this in the near future? 7. What are the hurdles to more building energy retrofits in both public and private buildings? Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 50 8. If you were to undertake this work again, would you do anything differently? 9. Is there additional information or other types of inputs that would have made your work more efficient or more effective? 10. Are there any retrofit design/installation issues that, given what you know, could have been addressed better? 11. How? Interview questions - School personnel Name:_________________________. Title:_____________________________ Institution:_____________________________. Date:____________________: Participation in project:_______________________________________________ 1 To what extent has this project met your expectations? 2 What was the condition of the building before the retrofits were undertaken? 3 How well do the retrofitted buildings function? How well do they operate? 4 Schools 79-1 and 63 received new boilers that did not operate successfully. Please explain what the difficulties were. 5 The new boilers can operate on coal or woodfuel. We understand there were problems with use of wood fuel. Please explain what the difficulties were. 6 What are the hurdles to obtaining wood fuel for the boilers? 7 The retrofits to School No. 63 will affect electricity consumption. Please explain these effects. 8 How does the school obtain coal for heating? Who manages this process? 9 What did you spend on coal per year before the retrofits? 10 Now that November has passed, can you estimate what coal costs will be this year? 11 What happens to the energy savings due to reduced fuel use at the schools? 12 How well do the people who run the heating system equipment (boilers, dampers, etc), and in No. 63, the solar hot water system, maintain and operate those systems? 13 How well so the school teachers and directors manage the heat inside of the building? Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 51 14 How has this project increased your knowledge of energy efficiency (technologies, benefits, costs)? 15 What are the lessons learned from the project? The strengths and weaknesses? 16 How many students attend your school, and what percentage are girls? 17 How will the learning module be used in your school? 18 Are there any aspects of the learning module that are particularly strong or weak? Interview questions – Government Officials (Ministries and UB City government) Name:_________________________ Title:_____________________________ Institution:_____________________________ Date:____________________: Participation in project:_______________________________________________ 1 Please describe for us the role of your Ministry/Department in this project. 2 To what extent has the project met your expectations? 3 What were the main achievements of the project? 4 What were the notable strengths and weaknesses of the project, in both project design and project implementation? 5 Are there any aspects of this project that have increased your Ministry’s/Department’s knowledge of building energy efficiency, in particular about energy efficient technologies, costs, and benefits? 6 What are the largest barriers to retrofitting other schools, and other public buildings, to reduce energy use? 7 What are the largest barriers to the private sector implementing retrofits to private buildings? 8 Do current construction codes and laws that pertain to new construction include any provisions for energy efficiency? 9 Are there any construction codes or laws that pertain to energy efficiency retrofits to buildings? 10 How are heat and electricity rates in UB City set? 11 Has this project influenced national/city policy on construction and/or building energy use? Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 52 Direct observation checklist: School__:__________________________________ Date:_____________________________________ Approximate outdoor Temperature:__________ Quality of Workmanship: Location: Workmanship (1 – poor; 5 perfect): Notes: Interior Comfort level: Room: Comfort level: Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 53 APPENDIX C. ENGINEER’S FIELD VISIT REPORT FIELD VISIT REPORT: INTRODUCTION From the “Statement of Work”, the “Final Evaluation of Thermo-technical Retrofitting Project” is intended to answer the following questions: 1. What is the percentage change in heat loss/square meter after the retrofits were completed? 2. What is the emission reduction per unit of saving? 3. What is the estimated percentage of decrease in GHG emissions as a result of the retrofits? 4. How much local capacity was built to design and implement retrofitting technology as a result of the project? 5. Did the level of knowledge about energy efficiency increase among school staff as a result of the project? In order to evaluate the model used to determine the percent change in heat loss/square meter after the retrofits were completed, a field review is required to ensure that field execution is consistent with the assumptions and inputs. The architectural plans were prepared in the Mongolian language. A general review of the plans was possible; however a detailed review was not possible as the plans were drafted in Mongolian. Interviews with Mr. Ralph Trosse, senior engineer for GIZ supplemented this review and indicated that the detailed requirements for the thermo-technical upgrades for all schools included: 1. Walls 15 cm of foam insulation 2. Roof 20 cm of foam insulation 3. Floor 10 cm of foam insulation 4. Windows Double paned, six chambered vinyl windows As a part of the roof insulation project the room vent chimneys were cleaned from the top, rain guards were installed and minor improvements to the roof drainage system were made. Schools 63 and 79-2 received new boilers – School 63, 2-80kw, School 79-2, 2-60kw boilers. Along with the boilers, the hot water distribution systems in the two schools were refurbished. This included cleaning the piping and radiator system, replacing piping and radiators as necessary and providing operating valves on the radiators. In general and despite the language difficulties, the architectural plans appeared well drafted with a high degree of detail. The level of detail and clarity of presentation would be considered superior in a more developed country. During the field visits, rooms were randomly evaluated for comfort level and radiators were randomly checked for heating with a FLUKE 61 infrared thermometer. According to the website, the accuracy31 of this device is: 31 http://www.fluke.com/fluke/usen/thermometers/infrared-thermometers/fluke-60- series.htm?PID=56094&trck=61 Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 54 -18 °C (0 °F) to -1 °C (30 °F) ± 3 °C (5 °F) above -1 °C (30 °F) to 275 °C (525 °F)± 2% of reading or 2 °C (3.5 °F) As such, it is useful in the field as a general indictor of relative temperature but is not suitable for more precise applications. A quality control program was implemented as a part of the project. A local engineering firm, Tsats Suvarga International Co,, Ltd., that prepared the plans, was also contracted to provide inspection services. Interviews with the contractors indicated that the independent inspection company was present at all times. The contractors also indicated that GIZ was on the construction site frequently, not only for training purposes but also for quality control purposes. As a part of a previous project, locally produced closed-cell extruded polystyrene foam was evaluated on behalf of GIZ by IBP.32 Their findings are: Designation of the delivered samples Measured Density (kg/m³) Thermal resistance R (m²K/W) Thermal conductivity λ (W/(mK)) 15 kg/m³ 10.3 1.20 0.0410 18 kg/m³ 10.2 1.09 0.0430 25 kg/m³ 11.1 1.20 0.0418 The report concluded, “According to the results there is only an insignificant difference between the material properties of the material samples of different densities. …. In this case, the cheapest material can be used for the insulation measure.”33 Heating upgrade includes a new boiler, cleaning of the piping and radiators, radiator and piping replacement as need and the installation of radiator control valves. The door upgrade consisted of replacement of the front door. School 63 was in the poorest condition and required the greatest intervention. Additional funds for handicapped interventions were obtained for the project. This funded the following: 1. Septic system for sanitary waste disposal. 2. Bathroom and shower retrofits. 32 “Scientific Monitoring of Building Restorations in Ulan Bator and Assessment of Energy Savings Potential as well as Derivation of General Measures for Quality Assurance”. Frunhofer Institut Bauphysik. Responsible Engineer Andreas Kaufmann September 24, 2008. www.ibp.fraunhofer.de. P. 31. 33 Ibid, p. 31 Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 55 3. Handicapped toilets and ramps on the first floor. 4. A new well. 5. A solar hot water system. 6. Sinks in all classrooms. SITE VISIT REPORT – SCHOOL 63 Project: Final Evaluation of Thermo-technical Retrofitting Project Location: Ulaanbaatar, Mongolia School: School 63 Prepared by: Clement Tingley, PE Date: 4-12-2012 Weather: Very cold Sunny, Min Temp -36° C, Max Temp -21° C34 Purpose of Visit: Field review to evaluate thermo-technical retrofits Thermo-technical retrofits: New boilers Piping/radiators to be removed, cleaned, piping replaced as necessary valves provided on radiators. Insulation: Walls 15 cm Roof: 20 cm Floor 10 cm New Windows, new front door. Handicapped retrofits (not a part of this assessment): Septic system for sanitary waste disposal. Bathroom and shower retrofits. Handicapped toilets and ramps on the first floor. A new well. A solar hot water system. Sinks in all classrooms. Comfort level: Generally good. Key observations: 1. The new boilers were on site but not operating. They were located in the boiler building. According to Mr. Trosse, the boiler will not sustain combustion. The current hypothesis is that there is insufficient air to maintain combustion. A forced draft system is proposed to 34 http://www.wunderground.com/history/airport/ Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 56 overcome this problem. The original boiler was cleaned and was in service for heating the school. The operator stated that it was not at full capacity. 2. There was an instrument mounted in the boiler building to monitor hot water circulation temperature and energy consumption. 3. Informal interviews with teachers indicated that the school had been sufficiently cold the previous winter that staff and children were required to wear coats. 4. No children or staff members were wearing coats during our visit. 5. Radiators were checked is all segments of the building. All radiators checked were heating. 6. Rooms were checked in all segments of the building. All rooms checked were sufficiently warm that coats were not required. One room on the first floor at the end of the hot water distribution line was cool. The temperature was measured as 17° C with the Fluke tester. 7. New insulated windows had been installed. 8. Teachers did not seem to be aware that the radiators were operable. One window was observed to be open. 9. The sports hall, which had been unheated and never used, was now in use by children. The temperature level in the room was comfortable. The windows on the north side of the room were reduced in size as shown on the plans. New windows were present on the south side of the room. 10. The insulation thickness on the walls at the windows appeared to be consistent with the 15 cm. specified. Direct observation of foam thickness of the roof and floors was not possible as the material had been covered and was inaccessible. 11. The boiler operator stated that the coal storage area had coal storage capacity for about a ten day supply. 12. The solar panels were producing water sufficiently hot to be used for bathrooms, showers and the kitchen. 13. An exhaust fan had been installed in the kitchen through a glass transom window. The glass in the window was cracked. This was not a part of the project. 14. The pit latrine building is still present at the site. 15. Bathrooms were new. 16. Sinks were present in the classrooms. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 57 SITE VISIT REPORT - SCHOOL 79, BUILDING 1 Project: Final Evaluation of Thermo-technical Retrofitting Project Location: Ulaanbaatar, Mongolia School: School 79 Building 1 Prepared by: Clement Tingley, PE Date: 5-12-2012 Weather: Very cold Sunny., Min Temp -36° C, Max Temp -21° C35 Purpose of Visit: Field review to evaluate thermo-technical retrofits Thermo-technical retrofits: Insulation: Walls 15 cm Roof: 20 cm Floor10 cm New Windows Comfort level: Temperatures were generally warm, humidity seemed high. Key observations: 1. The boiler was not replaced in this school. It is heated by new boilers that serve several buildings in the immediate vicinity. 2. Informal interviews with teachers indicated that the school had been sufficiently cold the previous winter that staff and children were required to wear coats. 3. There was no work performed on the hot water distribution system in this school. The radiators did not have control valves. 4. Rooms were checked in all segments of the building. All rooms checked were sufficiently warm that coats were not required. One room seemed exceptionally hot and measured 30° with the Fluke tester. 5. New insulated windows were installed. Excessive condensation was noted on a number of windows (more than in schools 63 and 71-2.) 6. One broken window was observed. Four windows were partially open. 7. The foam insulation on the exterior walls was measured as consistent with the 15 cm. specified. Direct observation of foam thickness was not possible for the roof and floors as the material had been covered. 8. A sports hall was added to this building prior to the start of construction for this project. The sports hall was not included as a part of the thermo-technical retrofit project. The heating in the new sports hall was in operation. Temperatures within the room measured 10° C. with the Fluke tester. 9. The school is crowded; the school was designed to accommodate 320 kindergarten children. It now accommodates 1300 children from grades one to five in two shifts.36 Second visit observations: 1. A second visit was made to explore possible causes of excessive window condensation. 35 http://www.wunderground.com/history/airport/ 36 Grant Number 438-OUGA-12-001 with Deutsche Gesellschaft Fur Internationale Zusammenarbeit (GIZ) GmbH, p. 14. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 58 2. The visit to the roof revealed that a number of vents were partially or fully obstructed by frost, ice or other material. 3. Rain guards were present on all but one vent chimney. The mounting brackets were present where the rain guard was missing. 4. The caps for the masonry walls were present. These are necessary to prevent moisture entering the walls from the top. SITE VISIT REPORT – SCHOOL 79, BUILDING 2 Project: Final Evaluation of Thermo-technical Retrofitting Project Location: Ulaanbaatar, Mongolia School: School 79-2 Prepared by: Clement Tingley, PE Date: 5-12-2012 Weather: Very cold Sunny, Min Temp -36° C, Max Temp -21° C37 Purpose of Visit: Field review to evaluate thermo-technical retrofits Thermo-technical retrofits: New boilers Piping/radiators to be removed, cleaned, piping replaced as necessary valves provided on radiators. Insulation: Walls 15 cm Roof: 20 cm Floor 10 cm New Windows. Comfort level: Generally good. Key observations: 1. The new boilers were on site but not operating. According to Mr. Trosse of GIZ, the boiler will not sustain combustion. The current hypothesis is that there is insufficient draw. A forced draft system is proposed to overcome this problem. The original boiler was put back into service. During the site visit, it was in use for heating the school. 2. Informal interviews with teachers indicated that the school had been sufficiently cold the previous winter that staff and children were required to wear coats. 3. Radiators were checked is all segments of the building. All radiators checked were heating. 4. Rooms were checked in all segments of the building. All rooms checked were sufficiently warm that coats were not required. 5. New insulated windows were installed. 6. Windows were observed as being open. This was perhaps to help cool the rooms. 37 http://www.wunderground.com/history/airport/ Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 59 7. The foam insulation on the exterior walls was observed as consistent with the 15 cm. specified. Direct observation of foam thickness was not possible for the roof and floors as the material had been covered. 8. The school is crowded; the school has capacity to accommodate 380 children. It now accommodates 1100 children from grades six to eleven in two shifts. The twelfth grade will start in 2014.38 9. A new brick building, apparently not insulated, for housing faculty has been constructed behind the school. Work is underway to connect this building to the heating system of the school. Mr. Trosse of GIZ stated that the school had been advised not to do this. 10. A privately owned building adjacent to the school had foam blocks applied to the exterior. 11. A recently constructed building identified as 79-3 was adjacent to the building. There was a new boiler in the boiler building that serves 79-3. 38 Grant Number 438-OUGA-12-001 with Deutsche Gesellschaft Fur Internationale Zusammenarbeit (GIZ) GmbH, p. 15. Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 60 APPENDIX D. CONFLICT OF INTEREST STATEMENTS Evaluation of the Ulaanbaatar School Buildings Thermo-technical Retrofitting Project 61