EVALUATION OF DRUM AND WENEXA April 2011 This publication was produced for review by the United States Agency for International Development. It was prepared by Dr. Kevin Warr, Mr. Michael Gaffen, Dr. ArunachalamRajagopal, and Dr. MeenuMishra through Social Impact. EVALUATION OF DRUM AND WENEXA ASSESSING THE IMPACT OF USAID/INDIA’S DISTRIBUTION REFORM, UPGRADES AND MANAGEMENT PROGRAM AND ITS WATER AND ENERGY NEXUS PROGRAM Dr. Kevin Warr, Mr. Michael Gaffen, Dr. Arunachalam Rajagopal, Dr. Meenu Mishra DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc., with Management Systems International, to the United States Agency for International Development under USAID Contract No. RAN-I-00-09- 00019-00, Task Order No. AID-386-TO-10-00003. EVALUATION OF DRUM AND WENEXA i ACKNOWLEDGMENTS This evaluation was complex, with many different components; it required extensive document review and numerous interviews with many different people in nine cities. We would not have been able to complete the task without the generous assistance of Tetra Tech, particularly Mr. Rakesh Kumar Goyal, Mr. Metikurke Vasuki, Mr. B. V. Rudresh, Mr. V. C. Shah, Mr. Shafaat Ali Khan, and Mr. Sachin George. The Tetra Tech team has worked tirelessly to make introductions and provide essential documents as well as valuable institutional memory. Your assistance with logistics was also most helpful. We would also like to thank Mr. James Hogan of Tetra Tech, whose memory is truly impressive. Our gratitude also extends to the management and staff of BESCOM and the Doddaballapur subdivision, MSEDCL and the Aurangabad Unit 1 subdivision, and MGVCL and the Umreth subdivision. Your hospitality and patience with all our questions were much appreciated by the team and helped make our jobs much easier. Thank you for taking the time out of your busy days to assist us. Finally, we would like to thank USAID, in particular Ms. Elizabeth Callendar, Ms. Monali Zeya Harza, Ms. Apurva Chaturvedi, and Mr. Chandan K. Samal. Your dedication to making this the best possible evaluation was much appreciated and we are grateful for your kind assistance. ii EVALUATION OF DRUM AND WENEXA ACRONYMS APDRP Accelerated Power Development and Reforms Program AT&C Aggregate Technical and Commercial (Losses) BESCOM Bangalore Electricity Supply Company CAIDI Customer Average Interruption Duration Index CLEEO Clean Energy and Environment Office (of USAID/India) DPR Detailed Project Report DRUM Distribution Reform, Upgrade and Management GOI Government of India GW Giga-Watts HVDS High Voltage Distribution System JNNURM Jawaharlal Nehru National Urban Renewal Mission KKPI Key Key Performance Indicator KPI Key Performance Indicator MahaGenco Maharashtra State Power Generation Company MSEDCL Maharashtra State Electricity Distribution Company, Limited MSI Management Systems International MW Megawatt MYT Multi-Year Tariff NABARD National Bank for Agriculture and Rural Development NDPL North Delhi Power Limited NMC Nagpur Municipal Corporation PFC Power Finance Corporation (the Ministry of Power’s finance arm) RAPDRP Restructured Accelerated Power Development and Reforms Program SAIDI System Average Interruption Duration Index SAIFI System Average Interruption Frequency Index SI Social Impact SOW Scope of Work USAID U.S. Agency for International Development WENEXA Water & Energy Nexus Activity EVALUATION OF DRUM AND WENEXA iii Table of Contents ACKNOWLEDGMENTS................................................................................................................................... I ACRONYMS ........................................................................................................................................................II EXECUTIVE SUMMARY ...................................................................................................................................1 1.0 INTRODUCTION.......................................................................................................................................1 2.0 BACKGROUND..........................................................................................................................................3 2.1 THE PROBLEM STATEMENT ......................................................................................................................3 2.2 THE THEORY OF THE INTERVENTION ....................................................................................................4 2.3 PROJECT OBJECTIVES.................................................................................................................................4 2.4 PROJECT IMPLEMENTERS ...........................................................................................................................5 2.5 GEOGRAPHY OF PROJECT IMPLEMENTATION........................................................................................6 2.6 COST...........................................................................................................................................................6 2.7 PROJECT DESIGN .......................................................................................................................................6 3.0 PURPOSE OF EVALUATION...................................................................................................................7 4.0 RESEARCH DESIGN AND EVALAUTION METHODOLOGY......................................................8 5.0 FINDINGS......................................................................................................................................................9 5.1 PROCESS FINDINGS....................................................................................................................................9 5.1.1 Effectiveness of management ..................................................................................................................... 9 5.1.2 Coordination with GOI and stakeholders...............................................................................................11 5.1.3 Ability to strengthen public/private/civil society linkages....................................................................12 5.2 OUTCOME AND IMPACT: DRUM ........................................................................................................ 13 5.2.1 Capacity Building...........................................................................................................................................13 5.2.2 Pilot Projects...................................................................................................................................................16 5.3 OUTCOME AND IMPACT: WENEXA .................................................................................................. 28 5.4 RELEVANCE: DRUM............................................................................................................................... 28 5.5 RELEVANCE: WENEXA......................................................................................................................... 30 5.6 EFFECTIVENESS: DRUM ......................................................................................................................... 30 5.6.1 Cost Effectiveness (Return on Investment)............................................................................................30 5.6.2 Management Structure...............................................................................................................................33 5.6.3 Knowledge Transfer.....................................................................................................................................33 5.6.4 Donor Coordination......................................................................................................................................33 5.7 EFFECTIVENESS: WENEXA ................................................................................................................... 34 5.7.1 Return on Investment ..................................................................................................................................34 5.8 SUSTAINABILITY: DRUM ....................................................................................................................... 35 5.8.1 Training............................................................................................................................................................35 5.8.2 Pilots..................................................................................................................................................................35 5.8.3 Exchange Program........................................................................................................................................36 5.9 SUSTAINABILITY: WENEXA................................................................................................................ 36 5.10 GENDER................................................................................................................................................ 38 6.0 CONCLUSIONS ....................................................................................................................................... 39 7.0 RECOMMENDATIONS .......................................................................................................................... 39 7.1 PROCESS RECOMMENDATIONS ............................................................................................................ 39 7.2 PROGRAMMATIC RECOMMENDATIONS............................................................................................... 40 8.0 LESSONS LEARNED................................................................................................................................ 40 iv EVALUATION OF DRUM AND WENEXA ANNEX 1: DRUM/WENEXA EVALUATION: STATEMENT OF WORK ....................................... 42 ANNEX 2: PERSONS CONTACTED........................................................................................................ 55 ANNEX 3: PILOT SITE KKPI TRACKING................................................................................................ 57 ANNEX 4: INTERVIEW QUESTIONS....................................................................................................... 60 ANNEX 5: TRAINING SURVEY.................................................................................................................. 65 ANNEX 6: GENDER ASSESSMENT ........................................................................................................... 72 ANNEX 7: REFERENCES............................................................................................................................... 85 Table of Figures Figure 1: DRUM and WENEXA implementation sites........................................................................................................2 Figure 2: Utility customer satisfaction scores in Doddaballapur across segments............................................21 Figure 3: Electricity supply customer satisfaction, 2005 and 2010 .........................................................................26 Figure 4: Customer satisfaction on key parameters, 2005 and 2010. .....................................................................22 Figure 5: Overall satisfaction with electricity supply, 2005 and 2010....................................................................31 Figure 6: Customer satisfaction across utilities, 2005 and 2010...............................................................................31 Figure 7: Overconsumption of an underpriced commodity .........................................................................................41 Figure 8: Quality of DRUM training ......................................................................................................................................69 Figure 9: Usefulness of DRUM training................................................................................................................................69 Table of Tables Table 1: Key Performance Indicators for electricity availability and reliability.................................................20 Table 2: Key Performance Indicator targets for Aurangabad....................................................................................24 Table 3: Electricity Act of 2003: Broad Distribution Highlights................................................................................29 Table 4: Cost-benefit analysis of installing capacitors at pilot sites........................................................................31 Table 5: Key Performance Indicators for Doddaballapur pilot site .........................................................................53 Table 6: Key Performance Indicators for Aurangabad pilot site...............................................................................54 Table 7: Key Performance Indicators for Umreth pilot site.........................................................................................55 Table 8: Year and number of people attending DRUM training................................................................................68 Table 9: Location and number of people attending DRUM training........................................................................69 Table 10: Women’s uses of electricity...................................................................................................................................73 Table 11: Commonalities and differences between Aurangabad and Umreth.....................................................78 Table 12: Women’s priorities for electricity use...............................................................................................................83 EVALUATION OF DRUM AND WENEXA 1 EXECUTIVE SUMMARY Since the electrification of Bangalore (the first city in India to be electrified) in 1906, India’s power sector has experienced tremendous growth and change, but also has been faced with challenges. Today, 78 million rural households do not have electricity and nearly 40 percent of Indians still rely on kerosene lamps for light1. Yet India is on the move with more than 10.4 percent growth last year2. As India quickly moves into the ranks of “developed countries” an electric power sector that is efficient and effective is imperative if the country is to sustain its upward trajectory. However, many utilities are unreliable suppliers and financially unsustainable. A modern and financially viable competitive electric power industry is essential to attract foreign direct investment. Against this background, the United States Agency for International Development (USAID) launched the Distribution Reform, Upgrade and Management (DRUM) program in 2004. The program sought to: • Establish the framework, institutional capacity, and project development functions at the central and state levels; and • Enable implementation of several full-scale, commercially replicable distribution initiatives in key reform states in India. USAID chose to focus on the distribution sector—the sector closest to the consumer— (rather than generation or transmission) because of the belief that without a well-functioning distribution sector, the rest of the sector would be unattractive to private investment, a goal of the Electricity Act of 2003. The reform of the distribution sector was carried out in three focus areas: • Large-scale training of distribution sector employees; • Three pilot sites that would serve as demonstration projects for technical and managerial upgrades to the distribution system; and • An exchange program with the Rural Utilities Service, a branch of the U.S. Department of Agriculture The training component consisted of the development of 24 short-term training courses on 19 subjects, delivered in 20 training institutes throughout the country. The original goal was to train 25,000 upper-, middle-, and low-level utility employees throughout the country by September, 2008. When the program was extended, the goal was revised to train 35,000 professionals by September, 2012. The training program is on track and to date has trained more than 33,000 people in short-term training courses, 150 people in a year-long energy management program, and over 1,500 people in a comprehensive distance learning program. Overall, the training component has been very successful and well worth the money USAID spent. There is now a culture of training (at all levels) in the pilot utilities as well as some non-DRUM utilities3. The DRUM training framework has been adopted by the Government of India (GOI) to impart its own training under its urban electrification program, RAPDRP. Additionally, some of the private sector training institutes are modifying the DRUM materials to continue the training after the end of the program. The training program is now able to stand on its own without USAID support. Indeed, the long-term energy management course initiated by DRUM has increased its tuition by 328 percent, while applications for enrollment have tripled. 1 Harmeet Shah Singh, “Indian village marks 25 years in dark despite electric posts,” February 14, 2010, CNN. 2 IMF World Economic Outlook 2011. 3 North Delhi Power Limited, NDPL, for instance. 2 EVALUATION OF DRUM AND WENEXA The main impediment to a fully successful program is the lack of any methodology to measure how much the participants learned as a result of the training program and how it affected their job. The focus on outputs (number of people trained) at the expense of outcomes (what they learned and how they used it) is a weakness in the design of the program. The evaluation team recommends future training programs include a method of measuring knowledge gained through training, as well as a follow-up mechanism, in the design phase. The pilot sites have also been successful in upgrading their distribution systems with a number of state-of-the-art technological upgrades, as well as significant improvements in billing and collection, metering, and customer service. As a result of DRUM, electricity losses in the three pilot sites were reduced from an average of 53 percent to an average of 15 percent, a notable accomplishment and a significant improvement in the utilities’ efficiency. Billing and collection have improved to 100 percent and transformer failures are now only 2 percent, compared to 10 percent prior to DRUM. Customers—especially agricultural customers—report increases in their incomes of up to 15 percent as a result of having more reliable power, and better quality power. Clearly, the impact of the DRUM pilot projects is positive. However, as projects meant to be replicated, the pilot sites have not met their stated objectives: • To establish the framework, institutional capacity, and project development functions at the central and state levels, and • To enable implementation of several full-scale, commercially replicable distribution initiatives in key reform states in India DRUM not only included technological upgrades and civil works, but also a systematic process of assessing the gaps and determining how to address them in the most cost-effective manner, while also identifying best practices (key performance indicators) and the best method to measure progress toward achieving those indicators. While technological aspects of DRUM have been replicated elsewhere in the utilities under the Ministry of Power’s Restructured Accelerated Power Development and Reforms Program (RAPDRP), no DRUM-like projects have been replicated. There are several possible reasons why the pilots have not been replicated. First, innovation within state-owned enterprises is difficult, as the locus of change remains external (either in the form of the Ministry of Power or the regulatory body). To that extent, a complex and demanding reform program like DRUM faces large hurdles in terms of replication. Second, there have been gaps in the dissemination of lessons learned, though it should be noted that the nationwide conference in February 2011 rolling out the DRUM results, as well as the book of DRUM case studies that was recently published are clear steps in the right direction. The DRUM website is excellent and the newsletter has a wide distribution, but for change to occur elsewhere within the target utilities or other Indian distribution utilities, key decision makers must be enfranchised. Without their support and leadership, DRUM-like reforms are unlikely to get off the ground. If USAID were to undertake similar pilot projects, it is recommended to design a comprehensive communication strategy aimed at key decision makers while fostering interactions for the key participants during the life of the program. This should happen at the design stage. The exchange program was also a success and is now self-sustaining. In fact, the target utilities for the program, Bangalore Electricity Supply Company (BESCOM) and the Maharashtra State Electricity Distribution Company, Limited (MSEDCL), have been paying their own way for the exchange visits to rural utilities in Texas, California and Colorado. Many executives with whom we spoke were able to list specific things they learned during the exchange visits that they have implemented upon returning home. One of the most important changes to come out of the EVALUATION OF DRUM AND WENEXA 3 exchange visits is the focus on the safety of the linemen and the development of copies of U.S. safety manuals in the local language. Additionally, in recent years, the DRUM program also included a conference and training on new smart grid concepts as part of the evolving advanced electric power technology in the 2011 program. The Water & Energy Nexus Activity (WENEXA) project was both timely and relevant to the Indian context. In India, the water and energy sectors are inextricably intertwined. Both water and electricity are virtually free to farmers and agricultural use accounts for 30 percent of the electricity used in the country, while 70 percent of the population makes their living from agriculture. This signifies plenty of users are receiving free water and electricity, and when a commodity is free there is no incentive for efficient use. WENEXA has had several, somewhat disparate, components. Though the latest version of WENEXA was started in 2004, it did not make much headway until 2007. This is partly due to the complexity of the problem, as well as the size of the program, which is considerably smaller than DRUM. Additionally, the program was managed out of USAID/India’s energy team, so energy was their key focus. In the end, the implementing partner, Tetra Tech, developed a very innovative program that would use a public/private/civil society partnership to replace energy inefficient water pumps with energy efficient ones, at no cost to the farmer. The scheme involves an agreement between an Energy Services Company, Enzen, and BESCOM, under which of both partners share a certain proportion of the benefits of the energy savings. For every unit of energy saved above the baseline consumption recorded in 2009, BESCOM will receive 11.57 percent of the benefits from the energy savings and the rest by Enzen. The program will last for 10 years, while USAID’s role in designing the program and conducting the baseline study (which was complex, costly, and lengthy) is only about two years. The evaluation team recommends that USAID develop a system to continue to monitor and evaluate the results of this innovative project over the life of the program, so as to capture the results and possibly replicate elsewhere. Another component of the WENEXA program concerns watershed management and water table replenishment through the construction of a catchment dam, as well as bunding and trenching activities in the Doddaballapur area. There is anecdotal evidence that the water table is rising, but scientific verification is impossible as there is no baseline study. To that extent, there is no way to measure results and it is impossible to determine whether these one-off activities were a wise use of funds. The evaluation team recommends avoiding one-off pilot projects, especially any with no mechanism to measure results. A third part of WENEXA involved an urban wastewater treatment project in Nagpur. This component is also an innovative project based on the collaboration between the Nagpur Municipal Corporation, Maharashtra Electricity Generation Company and USAID/Tetra Tech. The project also benefitted from the support of relevant government departments (Environment, Water Supply and Ground Water). The aim of the project was to treat about 245 million liters a day of municipal wastewater and provide it for use mainly in thermal power plants. The project also contributed to the formulation of a Maharashtra state policy on wastewater reuse. The project has clearly demonstrated the importance and usefulness of recycling wastewater through treatment. Wastewater recycling of this sort contributes to savings in the water and energy sectors. On the whole, the results of both DRUM and WENEXA are mixed. DRUM has seen notable successes in all three components, though the stated objectives have not been achieved. Nevertheless, people in the target communities are better off as a result of DRUM and the target utilities are able to operate more efficiently and provide better service. Both the training component and the exchange component are likely to continue and to that extent, they are fully sustainable. While there has been no replication of the DRUM reform initiatives to 4 EVALUATION OF DRUM AND WENEXA date, this still could happen, especially as the final results of DRUM become better known and more efficient dissemination of the information arrive at diverse utilities throughout India. The final impact of WENEXA, in particular the pump replacement program, remains to be seen, but the design of the program itself is impressive and, if the results bear out, replicable throughout the country. It would be a shame to lose the lessons learned as it is implemented simply because USAID funding for the program is ending. Based on the evaluation findings and the situation of India’s energy sector, the evaluation team recommends that in the remaining 18 months of the DRUM training program, USAID: • Develop DRUM environmental training related to distribution reform. • Improve outdated DRUM materials, especially technical modules. • Develop a pre- and post- testing mechanism for all new training. • Develop and implement short, Demand Side Management (how to use electricity efficiently) awareness programs for women in DRUM pilot sites. In terms of future exchange programs, the team recommends that USAID: • Set up an exchange program focused on Smart Grid development and implementation. • Set up an exchange program focused [on] renewable energy project development. • Provide an exchange of experts on establishing and maintaining a financially and economically viable electric utility4. Pilot projects are useful, but only if the lessons learned from them are used to develop better projects. The evaluation team recommends that the lessons learned from DRUM inform, and be integrated in, future pilot projects: • Replicate a revised DRUM platform for Smart Grid pilot projects, taking into consideration the requirement for replication. • Replicate the DRUM platform for the use of advanced technology and implementing the capabilities of the installed smart meters. The water-energy nexus will continue to be of significant importance to India and USAID should develop an assistance strategy for water and energy that has clear objectives and measureable results. Such a strategy could have notable development impacts in the future, but any strategy must take an integrated approach, which means including both water and energy officials, as well as civil society, in the design and management of the program. Further, USAID should: • Continue watershed work, including proper baseline studies, a comprehensive communication program, capacity building at the grassroots level, and involvement of farmers in a water and energy management program. • Replicate and refine the Nagpur wastewater model in other locations, with a view toward establishing best practices and disseminating lessons learned. Consumers evaluate the performance of the electric power system on the efficiency and dependability of the service. When there is sufficient electric generation and transmission, the 4 Potential contacts are the U.S. Energy Association (USEA), The National Association of Regulatory Utility Commissions (NARUC), the U.S. Department if Energy, and major electric utilities. EVALUATION OF DRUM AND WENEXA 5 primary culprit for a failure of delivery is at the distribution level and this has been the weak point for the Indian electric chain. The DRUM program was designed to mitigate this problem by transferring information and initiating innovative programs at three pilot sites. The results of the DRUM program have been excellent in terms of the transfer of knowledge to the technical staff and the pilot programs have bettered the lives of the beneficiaries. However, the challenges of implementing innovative ideas in an evolving energy sector have also been clear. The examples provided by DRUM and WENEXA can produce a positive model for utilities throughout the country. We anticipate that this evaluation will provide information and suggestions to improve USAID/India’s existing energy and water programs and help the decision makers at USAID develop support for future programs for the electric power and water sector during this critical period in the evolution of the economy. EVALUATION OF DRUM AND WENEXA 1 1.0 INTRODUCTION This evaluation covers two of USAID’s more innovative projects in India, the Distribution Reform, Upgrades and Management (DRUM) program and the WENEXA program, which is focused on the nexus between energy and water in India. The evaluation activities took place from February 28, 2011 to March 26, 2011 in Bangalore and Doddaballapur; Mumbai and Aurangabad; and Vadodara and Umreth. The evaluation team consisted of two Americans and two Indians, all experts in the electricity and/or water field, and all with strong project evaluation backgrounds. The evaluation is meant to document the impact of both projects in as much detail as possible, given the resources available. Furthermore, it is to serve as a guidepost for USAID as it develops new programs in the energy and water sectors. The evaluation is organized in the following manner: • Findings, organized around the following themes: o Process (management, donor coordination) o Outcome and Impact (What has been the effect of the projects on the beneficiaries?) o Relevance (At the time of inception, were the projects in line with the Government of India’s plans for the energy sector? Have the projects continued to contribute to the GOI’s energy sector plans and policies?) o Effectiveness (Have the interventions achieved what they were meant to achieve?) o Sustainability (Are the gains to the beneficiaries likely to continue over time?) o Gender Mainstreaming • Conclusions • Lessons Learned • Recommendations We have taken care to ensure the report is as accessible to people unfamiliar with the sector as to those familiar with it, and that the lessons learned from both projects have applicability to other development programs. It is our intention and hope that this report will serve as a learning tool to help develop better and more effective programs in the future. 2 EVALUATION OF DRUM AND WENEXA Figure 1: DRUM and WENEXA implementation sites EVALUATION OF DRUM AND WENEXA 3 2.0 BACKGROUND 2.1 The Problem Statement Demand for reliable and affordable energy in India is growing rapidly. As India’s economy grows (more than eight percent in 2010) the demand for electricity also grows because reliable and affordable electricity is a necessary condition for economic growth in today’s global economy. India has the fifth largest generation capacity in the world with an installed capacity of 171, 926.4 Megawatts (MW) as on 28 February 2011, approximately four percent of global power generation. The Government of India (GOI) has set ambitious goals for the power sector and aims to provide over 1000 Kilowatt Hours (kWh) of electricity per capita by 2012. In 2009, the average per capita consumption of electricity in India was estimated to be 720 kWh. It is estimated that India will need to add additional capacity of more than 100,000 MW to reach its 2012 goal5 . The Indian Power sector, however, is faced with difficulties as demand continues to outpace supply, while the quality of supply remains uneven at best. In 2003, the year prior to the development of DRUM, the gap between demand and supply was 16 percent and the shortage of power at peak periods was 24 percent. Aggregate Technical and Commercial (AT&C) losses were greater than 40 percent and the transformer failure rate was 10 percent. In 2003, 30 percent of India’s villages remained un-electrified. Indeed, historically, high technical and non-technical losses, widespread theft, poor billing practices, large subsidies, and frequent load shedding have characterized India’s power sector. Distribution is the most critical segment of the electricity business chain. However, at the time the Electricity Act of 2003 was passed, the system was in such disarray that the few attempts at privatization failed and interest from the private sector was weak. The Electricity Act of 2003 was the GOI’s initiative to reform the entire sector. The Act, among other things, calls for the privatization of the system. That was one of the many reforms that has created a new paradigm for the Indian power sector as a result of the Act. It is in this context that USAID conceived the DRUM project. The hypothesis behind the DRUM project is that reforming the distribution sector (the sector closest to the consumer) is an essential first step to reforming the rest of the power sector and a crucial step in increasing the attractiveness of the sector to private investors. In other words, unless the distribution sector is commercially viable, private investors will not be interested in investing either in the distribution sector, nor the generation sector. One observer noted that the Indian power sector is like a leaky bucket into which water is continually poured, never reaching the top. The solution, therefore, is not more water, but fixing the bucket. Reforming the electricity distribution system—both in terms of technical upgrades and capacity building for implementation of reform—is at the heart of DRUM. The reforms under DRUM dovetail with those of USAID’s Water-Energy Nexus (WENEXA) project and the problems that DRUM was designed to mitigate in the electric power sector are also apparent in the water sector. A significant portion of India’s power sector challenges comes from the agricultural sector. 70 percent of India’s workforce is agricultural and nationally, about 30 percent of total electricity consumption comes from the agricultural sector. Farmers are by far the largest users of water in India. In fact, India is the largest user of ground water in the world, with an estimated use of 230 cubic km of ground water every year, which is more than 25 percent of the global level. Ground water supports around 60 percent of irrigated agriculture and more than 80 percent of rural and urban water supplies6 . The heavy use of electricity and water resources poses multiple problems. First, according to the World Bank, around 29 percent of ground water blocks in the country are semi-critical, critical or 5 KPMG India, “Power Sector in India: White paper on Implementation Challenges and Opportunities,” For release at the Energy Summit, Nagpur - January 2010. Government of India, Ministry of Power, Annual Report. 6 Ajith Athrady “India’s Ground Water Table to Dry Up in 15 years,” Deccan Herald, March 7, 2011. 4 EVALUATION OF DRUM AND WENEXA overexploited. This situation is deteriorating rapidly and by 2025, an estimated 60 percent of ground water blocks will be in a critical condition. The potential impact of climate change will further strain ground water resources7 . Second, farmers pay nothing for water and very little (if anything) for electricity, thus making commercial viability of the electricity distribution utilities a Sisyphean task. Additionally, most farmers use energy inefficient water pumps, which further reduce the distribution utility’s ability to meet the energy demands of all its customers. It is in this context that USAID launched the WENEXA project, which aims to bring the energy and water sectors together to address common problems (lack of electricity, a rapidly-depleting water table) through an innovative public/private/civil society partnership. It should be noted that WENEXA was designed in two parts, WENEXA 1 and WENEXA 2. The focus of WENEXA 1 was to introduce water-energy best practices to farmers through a pilot in Andhra Pradesh while the focus of WENEXA 2 is to pilot water energy co-management not only in agriculture (through Agriculture Demand Side Management) but also in the Indian municipal and industrial sectors. WENEXA 2 does not end until July 2011 so the nature of our assessment of WENEXA 2 is more of a mid-term evaluation than a final evaluation. 2.2 The Theory of the Intervention The DRUM project aimed to showcase replicable models of commercially viable electricity distribution along with supporting institutional structures at the state and central level. DRUM had three focus areas: • Exchange visits between two of the target utilities, the Bangalore Electricity Supply Company (BESCOM) and the Maharashtra State Electricity Distribution Company, Ltd. (MSEDCL) and the U.S. Department of Agriculture’s Rural Utility Service (RUS); • Support for Functional Training at all levels (top, middle, and low); and • Distribution Reform Pilot Projects in three subdivisions of BESCOM, MSEDCL, and the Madhya Gujarat Vij Company, Ltd. (MGVCL) in Gujarat state. The three subdivisions were: Doddaballapur for BESCOM, a rural site; Aurangabad for MSEDCL, an urban site; and Umreth for MGVCL, another rural site. The WENEXA project (in its second phase) seeks to improve co-management of energy and water resources in the agricultural, urban and industrial sectors through enhanced power distribution and end-use efficiency, coupled with sound water management practices. 2.3 Project Objectives Overall, DRUM had two key objectives: • To establish the framework, institutional capacity and project development functions at the central and state levels; and, • To enable implementation of several full-scale, commercially replicable distribution initiatives in key reform states in India. Additionally, each of DRUM’s focus areas had its own sub-objectives: EXCHANGE VISITS WITH USDA RURAL UTILITY SERVICE (2004 to September 30, 2011): 7 Ibid. EVALUATION OF DRUM AND WENEXA 5 • Design alternative lending schemes for distribution entities (beginning with public investors such as the Rural Electrification Corporation (REC)), based on commercial lending principles rather than third-party government guarantees; • Engage Indian private credit rating agencies in credit worthiness assessments; • Develop of alternative financing mechanisms for distribution entities that include long-term debt, partial equity, grants and credit guarantees; • Develop term lending structures that include covenants (such as benchmark performance standards), pricing, and duration periods that match the life of distribution system assets; and, • Support the Accelerated Power Development and Reforms Program (APDRP) cell in the Ministry of Power to design and implement its distribution reform strategy; monitor APDRP’s commercial modernization projects for distribution entities in reforming states8 TRAINING (2004—Sept. 30, 2012): . The objective of the training programs is to build a cadre of distribution engineers and technicians by sharing experience on national and international best practices, and innovations in community based participative distribution business models. One of the unique characteristics of the program is its focus on building capacity at three levels: upper management, mid-level executives, and low-level staff, including field staff. PILOT PROJECTS (2004—April 30, 2011): The objective of the pilot projects was to demonstrate best commercial and technological practices that improve the quality and reliability of “last mile” power distribution in selected urban and rural distribution circles. The objectives of WENEXA2 (July, 2004-July, 2011) are to improve co-management of energy and water resources in the agriculture, urban and industrial sectors through enhanced power distribution and end-use efficiency, coupled with sound management practices. WENEXA and DRUM are complementary because improved energy-water management is, in large part, predicated on a healthy electricity distribution system. The three focus areas are meant to address both the mechanical and human aspects of distribution reform through technical upgrades and capacity building. 2.4 Project Implementers The U.S. Department of Agriculture’s Rural Utility Service (RUS) implemented the DRUM exchange program through a resource sharing agreement with USAID. CORE International carried out the DRUM training component. CORE was responsible for developing the training materials, consisting of 19 training courses targeting three levels: high, medium and low-level employees. The pilot projects were implemented by Tetra Tech, which was known as PA Government Services at the time of the DRUM contract signing. Tetra Tech provided both managerial and technical expertise to the three target utilities. WENEXA would be supported technically by Tetra Tech, which would devise a public/private/civil society scheme for replacing energy inefficient pumps with energy efficient ones. 8 The Accelerated Power Development and Reforms Program (APDRP) was replaced by the Restructured Accelerated Power Development and Reforms Program (RAPDRP). 6 EVALUATION OF DRUM AND WENEXA Additionally, there is a small water table renewal component carried out by NRDS (subcontracted to Tetra Tech) that consists of constructing a catchment dam to collect rainwater and allow it to replenish the water table, and a bundling and trenching exercise to also help replenish the water table. 2.5 Geography of Project Implementation The DRUM training component was nation-wide. The DRUM pilot projects were carried out in Doddaballapur, Bangalore; Umreth, Gujarat; and Aurangabad, Maharashtra. The DRUM exchange visits took place in Fort Morgan, Colorado (Morgan County Rural Electric Association; Fort Collins, Colorado (Poudre Valley Rural Electric Association); Akron, Colorado (Y-W Electric Association); and Holyoke, Colorado, (Highline Electric Association). Delegations from the U.S. utility cooperatives also visited Bangalore and Maharashtra. WENEXA is being carried out in Doddaballapur, Bangalore. 2.6 Cost The total cost of to USAID for DRUM was $13,619,306.43 (as of March 31, 2011), which is $3,430,958.57 less than the obligated amount of $17,050,265. The total contribution of the target utilities is $31.1 million (including loans from Ministry of Power’s finance arm, the Power Finance Corperation [PFC]), and the total contribution of the GOI is $3 million. The DRUM program ran from May 11, 2004 to March 31, 2011 (with the exception of the training program, which will end on September 30, 2012). The total cost of to USAID for WENEXA is $5,378,756.11 (as of March 31, 2011), out of a ceiling of $5,888,267. 2.7 Project Design CORE was responsible for developing the training materials, consisting of 19 training courses targeting three levels: high, medium and low-level employees. The target at inception was to train 25,000 utility staff throughout India. Initially, the design was for CORE to choose a number of private sector Indian training institutes to deliver the courses. USAID would provide a subsidy for each trainee and CORE would oversee the management of the training courses. As the course materials neared completion and a number of master trainers had been trained, the Ministry of Power sought to be more intimately involved with DRUM, a development welcomed by USAID. The agreement reached between USAID and the Ministry of Power was for the PFC to manage the training program and to pass through the USAID subsidy to the training institutions. Additionally, PFC would choose the 20 training institutions that would be involved in the DRUM training, a mix of public and private institutions. CORE would move its operations to the PFC and serve as technical advisor. However, there was disagreement between CORE and USAID regarding this new approach and CORE’s contract was terminated early, without bias and at the convenience of USAID. In absence of CORE, a DRUM secretariat was formed which acted as an interface between the Ministry of Power, USAID and PFC. The Secretariat was responsible for the day-to-day coordination and management of the training component while PFC was responsible for the financial reimbursements of the training institutes. The secretariat was supported by USAID, and reported to USAID and the Minestry of Power. The DRUM secretariat was also responsible for coordinating with the partner training institutes. Catchment Dam, Doddaballapur Photo: K. Warr EVALUATION OF DRUM AND WENEXA 7 The pilot projects were implemented by Tetra Tech, which provided both managerial and technical expertise to the three target utilities. A Detailed Project Report (DPR) was developed for each target site. The DPRs included detailed gap analyses, technological and process improvements, and the estimated cost of the upgrades. Tetra Tech also developed a set of best practices, or Key Performance Indicators (KPIs), which each target utility adapted for their own use. The KPIs measure electricity availability, reliability, cost, and a number of customer service parameters (see Annex 3). Tetra Tech also developed, with the target utilities, a standardized method to measure the KPIs. These KPIs were tracked monthly and published on the DRUM website (www.drumindia.org). The target utilities each secured loans from PFC (at 11.5% interest) and added their own funds. USAID provided a grant component through the PFC, and, in the case of MSEDCL, the Government of India also provided a small grant. For WENEXA, Tetra Tech subcontracted for the baseline study, a necessary component to determine the amount of energy saved with the new pumps. Establishing the baseline also required adding electricity meters to the pumps, a social challenge in itself, as farmers were suspicious that metering would result in them being charged for the electricity used. A competitive Request for Proposals (RFP) was put forward for an implementing partner who would interact with farmers using the energy inefficient water pumps, obtain their agreement for the old pumps to be replaced by new energy-efficient pumps (at no cost to the farmer), and then monitor and document the energy savings. Any energy savings above the baseline consumption of 2009 recorded under each pump set will result in a payment to Enzen (the Energy Service Company responsible for implementing the WENEXA II program) from BESCOM, thus eventually covering the cost of the new pumps and providing a profit for Enzen. Enzen will replace 604 old pumps and the project has a 10-year lifespan.9 Additionally, WENEXA 2 is implementing a watershed replenishment program pilot in one of the villages of Doddaballapur. The first involves construction of a catchment dam; the other involved bunding and trenching activities. 3.0 PURPOSE OF EVALUATION The evaluation of DRUM and WENEXA has three primary purposes: measuring the results compared to the initial aim, determining the impact of the programs, and providing a tool for helping USAID program future energy and water programs. To achieve these goals, the evaluation measures the projects’ effectiveness, relevance and efficiency using both qualitative and quantitative measures. This includes comparing the actual performance to the initial planned operation, defining the cost￾benefits for each program and providing the return on investment if the data support a defined analysis. Informing the USAID/India CLEEO team, stakeholders, GOI officials, and related experts on the project performance will assist in designing subsequent programs. Implementation of the evaluation team’s conclusions, recommendations and lessons learned will require the support of the 9 It is understood from BESCOM and Tetra Tech that though 46.67 percent of energy savings is quoted by Enzen, it is not strictly required by BESCOM as the basis for payment to Enzen. Payment will be made even if the saving is below the stipulated 46.67 percent. However the assumption is that the lower level of savings will affect Enzen’s return on the investment and therefore Enzen will have incentive for bringing higher levels of energy savings to make its investment viable. Farmers in Doddaballapur where water table replenishment pilot is taking place Photo: K. Warr 8 EVALUATION OF DRUM AND WENEXA senior management at USAID/India and HQ staff in developing the strategy for future projects in the energy and water sectors. For the detailed purpose of this evaluation see Annex 1. 4.0 RESEARCH DESIGN AND EVALAUTION METHODOLOGY In order to answer questions related to both outcomes and impacts (i.e., what happened and how did it affect the beneficiaries), the evaluation team gathered both quantitative and qualitative data. The team adopted a mixed-methods approach that would capture the necessary data. The DRUM￾WENEXA Evaluation Team used the following methods: • Primary and secondary source materials (project documents, policy papers, official records, and so on). • Key informant interviews using a semi-structured interview format, which allowed respondents latitude to fully capture the breadth and depth of their knowledge (see Annex 4). Among the key informants were: o Officials of USAID (present and past) with knowledge of the two projects; o Head of the DRUM Secretariat; o Officials at the Power Finance Corporation, the managers of the DRUM training program; o Management of CORE International in Washington, DC; o Tetra Tech management in Delhi; o Tetra Tech field managers in Doddaballapur, Bangalore; Umreth, Gujarat; and Aurangabad, Maharashtra; o Utility officials from BESCOM, MSEDCL, and MGVCL, both at the field sites and at headquarters; o Select officials from some of the training institutes (REMI, MDI, CENPEID); o Officials from ERDA, the Energy Resource and Development Association, who designed and implemented the WENEXA II baseline study; o Officials from Enzen, the Energy Service Company responsible for implementing the WENEXA II program; o A member of the Maharashtra Electricity Regulatory Commission, a champion of the DRUM reforms; o Officials from the Central Power Research Institute knowledgeable about DRUM’s roll in the development of a Smart Grid project; o Officials at the North Delhi Power Limited Company (NDPL) who carried out distribution reforms outside of the DRUM program; o Officials from TERI, The Energy and Resources Institute, as background on the Indian energy sector. • Internet Survey Analysis of past DRUM trainees (See Annex 5). This survey is, however, not scientifically valid for a number of reasons: o The sample size is too small. Out of 33,000 training participants, we were only able to obtain email addresses for 350 participants, 78 of which were not valid, leaving a survey population of 272. o The sample is not random as MDI is overrepresented10 o Many DRUM training participants do not have email addresses . o The number of survey responses (38) is not sufficient to make scientific conclusions. However, we have used the information provided by the participants as only one data point in drawing conclusions about the impact of the training course. Taken alone, the data are not scientifically valid, but as part of a data set, they are illustrative. • Personal interviews with farmers in their own villages to determine the impact of the DRUM reforms on their livelihoods. 10 MDI is overrepresented because MDI hosted the long-term “energy MBA” program, which means that their participants are likely to be in management positions after they complete the training and therefore are more likely to have email addresses and use email regularly. EVALUATION OF DRUM AND WENEXA 9 • Focus group with women11 • Site visits to all pilot projects. electricity users in Aurangabad, the urban pilot site. It should be noted that there are two outstanding data gaps. First, there is no available data from the utility subdivisions (i.e., the pilot projects) regarding profitability (or relative profitability). In other words, the subdivisions do not have in place a financial accounting system that allows them to systematically track revenues against expenses. At the outset, USAID requested that such a system be established, but the utilities felt that, as subdivisions of the larger utility, they could not institute such a system. The lack of this data makes it particularly difficult to address the second objective of DRUM: “to enable implementation of several full-scale commercially replicable distribution initiatives in key reform states in India.” Second, in order to determine the impact of the DRUM training program, more information is needed regarding the level of knowledge gained. In other words, number of people trained does not answer the question of whether people learned something from the training and were able to apply in their jobs. While each training course did have a post-training evaluation, it was focused on the perceived quality of the course, the instructors, and the facilities. There were no provisions in the design of the DRUM training component that would allow for systematic pre- and post-testing of major themes or key points, so there is no way to determine how much the participants learned beyond anecdotal evidence from the trainees themselves, as well as the utility management12. The parameters of this evaluation have been clearly laid out by USAID in the Scope of Work (SOW) for the evaluation (see Annex 1). We have chosen to assign letter grades to each of the first five focus areas of the evaluation (impact, relevance, effectiveness, efficiency, and sustainability) based on our findings. The letter grades are shorthand for achievement and, we hope, a useful tool for the reader. The grading system is as follows13 A = Excellent : B = Good C = Fair/Average D = Poor 5.0 FINDINGS 5.1 Process Findings 5.1.1 Effectiveness of management We have divided management effectiveness into three parts: USAID, Tetra Tech and PFC. USAID: Overall, USAID’s management of the DRUM program has been effective. Despite the fact that USAID management of the program has changed hands twice in the course of DRUM, the management of the program has gone relatively smoothly. However, some pilot sites appear to have received more USAID attention than others, notably MSEDCL, where USAID visited “almost every month from 2008 onward,” while MGVCL reported USAID visits only twice a year. The lack of site visits, however, does not seem to have affected the outcomes of the Umreth (MGVCL) pilot. What notable problems that exist are systemic in nature, namely the challenge of document management, especially when USAID managers change. This problem is exacerbated by the fact that 11 Women are major consumers of domestic electricity and therefore key beneficiaries of improved electricity. 12 It should be noted that other Indian training institutes, REMI in particular, routinely carry out pre- and post-testing of their trainees, as well as follow up six months and a year later, both with the trainee and his or her supervisor. REMI did not carry out this pre- and post￾testing protocol with the DRUM training because it was not part of the DRUM design. 13 A+=10, A-=9, A=8, B+=7, B=6, B-=5, C+=4, C=3, C-=2, D=1. B 10 EVALUATION OF DRUM AND WENEXA many documents are stored on staff members’ personal drives, or email files and may, or may not, be transferred to the new manager once the previous manager leaves. However, based on our experience, this is not unique to USAID/India and rather a challenge for USAID as a whole. There are notable examples where the lack of document management has affected this evaluation. First, the objectives of the USDA/Rural Utilities Service have changed from a focus on electricity project financing to a sharing of best practices in utility management particularly safety, and a demonstration of new technologies14. However, the evaluation team has found no documentation of this change in objectives. The problem of document management is even more highlighted with regard to WENEXA, which has experienced numerous changes in USAID management. Compared to DRUM, WENEXA was not given adequate attention by USAID. One of the reasons for this is that there was little continuity in project management since 2004. Document management related to WENEXA was also problematic due to changes in the team members. From 2004 to 2007, WENEXA was essentially not active, with intensive work starting in 2008. During the period 2004- 07 the Nagpur waste water treatment and reuse plant was completed with the partnership with Mahagenco and NMC. While WENEXA is both an energy and water project, it was managed from the energy office of USAID and, given the magnitude and complexity of DRUM (a purely energy project), WENEXA suffered from neglect for many years. Tetra Tech: Tetra Tech’s management of the DRUM and WENEXA projects has been excellent. The company has provided expert technical and managerial support to the pilot projects, and indeed, without such support, the pilots would not have been successful. From the establishment of the Detailed Project Reports (DPRs) to the identification of best practices for distribution utilities in the form of Key Performance Indicators (KPIs), to the creation of WENEXA’s innovative public/private/civil society model for improving water pumping efficiently, to providing management support for the pilot projects (both DRUM and WENEXA), Tetra Tech has demonstrated a high level of professionalism, technical competence, creativity, and managerial aptitude. Additionally, Tetra Tech’s information distribution and dissemination of ideas was notable, from the national DRUM results conference in February, 2011, to the DRUM documentary produced by Tetra Tech, to the website (www.drumindia.org) and the newsletter, “Drumbeats,” Tetra Tech did a excellent job of disseminating information. The team observed one key oversight, however. The Tetra Tech field representatives responsible for the three pilot projects never met together to share experiences or lessons learned. This would have been an easy and useful coordination tool that may have assisted the field representatives in implementing the project. While the Chief of Party made frequent visits to all the sites, occasional and planned meetings with the whole Tetra Tech team would have been useful. 14 The original objectives of the RUS exchange program are: • Design alternative lending schemes for distribution entities (beginning with public investors such as the Rural Electrification Corporation (REC)), based on commercial lending principles rather than third-party government guarantees; • Engage Indian private credit rating agencies in credit worthiness assessments; • Develop of alternative financing mechanisms for distribution entities that include long-term debt, partial equity, grants and credit guarantees; • Develop of term lending structures that include covenants (such as benchmark performance standards), pricing, and duration periods that match the life of distribution system assets; and, • Support the APDRP cell in the Ministry of Power to design and implement its distribution reform strategy; monitor APDRP’s commercial modernization projects for distribution entities in reforming states. Recommendation: Establish and consistently use a shared drive where each project stores important project documents (including pertinent email exchanges). A- EVALUATION OF DRUM AND WENEXA 11 Recommendation: Establish regular managerial meetings for projects with multiple sites and multiple managers. Additionally, there were significant delays in getting WENEXA 2 off the ground, though we realize the background work to successfully launch such an innovative scheme was time consuming, the lags in even beginning the work are notable, though lack of attention from USAID was likely a contributing factor. Power Finance Corporation (PFC) and DRUM Secretariat: The DRUM Secretariat and the PFC have done an admirable job of managing the training component. PFC has kept accurate records and dedicated adequate staff to manage the process, while the DRUM Secretariat has aptly managed the process, as well as kept the lines of communication open between USAID and the Ministry of Power. The one complaint we received from some training institutes was with regard to the length of time to receive payment once the institutes had submitted the (very detailed) invoices to the PFC. In some cases, it was reported that the lag time was up to six months. If it were within its purview as a government agency, PFC could improve on the time taken to pay contractors. 5.1.2 Coordination with GOI and stakeholders For DRUM, coordination between USAID, GOI (Ministry of Power and PFC), the training institutes, the target utilities, and Tetra Tech has been excellent. The establishment of the DRUM Secretariat and the regular meetings between USAID, the Ministry of Power/PFC, and TetraTech contributed to smooth coordination while also ensuring ownership of the DRUM program by the GOI. WENEXA’s coordination with government and other stakeholders is mixed. The project established effective coordination with the Karnataka State Government, the Electricity Regulatory Commission, and BESCOM. It also has established good cooperation and coordination with central government agencies like MOP, PFC, and the Central Power Research Institute. While Tetra Tech has contacts with many agencies in Karnataka and other states, these contacts are all related to electricity. Coordination and working with other stakeholders at the local level, especially those related to water and agriculture is lacking. Recently, Tetra Tech has had some interaction with the National Bank for Agriculture and Rural Development (NABARD) in connection with financing the pump set replacement program, yet it has not coordinated with the Ground Water Department in the area, even though sustainable ground water management is one of the key aspects of the project15 At the village level, Gram Panchayats are responsible for local governance and Karnataka is known for good local governance. However, this important peoples’ organization has not been utilized. Had there been coordination with the Gram Panchayat, the project might have been implemented sooner, . Similarly, the project has not coordinated with the state Agriculture department, which is involved in watershed management and agriculture works. 15 The Ground Water Department is the key agency for regulation of ground water under the Karnataka Ground Water Regulation Act. A￾A C+ Recommendation: To promote policy change in the water sector, develop effective coordination with state and local agencies responsible for ground water and watershed management, and agriculture development. 12 EVALUATION OF DRUM AND WENEXA in terms of mobilizing people in the area for watershed works and pumps replacement16. It is also important that the project develops special interest groups such as Water Users Committees, Watershed Committees, Ground Water Monitoring and Management Committee, and involve them in the project management. 5.1.3 Ability to strengthen public/private/civil society linkages Although one of the key reasons for reforming the electricity distribution system in India is not only to improve the quality and reliability of electricity in India, but also to make the sector more attractive to private investment. There is, however, little evidence of an increase in private sector participation at the pilot sites as a result of DRUM. There has been no movement toward financial accounting (at least at the subdivision level) that would take into consideration profit and loss for each pilot project, thus making “commercial viability” difficult to ascertain. While it is true that MSEDCL Unit 1 (Aurangabad) is being privatized though a franchise agreement with the telecommunications company, GTL, the franchising was a precondition of the MOU between USAID and MSEDCL, not a direct consequent of the DRUM upgrades. However, as a precondition, it is a consequence of DRUM (without DRUM it may not have happened) and the only example of private sector participation in the pilot sites. Where WENEXA is concerned, the story is quite different. There is strong evidence of public￾private-civil society linkage and indeed, the nature of the pump replacement program is partnership oriented. BESCOM, a state-owned distribution utility, is partnering with Enzen, a private Energy Services Company (ESCO), to reduce electricity losses through the replacement of energy inefficient pumps with energy efficient ones. Enzen, in turn, must interact closely with each and every farmer whose pump is to be replaced in order to explain the program to him and get his approval to change out the pumps. In almost all the cases, this is a difficult challenge as the farmers are reluctant to give up their pumps that are working well (though inefficiently) for untested new pumps. In order to help raise awareness about the program and to gain the farmers’ concurrence, Tetra Tech hired a local organization to create and execute a number of street 16 The project aims for replacement of about 600 pumps with energy efficient pumps thus saving energy and water use. As of March 11 2011, energy audit has been completed for only 460 pumps. Of these, 35 farmers did not agree to have their inefficient pumps replaced, even after the completion of the energy audit of the pumps. Additionally, about 43 pumps dried up making it difficult to replace the pumps. There are still other issues: some farmers re-bored their tube wells due to the depletion in ground water; some have abandoned the wells and shifted the pump set to a new bore well as the previous one had dried up; some of the bore wells are deeper than reported during baseline (thus require more energy to pump the water), and others have technical problems. C A+ Recommendation: Whenever possible, utilize local government, such as the Gram Panchayats. Recommendation: Build civil society organizations where appropriate, such as pumpset -farmers associations (for participatory ground water regulation and management), watersheds committees (for watersheds works), and an apex citizen body (as a multi-stakeholder platform). EVALUATION OF DRUM AND WENEXA 13 plays about the importance of efficient use of energy in agriculture. These street plays were widely attended and considered quite successful. 5.2 Outcome and Impact: DRUM 5.2.1 Capacity Building A major component of DRUM was dedicated to building capacity to carry out the reforms, and in this area, DRUM shined. The DRUM training program is well on track to meeting its goal of training 35,000 people by September 2012, and as of January 2011, has provided short-term (three-five days) training for 33,135 persons and long-term (one year) executive management training to 150 persons as well as a comprehensive distance-learning program to 1,589 persons. DRUM provided training for three levels of electricity distribution utility employees: upper management, middle management and linemen (and women). In addition to training more than 33,000 electricity sector employees, training now is considered essential among the utilities surveyed for this evaluation in both technical and non-technical areas (such as change management). DRUM has succeeded in establishing a culture of training at least within the target utilities, and, based on the popularity of the DRUM training, also likely within other distribution utilities throughout the country. Moreover, the adoption of the DRUM training program by the Government of India for its RAPDRP program is a success. In a survey of 38 past trainees, 94.5 percent of the respondents felt the quality of the training was good or excellent, and nearly one-third use what they’ve learned in the DRUM training “all the time,” while another third use what they learned “from time-to-time17” (see Annex 5). There is ample evidence, both anecdotal and concrete, from the management of the target utilities that the training is valued. One of the most telling indicators of the training program’s value is the utilities’ willingness to continue to send their employees for DRUM training even after USAID support stops. Additionally, some utilities, as well as some of the training institutes, have either adopted or adapted the DRUM training materials for their own use. Capacity development is not only about formal training. The USDA/RUS exchange visits were universally praised and each person interviewed was able to note specific things he learned from the visit and subsequently implemented at his utility. There was widespread praise for learning about improved safety procedures as well as the use of monopole transformers. As a result of the exchange program, U.S. safety manuals have been translated into three local languages. One indicator of the value placed on the exchange visits by BESCOM and MSEDCL is the fact that the utilities now pay for the exchange visits themselves. USAID pays RUS only for U.S. utility officials to visit India and for RUS staff time in organizing the exchange visits. As successful as the DRUM training component has been, there are gaps in the design of the program that, in retrospect, would have strengthened the program. Specifically, there is no requirement for pre- and post-testing of training participants and therefore, there is no scientifically valid way to evaluate the level of knowledge gained. The design of DRUM was perhaps overly focused on outputs (number of people trained) and not enough focused on outcomes (what did the people learn and how did they use it in their jobs). In addition to the pre- and post-tests, the DRUM training component could have been designed to also follow up with a random sample of participants (via telephone, as many do not have email addresses) six months and one year after the training. Additionally, a mechanism to follow up with the trainees’ managers would have been useful. By gauging the level of learning, courses could have been modified (if needed) to meet the needs of the trainees. This oversight accounts for the A- rating, rather than an A rating. 17 DRUM training effectiveness survey, March 2011. A￾Recommendation: In designing future training programs, establish a mechanism for determining level of learning, as well as a mechanism for follow-up regarding effectiveness. 14 EVALUATION OF DRUM AND WENEXA Along the same lines, the design of the DRUM training program did not account for changes in technology. That is to say, technological advances quickly made some of the technical portions of the DRUM training program obsolete. It is unclear whether the lack of a formal updating mechanism is a result of CORE International’s premature exit from the program or whether such a mechanism was never a part of the design. Nevertheless, the training materials would have been more effective if there had been an appropriate contractual mechanism to periodically update the materials that were technically and operationally outdated after five years. CORE could have done this if their contract had not been completed early, or it could have been added to Tetra Tech’s scope of work. It should also be noted that there has not been a concerted effort to train women who are employed in supportive rolls, with the possible exception of the female employees of NDPL. For WENEXA, there has been no specific training related to Agriculture Demand Side Management. Tetra Tech has undertaken a number of capacity building efforts for BESCOM and others during the implementation of WENEXA, but many of programs were in the form of presentations and discussions with top BESCOM management to get their consent for the pump set replacement program. Similar meetings were conducted with Energy Service Companies (ESCOs) and Enzen. In addition, Tetra Tech provided training for BESCOM staff on the provisions of the agreement signed between Enzen and BESCOM, as well as instruction on how to generate the energy saving invoices when the program starts. Farmers were also included in the WENEXA capacity building process through classroom training on topics of efficient use of water and electricity, crop rotation, better cultivation practices, water saving techniques like drip irrigation, and so on. Moreover about 100 street plays were conducted in the area to educate the farmers about the importance of energy savings through pump replacement and drip irrigation. During the field visit, the evaluation team noted farmers’ positive comments about these programs Recommendation: In designing future multi-year training programs, establish a mechanism for periodic updates. C+ Recommendation: Develop specific programs for capacity building of farmers and other stakeholders for the energy efficient pump set replacement program. Recommendation: Develop specific and detailed training, possibly under DRUM, for Agriculture Demand Side Management, which will be useful not only for BESCOM, but other distribution utilities with a heavy agriculture base. EVALUATION OF DRUM AND WENEXA 15 DRUM TRAINING AND PRIVATE SECTOR SUCCESS: A POSITIVE RETURN ON INVESTMENT Sector reform training is most likely to have the greatest impact when a company is committed to reform from the highest levels. An example of the possible impact of the DRUM training program can be found in North Delhi Power, Ltd. (NDPL), which is a private distribution company in New Delhi and an original target utility for DRUM. Though they chose to undertake the technical reforms on their own, they made good use of the DRUM training for both technical and non-technical capacity building for reform. The move to a Public-Private enterprise or joint venture of TATA power and the Delhi Government in 2003 was designed to illustrate the positive prospects for change. NDPL was a failed electric utility when acquired by TATA Power, part of the largest investment company in India. During the past five years there has been a strong growth of industrial and commercial development in the region, particularly shopping centers. At acquisition of NDPL the utility had frequent power outages, particularly during the monsoon season, poor consumer relations, lack of maintenance, obsolete equipment, limited training, lethargic employees, transmission and distribution losses of 53% [32% due to theft] and was considered bankrupt. Today, implementing the TATA approach at the utility has changed the equilibrium and it now provides continued service to all consumers, replaced all mechanical meters with smart digital meters, had a positive approach to consumer interactions, expanded the industrial and consumer sector to 65% of demand and reduced T&D losses to 13%, particularly focusing on eliminating theft to below 5%. The Delhi Electric Regulatory Commission in 2010 noted the tremendous effort over the past eight years in improving the financial health of the utility. But it also indicated the concern of the current liquidity position since current tariffs do not reflect the maintenance and operating costs, or provide for expansion of equipment and a insure positive return on the investment by TATA. Training has been an essential component that encouraged changes at NDPL. The DRUM training program is based at the CENPEID facility (Center for Power Efficiency in Distribution) that was opened in Jan. 2005, where 11 trainers provide a wide range of courses for the employees. The DRUM program has provided training on improved safety, installation of smart meters, and expanding consumer interaction. Advanced financing and billing courses were important to modernize customer payments and had the added benefit of increasing training of women, since about 1/3 of the finance staff at NDPL are women (this is markedly different from the DRUM pilot utilities, where very few women are employed and very few have been trained). The DRUM training helped instill a culture of training, now requiring eight days per employee per year, and helped move NDPL into the world-class utility category. The NDPL strategy includes plans for CENPEID to be considered an independent profit center as part of the 2011 operating plan and new NDPL operations and will maintain training courses on a self-sustaining basis. Although it can be difficult to define the exact impact of the DRUM program on these dramatic improvements at NDPL, discussions with the management indicated that this program provided a catalyst for the positive financial and technical move with each employee completing a course providing additional positive resources for the utility. This can be considered both a qualitative and quantitative success for the DRUM program. 16 EVALUATION OF DRUM AND WENEXA 5.2.2 Pilot Projects 5.2.2.1 BESCOM BESCOM is the electricity distribution utility in Bangalore and its Doddaballapur subdivision is one of the pilot sites. Created in 2003 under the Indian Electricity Act, BESCOM is the largest distribution company in the state of Karnataka . Ten percent of BESCOM’s customers are farmers and 78 percent are domestic consumers. Doddaballapur is a rural site serving 281 villages and about 30 percent of Doddaballapur’s total power load is connected to agriculture, specifically to about 12,000 water pump sets. Hence the site was also chosen for the WENEXA program, which addresses the issues related to energy and water savings in agriculture. One of the major achievements of the DRUM project under BESCOM in Doddaballapur in Karnataka is the technical upgrade of the distribution system, which has improved the reliability and quality of power. Provision of a High Voltage Distribution System (HVDS) and refurbishment of feeders with transformers and meters, provision of fault passage indicators, switch capacitors and polymeric surge arresters are important technical upgrades that have improved tail-end supply and reduced interruptions in electricity supply. As a result of these improvements in technology, the Aggregate Technical and Commercial (AT&C) losses have been greatly reduced from 38.9 percent in 2005 to 12.8 percent by the end of 201018. Similarly the transformer failure rate has reduced from 7.3 percent to 0.95 percent. Billing efficiency has improved from about 60 percent to 100 percent, and collection rates have improved thanks to the installation of new static meters in the place of old mechanical meters. According to the analysis by Dhiya consultancy, the Tetra Tech subcontractor managing the Doddaballapur project, the subdivision has seen profits of 14- 17% since the DRUM reforms. BESCOM was able to make profit in past seven out of eight years19. Under DRUM, agriculture feeders were separated from non-agricultural feeders and given three-phase supply, which has improved reliability of electricity supply to agriculture. All the farmers (14) interviewed during the field visit expressed satisfaction with the reliability of the electricity supply and noted significant improvement in the water supply to their farms. This has had positive income effects, with 18“Experience and learning from DRUM –Doddaballapur-“ Presentation by BESCOM in the DRUM conference on Feb,28,2011 19 According to Mr. Vasuki Dkiya, of Dhiya consultancy, in 7 of the last 8 years, BESCOM made a profit (interview, March 8, 2011). Capacitor, Doddaballapur Photo: K. Warr Part of the DRUM team, Doddaballapur Photo: K. Warr B EVALUATION OF DRUM AND WENEXA 17 some farmers reporting increases in their income of 15-20 percent20. The technical improvements have also greatly reduced burning of water pumps due to fluctuations in the electricity. Farmers also reported that high-capacity pumps that were not working earlier when there were fluctuations in the supply, are now working well. Moreover, better quality and access to electricity has reduced tensions among the farmers. Prior to the DRUM upgrades, all the pumps connected to a single transformer could not be used at the same time because the inferior quality of power would not allow them all to run at the same time. This required many of the pumps to be switched off, thus causing conflict among the farmers as to who would switch his pump off. Additionally, some of the more influential farmers used hooking (i.e. illegal connections) to get electricity for their pumps, which also resulted in other pumps having to shut down. After the DRUM reforms, each pump has a separate transformer and the problem has been solved. Now the practice of farmers tapping (i.e. illegally connecting) single-phase supply meant for non-agricultural purposes has almost stopped and as a result, the quality of the electricity supply for domestic use has also improved. The DRUM training was an important component to the Doddaballapur pilot, and all the subdivision employees have undergone training associated with change management and absorption of new technologies21. They were trained in both technical areas and customer satisfaction aspects. It is noted that each assistant engineer has undergone four trainings in the subdivision. Additionally, there have been significant improvements in customer care. There is now a 24- hour customer care centre operated in the subdivision thanks to the DRUM upgrades. Additionally, the building has been renovated and now includes a customer-friendly center where customers can come inside of the building to pay their bill, as opposed to other areas where they have to line up outside in the sun. Additionally, the customer service center now is able to attend to consumers’ complaints by phone. However, in spite of the upgrades to customer service and the improvement in quality and availability of electricity supply, it is surprising that the consumer satisfaction survey undertaken in BESCOM by an independent agency revealed that the overall level of satisfaction with BESCOM, compared to other utilities, is low22. The percentage of customers satisfied with BESCOM has fallen from 37 percent in 2005 to 14 percent in 2010. One reason for this may be that customers’ expectations rise in step with the quality of service. That is to say, with better service, their expectations also rise, so today an hour of load shedding may be cause for dissatisfaction, whereas in 2005, an hour might have been considered acceptable, or at least better than three hours. Similarly there is a decline in satisfaction in terms of availability of power in both the domestic and agriculture sectors. Not surprisingly, the survey indicates that erratic power cuts are the most dissatisfying factor in the agricultural sector. Satisfying agricultural electricity customers depends upon timely and adequate water supply, which is a function of the quality of electricity supply. Doddaballapur is an area where water is severely over-exploited, requiring more and reliable electricity for irrigation. A slight disturbance in the electricity supply might affect timely availability of water for crop growth (and farmers’ satisfaction). Though there has been overall improvement in the electricity supply in the agriculture sector, the dependence of farmers on electricity supply has also increased. The new, higher level of expectation for quality electricity might be the reason for farmers’ lower level of satisfaction. 20 The income impact is based on farmers perception and requires evidence based on a scientific sample survey in the areas where agriculturists are major consumers of electricity. 21 All three pilot project sites received change management and new technology absorption training under the program. All of the employees from BESCOM and MSEDCL and nearly ¾ of the MGVCL employees received this specialized training. 22 Report of the Consumer Satisfaction Survey by GFK Customs Research,Feb,2011 18 EVALUATION OF DRUM AND WENEXA These results, though not yet finalized, require careful review especially in the agriculture sector since our report reveals that DRUM has contributed to a significant improvement in rural electric supply in Doddaballapur, which has also improved income and livelihoods. 5.2.2.2 MSDECL Aurangabad, one of MSEDCL’s 179 divisions, has 95,208 customers covering an area of 92.5 square kilometers (36 square miles). Its DRUM technical upgrades included: • Network refurbishment, including installation of capacitor banks, which improve voltage, reduce line losses, and increase line efficiency. Total cost for the network refurbishment is INR272.1 million, or $6 million. • Construction of new sub-stations and refurbishment of old, which improves the system reliability and availability, while allowing for system growth. The total cost of the substation component was about INR39.4 million, or $800,000. • Provision of Land for the new substations, which was a difficult and costly project, as land in Aurangabad, an urban center, is limited and the “not in my backyard” syndrome was widespread. The total cost of the land for the substation is INR137.5 million, or $3 million. • Replacement of 48,000 (target 52,000) single-phase household and commercial meters and 2,500 three-phase for large industrial customers. The new meters are static electronic meters (rather Replacement meter in Aurangabad Photo K. Warr Figure 2: Utility customer satisfaction scores in Doddaballapur across segments B EVALUATION OF DRUM AND WENEXA 19 than the old electromagnetic meters which are less accurate and prone to theft). The new meters were placed in tamper proof boxes. The cost of the meter replacement was $3.9 million, or INR174.4 million. As a result of the installation of the new meters, system losses were reduced from 29.9% to 18.41%, with a monthly savings of 93,951 kWh and a monthly savings of nearly INR 403,989 or $9,000. With these savings, the payback time is estimated to be about 11 months. • Breakthrough Projects: These are special projects that are meant to demonstrate new technology. They include: IT infrastructure development, installation of polymeric surge protectors, installation of fault passage indicators (which allow linemen to tell exactly where the fault has occurred, thus reducing the time to fix it from several hours to several minutes), initiation of load survey and quality of power indicators. There were two breakthrough projects that were cancelled by MSEDCL, namely the installation of prepaid meters and an Automatic Meter Reading (AMR) system that would allow remote meter reading. The reason given for cancelling these two projects is that the new franchisee, GTL, would undertake these projects. The total cost for the breakthrough projects was INR114.4 million, or $2.5 million. • Installation of a High Voltage Distribution System (HVDS), which reduces electricity theft (illegal hook-ups do not work with high voltage systems as there is no transformer to bring the voltage down to usable levels), reduces system faults, provides for easy upgrades in the system, improved reliability (because only a few consumers are connected to a transformer so when there is an outage in one area, the others aren’t affected), and increases system capacity since electricity losses are reduced significantly. The cost of this upgrade was INR208.2 million, or $4.6 million. One key informant noted that this component was the “greatest achievement of DRUM.” In terms of customer service improvements, the Aurangabad Unit 1 subdivision purchased a hydraulic lift van, which reduces maintenance time and an equipment van to address outages in a timelier manner. They also installed customer feedback boxes in all consumer payment points. Furthermore, staff has received customer service training. In addition to the 173 linemen trained under DRUM, MSEDCL has used DRUM training materials to train an additional 1708 through MSEDCL’s own training program. As a result, the accident rate of staff has been reduced to zero (though we were unable to obtain the accident rate prior to DRUM). Part of the DRUM project was the establishment of a number of key performance indicators (KPIs), and key key performance indicators (KKPIs). The careful tracking of these indicators throughout the project allows us to determine the outcomes of the upgrades. MSEDCL chose to report on 15 KKPIs, divided into sections regarding availability of electricity, the reliability of the electricity, the cost in terms of electricity losses, and customer service parameters. The Ministry of power purposefully set ambitious targets for the pilot projects, and while many of these targets were not achieved, there is clear improvement in the KKPIs Availability Reliability SAIDI System Average Interruption Duration index, measured in hours SAIFI System Average Interruption Frequency Index, measured in numbers Transformer Failures Measured as a percentage CAIDI Customer Average Interruption Duration Index, measured in minutes per occasion Overhead Measured in faults per 100 circuit kilometers DRUM training at MSEDCL training center, Aurangabad Photo: K. Warr 20 EVALUATION OF DRUM AND WENEXA Table 1: Key Performance Indicator targets for Aurangabad AURANGABAD URBAN DIVISION 1 Targets of KKPI Sr No Parameter Unit of Measurement Target Availability 1 SAIDI Hours 2 2 TXR Failures Percentage nil 3 CAIDI Minutes per Occasion 90 4 Overhead Line failure rate Faults per 100 ckt-km of 11 kV Overhead Line 2 Reliability 5 SAIFI Nos 1.3 Cost 6 End to End Efficiency Percentage 7 AT&C Losses Percentage 8 8 Return on Capital Equity (ROCE) Percentage 10 9 O&M (Revenue Expenses) per unit Energy Input. Paise per Unit 10 Customer Service 10 Customer Satisfaction Index Percentage 100 11 No of voltage complaints received during the month Nos 0 12 % of New Service Conn. Effected within 3 days Percentage 100 13 % of Billing complaints resolved within Regulatory time limits Percentage 100 14 % of Supply Complaints resoved within 2 hours Percentage 100 15 % of employees having being trained on ytd basis Percentage 100 Line Failure EVALUATION OF DRUM AND WENEXA 21 In terms of the availability of power, customers at the beginning of the project (October, 2005) experienced 29 hours of power interruption per month. By the middle of 2010, customers were experiencing less than four hours of power interruption per month. Transformer failures have decreased from more than 13 percent in October 2005, to 1.5 percent in 2010. The average duration for power outages per month fell from 50 minutes per outage in 2009 to 31 minutes per outage by mid-2010. In October 2005, customers in Aurangabad Unit 1 experienced 74 power interruptions per customer per month. By June 2010, customers experienced only 7 power interruptions per customer per month. Aggregate Technical and Commercial (AT&C) losses were reduced from 29 percent to 15 percent. In terms of customer service parameters, the percentage of new connections effected within three days has increased, but only slightly, still at a relatively low level of 17 percent. However, the percentage of billing complaints resolved within the limits set by the state energy regulatory body has increased from a little over half to almost three-quarters. The percentage of supply complaints resolved within two hours has only slightly decreased from 2009, when such figures began to be recorded. A third-party assessment of customer satisfaction in Aurangabad, carried out by GkK indicates customer satisfaction levels have markedly improved since the initiation of DRUM. In 2005 only 49 percent of all customers (domestic, agricultural, commercial) were satisfied with their electricity supply. Today, that figure is 82 percent. As noted below, domestic customers are much more satisfied with the utility’s performance than they were in 2005. Women electricity customers with Dr. MeenuMishra Photo: M. Mishra Electricity Bill with Photo of Meter Reading, Aurangabad Photo: K. Warr 22 EVALUATION OF DRUM AND WENEXA Compared to other utilities, satisfaction with MSEDCL is second only to the water supply utility, which could indicate that Aurangabad’s improved customer care measures have had a significant positive effect. Figure 4: Customer satisfaction on key parameters, 2005 and 2010. Figure 3: Electricity supply customer satisfaction, 2005 and 2010 EVALUATION OF DRUM AND WENEXA 23 The advances made by the utility as a result of DRUM are impressive. In terms of impact, there are a number of beneficiaries. Customers are better off now because they have more electricity and more reliable electricity supply. Additionally, customers are satisfied that their meters are read accurately on a regular basis23. The Aurangabad Unit 1 subdivision is better off having reduced their AT&C losses and improved their overall system. In terms of the larger utility, any measurable impact remains to be seen. There was no systematic way to share best practices or lessons learned from the Aurangabad pilot with the larger utility, though some have claimed the lessons learned “trickled down”. Moreover, there are no specific plans to replicate the DRUM reforms, though there are plans under the government’s urban electricity reform program (RAPDRP) to undertake similar system upgrades. However, as a complete distribution reform initiative, there are no plans to replicate DRUM in MSEDCL. While DRUM’s achievements and impact to the consumer and the utility in Aurangabad Unit 1 are impressive, DRUM has not established “a framework, institutional capacity, and project development functions at the central and state level,” nor has it implemented “several full-scale, commercially replicable distribution initiatives in key reform states.” The stated objectives have not been met, yet the impact has been quite positive. 5.2.2.3 MGVCL The third pilot site selected by DRUM is in the Umreth subdivision of the Madhya Gujarat Vil Company Ltd. [MGVCL] electric utility located in Gujarat State. The MGVCL has 1.8 million customers and the Umreth subdivision had 162,428 customers, mostly rural farmers. The site successfully completed technologically innovative projects, including: installing fault passage indicators, innovative surge arrestors, line capacitors, and remote monitoring of reliability indicators. The results were positive as transformer failures dropped from 12 per month in 2005 to three per month in 2010, and system losses (AT&C) and unaccounted for losses or theft dropped from 40 percent in 2005 to 18 percent in 2010, the period when the DRUM program was active. The increased confidence in the supply and efficiency of electricity has had a positive impact on the Umreth community. Some farmers have expanded their production, diversified their crops and increased output claiming a 15 to 25 percent increase in income. Individuals interviewed in the area farms, and the customer care centre, all had positive experiences with the electric system over the past several years, coinciding with the DRUM project. The technical improvements have reduced fluctuations in electricity, which allowed agricultural water pumps to operate longer without failure. Fault passage indicators (FPI) were installed on the 11 KV feeders to help locate a fault quickly and reduce the duration of interruptions. 821 FPI’s were installed on all 11 KV feeders at this DRUM site. This provided significantly improved reliability indices as indicated by SAIDI and CAIDI (See Annex 3). 23 Nearly 100 percent of Aurangabad customers indicate their meters are now read on a monthly basis, versus 23 percent in 2005. Also, 82 percent believe the meter readings are accurate, compared to 69 percent on 2005. Fault Passage Indicators, Umreth Photo: K. Warr B 24 EVALUATION OF DRUM AND WENEXA A GIS system was also installed to gain experience in the planning and management of a rural distribution network. At the Umreth, eight rural feeders and two urban feeders were covered. Improvements by the GIS helped identify low voltage areas, reconfigure the high￾tension links and reduce losses, and select optimum placement of the high-tension shunt capacitors. The challenges to this operation included working in the rural areas, which are more difficult than the urban networks. Keeping the system documented and updated is essential to maintain reliable billing. Motivating the field engineers to use and maintain the GIS on a continuous basis was a challenge and perhaps more training on this component would be required. The new customer care facility was built to provide a comfortable environment for electricity consumers to pay their bills. Instead of the older structure that required individuals to wait outdoors in the sun, the new center is designed to be a friendly site, with a covered indoor seating area, fans and positive information on electric power. Consumer complaints on the invoice or service are handled directly at this site. Payments are accepted by a 24-hour a day APT (Automatic Payment Terminal) machine that accepts cash or credit instruments and is used by around 4000 consumers per month. It appears the APT installation was funded by USAID according to the USAID stickers, although seven other APT payment machines are located in the MGVCL networks that are not. Additionally, a photo of each meter reading is provided to the consumer every four months to eliminate inaccuracies and reduce staff collusion. One unique aspect of the Umreth pilot site was the instillation of a 15 kW solar PV grid power plant on the roof of the costumer care center. There is a real-time screen indicating the solar electricity provided by the panels on a daily, monthly and yearly basis. The panels are adjusted seasonally for optimization to suit the moving angle of the sun. The Solar PV panels are visible as one approaches the facility and provide a positive display of the potential for solar power. The DC power from the solar panel is tapped by an inverter to supply the office lighting so the generation is not wasted in case of a grid failure. There is a seamless switch to grid power in case of an absence of solar power. This project is designed to show the operations of a grid connected and load connected solar plant at a conspicuous site and sensitize the public to the potential of this renewable technology. However it has not been replicated in the area and the success has been lost on the public since it is displayed as a commercial appendage instead of an item that can be placed at a residence. In addition, a display of solar panels used for pumping water could provide a more stimulating option for off-grid requirements. A limited pre-paid meter pilot was initiated at this site to manage revenue and reduce non-payment of bills. The results Each electricity distribution pole must have a unique number for the GIS system to function. Photo: K. Warr Solar Energy Pilot, Umreth Photo: K. Warr Automatic Payment Terminal in Umreth Photo: K. Warr EVALUATION OF DRUM AND WENEXA 25 were mixed (only 150 meters have been installed), and acceptance by consumers of this new approach is crucial for this payment operation to succeed. One problem is the fuel cost adjustment, and concomitant tariff changes, which are charged retroactively, are difficult for consumers to accept, believing they had already paid for the service. Perhaps better publicity and support could improve acceptance of this approach. As indicated in Figure 5, the customer satisfaction survey findings showed a move from 52 percent in 2005 to 81 percent consumer satisfaction in 2010, a significant improvement in overall consumer satisfaction. Figure 5: Overall satisfaction with electricity supply, 2005 and 2010 26 EVALUATION OF DRUM AND WENEXA Figure 6, provides a comparison of electricity with other utilities (water, telephone, rail, post) and indicates that consumers are most satisfied with the electricity utility service in 2010, compared with 2005. The feedback from the participants in the training program indicated that the behavioral training helped the participants improve their interpersonal skills, expand confidence at work, encouraged creative thinking, as they served consumers with more patience, courtesy and politeness. To review, the objectives of DRUM are: • To establish the framework, institutional capacity, and project development functions at the central and state level; and • To enable implementation of several full-scale, commercially replicable distribution initiatives in key reform states in India. It is against these objectives that we have assessed the outcomes and impact of DRUM. However, this poses a quandary—the evaluation team found that the pilot projects did not achieve the stated objectives, yet the impact of the projects was uniformly positive. People are better off as a result of the DRUM upgrades, the target utility subdivisions are functioning better, and the capacity of the staff of the subdivisions has been improved. Yet, in none of the pilots have established “a framework, institutional capacity, and project development functions at the central and state level”. While the framework is sound (development of the DPRs, establishment and measurement of the KPIs) there is little evidence that this systematic approach is being replicated. In the case of BESCOM, engineers in four other subdivisions of the utility were trained in tracking KPIs with a view toward expanding DRUM into these other subdivisions; however, given that there were no technological upgrades in the other four sites, nor a strengthening of their accounting systems, the data and results from the four “shadow” sites are not strictly comparable with the DRUM pilot sites. While the exercise has contributed to an understanding of the key performance indicators in the four shadow utilities, the DRUM system of reform has not been replicated. Certainly electricity project development capacity exists at the state and national level (and PFC is a large part of that), and new projects are being developed constantly, but there is no evidence that alternative financing schemes Figure 6: Customer satisfaction across utilities, 2005 and 2010 EVALUATION OF DRUM AND WENEXA 27 have been put into place as a result of DRUM. The apparent abandonment of the alternative financing part of the project is perplexing. In terms of the institutional capacity to carry out similar reforms to DRUM, that remains to be seen. No doubt the managers from the pilot sites benefitted from the DRUM experience as well as the hands-on project management training from Tetra Tech, but there is little evidence that these project management practices have been systematically shared with the larger utility (with the possible exception of the BESCOM shadow sites). To that extent, the “implementation of several full-scale, commercially replicable distribution initiatives in key reform states,” seems unlikely in the near term. DRUM was costly and without financial accounting practices that would allow for the analysis of profits against losses, the commercial viability of a DRUM-like initiative is uncertain. At the end of the day, as successful as the DRUM pilots were in terms of benefitting both consumers and the pilot utility, they have not been replicated anywhere, nor is there any plan to do so. This has affected our final scores for this section. What were in all cases “excellent” have been downgraded to “good” because the stated objectives were not met. There are two key reasons the objectives have not been fully met. First, they may have been overly ambitious. A careful reassessment mid-way through the project and a formal revision of the objectives may have yielded a more positive outcome in terms of meeting the project’s objectives. Second, there has been a lack of strategic dissemination of lessons learned. This is not to take away from DRUM’s excellent website, which is frequently visited,24 or its top-notch newsletters. However, the audience for these media are often not at the decision making level. For reform to happen, it needs to come both from the top and the bottom. What is lacking in the pilot project is a communications strategy aimed at convincing the top management. For instance, the meeting to roll out the DRUM results on February 28, 2011 was an excellent example of how to reach key decision makers and share lessons learned in a CEO-friendly atmosphere. Had this type of gathering been a yearly event, hosted by the Ministry of Power, DRUM-like pilots may have gained more traction at the top. At the end of DRUM, the locus for change or innovation remains largely external to the utility, and, to be fair, this is common for state-owned enterprises. Fostering change from within requires buy-in at the highest levels. Rural Utility Service Exchange Program The initial DRUM program design for component 1 was to provide a national strategy and alternative financing for electric utilities utilizing the experience of the Rural Utility Service (RUS) of the U.S. Department of Agriculture. This component was to provide technical assistance based on the U.S. rural electric utility experience to plan, design, finance and implement distribution reform and rural electrification programs. However, the Indian counterpart, REC, stopped participating in this program in 2006 and the focus of this component shifted from finance to best 24 To date, www.drumindia.org has been visited 565,337 times, with 13,363 in February, 2011. Lesson Learned: Large, multi-year projects warrant careful midterm reviews and a frank assessment of the program’s stated objectives. Lesson Learned: Demonstration projects require (i) comprehensive communication strategies that target decision makers as well as managers, and (ii) periodic assessments of the strategies to determine their impact. A 28 EVALUATION OF DRUM AND WENEXA practice training through a partnership between Indian electric distribution companies and U.S. electric cooperative groups. This activity included exchange programs, study trips, workshops, expert discussions and training programs with positive participation from both sides. This was considered an effective program as Indian managers, engineers, and linesman were trained on the best practices at U.S. electric cooperatives, and experts from the U.S. visited MSEDCL and BESCOM to exchange ideas and working procedures and to share experiences. The participants from India considered their experience positive and implemented the U.S. ideas and operational procedures upon return to their home utility. 5.3 Outcome and Impact: WENEXA For WENEXA, it is too early to determine outcome or impact. The pump replacement program is just starting and the energy savings scheme between Enzen and BESCOM has not begun yet. Moreover, this scheme has a ten year life span. In terms of the water table management component, again, it is too early to determine impact, and without any baseline study, it will remain impossible to measure the impact of the catchment dam and the bundling and trenching exercises. This is an example of a pilot project carried out without thought to measurement of results and therefore, without consideration of replicability. 5.4 Relevance: DRUM In India, power sector reform has undergone three distinct phases: • Phase I covers the early years from the time electricity was first introduced in the country until 1948, when production and distribution of electricity was largely in the private sector and concentrated in major towns and cities. • Phase II covers the five decades after independence. The development of the power sector was a subject in the concurrent list of the Constitution of India and each state had a vertically integrated, state-owned monopoly to distribute electricity in an assigned service area. • Phase III – the reform phase started in 1991, but actual implementation started in 1995. This period is marked by growing commercial attitudes, attempts to attract private investment and participation, restructuring of utilities in several states, and establishment of IRCs (Independent Regulatory Commissions) at the Central Government and the States that paved the way for the Electricity Act 2003. THE ELECTRICITY ACT – 2003 focused on efficiency and commercialization. “Competition with regulatory oversight is the framework around which the Electricity Act 2003 is woven – competition, to encourage efficiency in performance and regulatory oversight, to safeguard consumers’ interests and at the same time ensure recovery of costs for the investors”. Distribution is the most critical segment of the electricity business chain. The enactment of the Electricity Act in2003 created a new paradigm for the development of the power sector in the country, thus creating a new competitive framework for the development of the power sector in Recommendation: Develop a mechanism (maybe as part of USAID/India’s overall Monitoring and Evaluation activities) to measure the outcome and impact of WENEXA over time. Lesson Learned: One-off pilot projects are rarely effective. They must be carefully planned and, in the planning stage, must take into account a baseline survey and systematic measurement of results. A EVALUATION OF DRUM AND WENEXA 29 India with a focus on the consumers and safeguarding of their interests by independent Regulatory Commissions. The Electricity Act 2003 provides for a robust regulatory framework for distribution licensees to safeguard consumer interests, and creates a competitive framework for the distribution business, offering options to consumers through the concepts of open access and multiple licensees in the same area of supply. The enactment of the Electricity Act 2003 provides a framework for more competitive, transparent and commercially driven power sector. The Electricity Act 2003 stems—at least partially—from the realization that a viable distribution sector is a prerequisite for attracting investments throughout the sector. As such, the 2003 Act emphasized three major focus areas for reform: • Institutional Changes: The establishment of an Independent Regulatory Commissions will lead to distancing of Government from tariff determination; tariff rationalization, discipline in terms of payment of subsidies; and the reorganization of the State Electricity Boards. • Operational Issues: The Act calls for sector governance reforms, such as implementation of metering, energy audits, control of theft and pilferage of electricity, and the privatization of distribution wherever feasible. Is also calls for technological improvements in terms of improving the transmission and distribution systems, installing High Voltage Distribution Systems (HVDS), and increased use of information technology. • Executive Initiatives: The Act also calls for the rating of State Electricity Boards/Utilities, and the implementation of widespread rural electrification programs. The Electricity Act, 2003 provides for a competitive framework emphasizing special measures for consumer protection; and the responsibility to the Government to ensure rural electrification25.The following table indicates specific distribution sector requirements of the Electricity Act of 2003 and whether DRUM addressed them. ELECTRICITY ACT OF 2003—BROAD DISTRIBUTION HIGHLIGHTS DRUM PILOTS DRUM TRAINING DRUM EXCHANGE Assignment of Power Purchase Agreements to distribution utilizes Competition for consumer benefits Consumer indexing and mapping  Consumer protection and quality of standards    Efficient management of distribution sector    Focus on energy conservation    High Voltage Distribution Systems to reduce technical and nontechnical losses    Metering plans to ensure supply through the correct meter    Minimum area of supply for second licensee Multi-Year Tariff framework for minimizing risk and promoting efficiency Open access in distribution Planned reduction of T&D losses    Promotion of renewables   Recovery of cost of service and targeted subsidies Special Electricity Courts to be set up by all states. Table 2: Electricity Act of 2003: Broad Distribution Highlights 25 See “Electricity Act, 2003,” Section-57 (Consumer Protection: Standards of performance of licensee); Section-58 (Different standards of performance by licensee); Section-59 (Information with respect to levels of performance). 30 EVALUATION OF DRUM AND WENEXA Under the DRUM project the three pilot site distribution companies (DISCOMS) have taken steps to improve their performance by undertaking various measures such as reduction of technical and commercial losses, improvement in load management, strengthening of metering, billing and collection avenues, enhancement of attention towards the quality of electricity supply and customer care, and so on (as mentioned earlier). The pilot sites have developed performance benchmarks in order to gauge their efficiency levels pertaining to the technical upgrades and customer service. DRUM was clearly in line with GOI’s Electricity Act of 2003 and the project took decisive measures to address a number of the key electricity distribution issues outlined in the Act. To that extent, it was relevant at its inception, as it is in its culmination. 5.5 Relevance: WENEXA WENEXA is very relevant in the Indian context, as about 30 percent of energy is used by agriculture at the national level and more than that (30-45) in many states like Karnataka, Tamilnadu, Andhrapradesh, and Gujarat where ground water plays an important role in agricultural production. At the national level about 60 percent of irrigated agriculture depends upon ground water, which provides livelihoods to about half of India’s farming community. From 1950 to 2000, Indian farmers have spent about $12 billion for the purchase of pump sets. Most of this investment comes from farmers’ own resources.26 The National Water Policy of 2002 (which is under revision) has given adequate emphasis to recharging ground water and using it efficiently through regulations at the state and local level. With direction from the central government, many states have passed ground water regulation Acts (including Karnataka) aimed at fulfilling the GOI’s National Water Policy. Yet, ground water continues to be depleted at an alarming rate throughout India, endangering the livelihoods of farmers, and indeed the nation. This trend has wider ramifications for energy use in the agriculture sector. Energy used for irrigation has been steadily increasing thanks to free electricity in many states. Compounding the problem, many of the pumps are energy inefficient and thus contribute to increased and wasteful energy use. In this context, the project site in Doddaballapur is quite relevant as all the problems associated with ground water-energy nexus discussed above are also noted in this area which the project tried to address. 5.6 Effectiveness: DRUM 5.6.1 Cost Effectiveness (Return on Investment) Return on Investment (ROI) is a concept utilized in financial analysis to define the financial efficiency of the investment or the rate at which the investment is returned to the investors. In this context, the DRUM programs will be reviewed to determine the quantitative and/or qualitative return on the USAID investment to the participants, electric utility, consumers, region and the impact on the local, state, regional, or national economy. The DRUM program was initiated on May 11, 2004 and will be completed on March 31, 2011, except for the training program, which has been extended until Sept. 30, 2012. The DRUM obligation is $17,050,265 of which $13,619,306 has been spent as of March 31, 2011. WENEXA has an allocation of $5,888,267 of which $5,400,000 has been obligated through March 31, 2010. The 26 IWMI TATA-Water Policy Briefing-International Water Management Institute, Colombo, B A+ EVALUATION OF DRUM AND WENEXA 31 total funding for the program is $31.1 million, which includes loans from PFC and GOI of $3 million. This investment purchased technical assistance, training activities, funded three pilot projects, the RUS component, and both the contractor and USAID/India management operations. However, it would be difficult to provide a direct financial relationship with these specific activities. Some examples will be provided to illustrate the return on investment on a generic and specific basis. The normative approach is to define ROI as [Vc>Vb+ DRUM] or to determine if the current year Value is greater than the base year Value with the added support of DRUM. If the value of the impact or growth of the program is positive the effort can be considered a success. If the value of the results of the current year is less then the base year investment, ROI is considered negative. When comparative data is available for both the base year and current year a quantitative process will be initiated, however many of the programs will require a qualitative review utilizing subjective judgment for the program results. Different approaches are utilized for each program and the methodology of this evaluation is designed to provide the basis for the defining the effectiveness of the program. If data is available, the financial approach for determining ROI can be utilized: ROI = Gains-Investment Costs Investment Costs The attempt to measure the performance success for a government entity program includes the inherent difficulty of defining and measuring targets. In the private sector, a single number, net profits, can provide a summary of the performance of a company for a comparative period, usually a year. But this evaluation will attempt to provide examples, which can indicate the impact of the DRUM program on the beneficiaries. Additional data available indicate that for the three DRUM pilot projects the transmission and commercial losses dropped from 53% to 15%, and transformer failure rates dropped from 10% to 2%. These improvements convert to an energy savings at the three sites of 118,875 MWH, or the reduction of the equivalent of 94,000 tons of CO2. The consumer satisfaction index has increased in two of the three sites and in all three site, socioeconomic life has improved. The substantial reduction of T&C losses can be directly converted into additional direct power, since improved demand side management is the most efficient source of supply for this and subsequent years. The three pilot sites developed KPI (Key Performance Indicators) that showed continued improvement in electric output at each facility. These results can be evaluated as a positive return on investment. 27 The installation of capacitor banks has also had a positive return on investment28. Capacitor banks can improve voltage drops, reduce loss problems in the network, and increase the line efficiency. A cost-benefit analysis of installing capacitors at the three pilot site utilities is indicated in the following table. 29 Location of Pilot Site Cost of Capacitors Savings for Feeder Payback period [Months] Doddaballapur INR 1,064,990 ($23,439) INR 75,390 ($1659) 14 Aurangabad INR 1,899,700 ($41,811) INR 108,681 ($2392) 17.5 Umreth IRN 320,000($7043) INR 19,343 ($425) 16.5 Table 3: Cost-benefit analysis of installing capacitors at pilot sites 27 Electric Power Distribution, Case Studies, pg 7 28 The most common method used to improve the power factor, a measure of how effectively electrical appliances convert electricity into the required output, is to add capacitor banks into the existing system. 29 Electric Power Distribution, Case Studies, Table 7.9 32 EVALUATION OF DRUM AND WENEXA The relatively short payback of less than two years indicates a positive rate of return for the investment in these pilot projects and would strongly suggest the instillation of capacitors should be replicated at other sites by the utilities. DISTORTED TARIFFS DISTORT ECONOMIC ANALYSIS A nominal Supply-Demand chart for a commodity, including oil, coal, electricity and water, can provide an indication of the impact of tariff distortions on the sector. In the chart, the vertical axis is Price and the horizontal axis is Quantity. In the Supply curve the normal Quantity of a commodity increases as the Price increases, conversely the Demand curve indicates that the Quantity increases as the Price decreases. The equilibrium point is the intersection of the Demand and Supply curve at P1 and Q1. However, when the price is undervalued, as in P2, instead of providing the equilibrium at Q2, the demand moves to the higher quantity level at Q3. This indicates that when a commodity is underpriced, by government policy, the inaccurate signals sent to consumers will increase the propensity to over-consume it. Figure 7: Overconsumption of an underpriced commodity This inability to provide electric power to consumers with appropriate economic signals of the cost of providing the electric service, particularly in the agriculture sector, has continued to be an obstacle to creating a commercially viable electric utility sector in India. To be effective, the tariff for a specific consuming sector must reflect the cost of providing the service, including the capital and maintenance cost, in addition to providing sufficient profit to reinvest in the business and expand operations. From an economic viewpoint this approach indicates a number of issues: • Inaccurate tariffs are inefficient: Providing subsidies for tariffs distorts the consumer perception of the real price, and complete subsidies to agriculture consumers will guarantee a distorted demand for this sector; • Underpriced commodities are wasted: The lower priced electricity will produce higher then normal consumption and wasteful use of electricity by the agriculture sector. The excess use of precious electricity by one sector is reflected in increased tariffs for the other sectors; • Below cost tariffs discourage investments: The rural areas are most in need of improved electric service, but the expectation that low or no income can be expected from this consumer segment, will reduce incentives by the utility to invest and provide the expanded service; • Subsidies for tariffs are inefficient: Direct subsidies for tariffs for specific sectors are ineffective since it supports the range of poor to rich farmers and alternative approaches, including direct subsidy payments to farmers who then pay appropriate tariffs is more effective; • Excessive water pumping will exhaust the aquifers: Since there is no tariff for both electricity and water in the agriculture sector, the aquifers will be depleted at an excessive rate and the next generation of consumers will be faced with inadequate and poor quality water. This would indicate that an effort to move to a viable tariff system would benefit all consumers and provide a path for the electric power sector to become financial independent and attract foreign direct investment. It is not EVALUATION OF DRUM AND WENEXA 33 5.6.2 Management Structure The management structure for DRUM was effective to the extent that the three focus points of the project all progressed at a reasonable pace. As mentioned above, the DRUM Secretariat was a useful coordination tool between USAID, PFC, and Tetra Tech. Coordination between the DRUM partner organizations has been excellent. The target group for the pilot projects was appropriate at the time. 70 percent of India’s workforce is agricultural, so two rural sites was appropriate. However, the objective of “commercially viable distribution utilities” is hindered by the lack of tariffs for agriculture consumers. For the training component, the target groups were also appropriate. The focus on upper management, mid-management and linesmen was especially useful and innovative, as prior to DRUM, linemen received no systematic training. 5.6.3 Knowledge Transfer The effectiveness of the program can also be assessed by whether the target utilities have adopted best practices, as per the project design. A number of best practices—including measurement and tracking of Key Performance Indicators, installation and use of GIS software, establishment of customer-friendly business practices, installment of state of the art technologies for loss reduction, establishment and implementation of modern safety procedures, and the establishment of a culture of training for all levels of employees—have been adopted by the utilities. In terms of best practice dissemination, the DRUM training component is likely to have the most impact. Not only have most of the target pilot site employees been trained under DRUM, a large portion of the larger utilities’ employees have also taken DRUM training. There is, however, no evidence that the best practices from DRUM have been adopted by the larger utilities or by other Indian distribution utilities. 5.6.4 Donor Coordination There are a number of significant donor programs targeting the energy and water sectors in India, primarily sponsored by the World Bank and the Asian Development Bank. However, there was no indication that the DRUM program has coordinated activities with these donor programs, though there has been some coordination with the Japan International Cooperation Agency (JICA). Since the level of international donor aid is being reevaluated and some countries have withdrawn from India, including UK and France, it is essential that USAID coordinate with other multilateral development organizations and country donors to maximize support to the energy sector in India. Included in the World Bank’s active portfolio of Indian energy projects (19 projects) is a $60 million project to improve a financing energy efficiency program that will build capacity for energy efficiency at small and medium sized enterprises, and increase energy efficient investments. It will also improve access to commercial finance options for demonstration purposes. The Asian Development Bank (ADB) has 23 active energy programs, some closely related to the DRUM activities30 30 Active ADB energy projects: . • Solar Power Generation Facility, 31, Dec 2010; • Capacity Building for Water Response Sector, 16 Dec 2010; • Power Grid Transmission, 28 Feb 2011; • Madhya Pradesh Energy Efficiency, 15 Dec 2010; • National Action Plan on Climate Change, 21 Oct 2010; • Energy Efficient Enhancement in power Generation, 13 Oct 2010; • Integrated renewable Energy Development, 9 Sept 2010; C B+ A 34 EVALUATION OF DRUM AND WENEXA 5.7 Effectiveness: WENEXA WENEXA is an innovative concept that tries to address issues of inefficient and un-regulated ground water and energy use in agriculture. In Doddaballapur, the WENEXA project has not promoted collaboration between providers of energy and others who are involved with water provision/regulation services in agriculture such as the Water Resources Department, Agriculture Department and others. This lack of coordination is due to the focus of the project being on the energy side of AgDSM, while water-related interventions are not given adequate attention. WENEXA is mainly focused on energy savings through the replacement of energy inefficient pumps with efficient ones. However, energy savings in ground water agriculture depends upon not only efficient pumps but also on the level of ground water (which again depends upon extent of recharge from rainfall), as well as regulation of ground water use as a common property resource based on collective action rather than individual action. This approach has been proved to be useful by some of the NGOs working on Ground Water Management in dry regions of India31 To date, the ground water interventions have been piecemeal. There were some interventions related to ground water recharge through a watershed in the area, but this has come about only as the project is ending. There were also demonstrations on drip irrigation under the watershed management component. There was some attempt to inform farmers about the availability of subsidies for drip irrigation for sericulture through the NGO-IYD. A water balance study at the basin level was conducted to assess the water resources of the area. Yet all these activities have been undertaken in a piecemeal and isolated manner with no verifiable impact in the area. . Hence, an integrated approach is essential for improving the utilization of water and electricity resources. While implementation of such a holistic approach to energy-water management is complex, the passage of the Ground Water Regulation Act in many states, including Karnataka, provides the legal basis for the regulation of ground water as common property, which can be used for building up such a holistic approach. Has WENEXA impacted GOI policies in Agricultural Demand Side Management? There is no evidence that WENEXA has impacted the policies of the GOI in Agriculture Demand Side Management. However, the project has been instrumental in bringing some changes in the government policy in Karnataka and the state has announced its implementation (replacement of pump sets) in a few taluks on a pilot basis.32 Implementation of the scheme in other energy distribution utilities is also under consideration of by the government.33 5.7.1 Return on Investment WENEXA has made many efforts in spreading the idea of AgDSM based on PPP model in BESCOM with other state utilities like Tamilnadu, Andhrapradesh, Haryana, Rajasthan, Gujarat and Maharashtra state governments. Meetings were conducted with the planning commission, GOI, the Central Electricity Regulatory Commission, and NABARD for sharing the idea at the national level. Since the project is not yet complete and the results are not known, it is too early to see the policy impact of the project. The WENEXA website is good source of information about the program for those who have access to the Internet. For WENEXA, a return on investment will be only forthcoming after several years of the program working at full capacity. • Clean power Development-Himachal, 31 Aug 2010 31 Kulkarni and Vijayashankar (2009) ‘Groundwater: Towards an Aquifer Management Framework.’ Economic and Political Weekly, Feb7 32 Budget speech, Agriculture Budget-2011-12, Government of Karnataka 33 Ibid-page 4 C EVALUATION OF DRUM AND WENEXA 35 5.8 Sustainability: DRUM The question of sustainability encompasses a number of factors. Sustainability refers to whether the benefits of the program can continue over time. Included in that definition is the question of whether institutional capacity has improved to such an extent that the benefits of the program can continue and whether they can continue even after USAID support stops. Could the Government of India continue to support such activities without donor support? Could the activities continue without any external support? The question of sustainability also encompasses the extent to which best practices have been institutionalized by the utilities. We have chosen to assess the sustainability of DRUM by focus area, as the sustainability rating will be different for the training component, the exchange program, and the pilot projects. 5.8.1 Training The DRUM training program has been extremely successful and there is evidence that the program will continue once USAID support stops next year. Indeed, GOI has adopted the DRUM training for its RAPDRP program. Additionally, the DRUM materials are being modified and used by training institutes and in the utilities’ own training programs. The “energy MBA” program at MDI, the only face-to-face long-term training program34, no longer receives USAID support and yet has been able to raise the price of the course by four times, while enrollment has increased by three times. REMI is in the process of modifying and upgrading the DRUM materials to be included in their own course offerings35 5.8.2 Pilots . As a result of DRUM, a culture of training has been established in the target utilities, particularly where linemen are concerned. Now every lineman receives training. Even when the USAID subsidy was reduced, enrollment in the DRUM training courses increased, indicating the value the utilities placed on the training program. Without external support, the training component will continue, though in modified, upgraded form. The pilot projects had impressive improvements in efficiency and clear benefits to the customers, as well as to the utility subdivisions. The pilots were innovative in their approach to infrastructure improvement, management of the process, and the upgrades themselves. The benefits of the improvements are likely to continue over time and the pilot sites are likely to continue to track KPIs, thus demonstrating best practices to the larger utilities. However, there appears to be no systematic maintenance plan for the technological upgrades, which, in time, could dampen the benefits of the upgrades. In terms of replication, or scaling up, there has been no reproduction of the pilot projects and institutional support for innovative projects is lacking. That is to say, the locus of change is still external (MOP, the regulatory body) and to that extent, it is unlikely that new DRUM-like projects will be pursued. Additionally, Tetra Tech project management and expertise were crucial to the 34 IGNOU has an online energy certification program that encompasses nearly all the DRUM modules. 35 REMI is developing their own training materials, post-DRUM: For Executives: Best Practices in O&M; Change Management in Distribution Business; Customer Management in Distribution Business; Distribution Automation and SCADA for Power Utilities; Distribution Loss Reduction; Electricity Act of 2003 and Amendments; Financial Management of Distribution Business; Geographical Information System (GIS); Human Resource Management in Power Sector; IT Interventions in Power Sector; Maintenance Management in Distribution; Metering in Distribution System; Open Access, Power Trading and Tariff—ABT Scenerio; Pilferage of Electricity—Technical and Legal Remedies; Power Transformers and Distribution; Transformers—Modern Trends and Practices; Safety in Distribution Operation; Smart Grid; Tariff Regulations and Policy; Vigilance & Theft Reduction. For Technicians: Performance Enhancement for LT Jointers; Performance Enhancement for Fuseman/Linesman; Performance Enhancement for Fitters; Performance Enhancement for Supervisors—Street Lights; Performance Enhancement for Supervisors—HT & LT Maintenance; Performance Enhancement for Supervisors—HT & LT Construction. A+ Recommendation: Before DRUM officially ends, help the pilot sites develop a systematic maintenance plan. C 36 EVALUATION OF DRUM AND WENEXA success of the pilots. Similar programs without technical and managerial support are unlikely as managers’ time is taken up with the day-to-day running of the subdivision. In terms of financing innovations and upgrades, it is unlikely that this could happen without the support of the PFC, as commercial banks are wary of lending to distribution utilities, many of which run at a loss. The abandonment of the alternative financing piece of DRUM has not improved the ability of state-owned distribution utilities to finance system upgrades without government and/or donor support. It is too early to determine DRUM’s possible influence on GOI’s Reformed Advanced Power Development Program (RAPDRP). RAPDRP was proposed at a Government Summit on March 25, 2011 in New Delhi. If passed, it will be part of the next five-year plan for the utility sector. It has two parts: Part A is designed to improve the utilities by incorporating IT applications for accounting, auditing and consumer services. Part B is focused on modernizing and renovating the transmission and distribution system. Financing is through a loan from PFC/REC but converts to a grant if targets are achieved. If implemented as proposed, RAPDRP will use the Indian IT sector to support the utilities’ move to a smart grid network. Some of the technological upgrades, tasks and goals are similar to DRUM. The key difference is financing where the RAPDRP is a loan converted to a grant if the program’s goals are achieved. The lack of this sort of incentive-based financing scheme was a possible constraint to DRUM’s replication, as DRUM was designed to show the financial self-sustainability of the reform program. If RAPDRP is successful, DRUM could get credit for some of the initial ideas, but the evaluation team would have to return in five years to reevaluate the situation. At this stage, it is beyond our scope. 5.8.3 Exchange Program The DRUM exchange programs have been a tremendous benefit to BESCOM and MSEDCL. Both utilities now pay their own way for the exchange visits, with RUS paying only for their staff’s time, and for visits from U.S. utilities to India. Executives who have participated in the exchange program have been able to list specific things they learned from their interaction with U.S. utilities and indeed, the exchange visits may be one way to encourage reform from the top (as mentioned above) by including key decision makers. It is entirely possible (and probable) that the exchange visits will continue without further USAID support, though in this case, the benefits appear to exceed the costs of the program. The key benefit of RUS involvement is in making the initial linkages between the utilities and arranging for the visits. While the visits could continue without this support, the cost of this assistance is minimal. 5.9 Sustainability: WENEXA Is the public-private partnership model for ADSM on a performance-based contract basis sustainable? The success of AgDSM model in BESCOM depends on a number of factors (apart from high energy efficient pumps) that are not under the control of the Enzen-the implementing ESCO. These exogenous factors include the depth of the tube well, the level of the ground water, regulation of ground water use by farmers, changes in cropping pattern towards water- intensive crops over a period (influenced by markets),reduction in the efficiency of the pumps and so on. A basic problem in implementation of this model in Karnataka is that electricity is provided free of cost and hence there is absolutely no incentive for farmers to use it efficiently. Also when there is no regulation regarding the use of ground water, especially by the community, resource rich farmers will use it more intensively, affecting others in the project and leading to skewed use by a few. All these factors A Recommendation: If pilot projects are to be replicated, and as long as decision-making capacity lies ultimately with the government, it is crucial that the government (state and national) is an active partner in future pilot projects. EVALUATION OF DRUM AND WENEXA 37 will affect the energy baseline of the project drastically and therefore the calculation of energy savings. It is also difficult to establish a revised baseline as it is expensive and involves considerable hands-on work like fixing of new pumps, new meters, and recording of data for a reasonable period before for establishing the new baseline. Though it is too early to conclude, the efforts of Enzen in the replacement of pump sets of all the 604 farmers covered under the program look bleak. Out of 604 , only 460 pump sets have been energy audited as of March 11 2011. Out of the 460 pumps for which energy audits have been completed, there are still 35 farmers who will not agree to have their inefficient pumps replaced, even after the completion of the energy audit of the pumps. Additionally, about 43 pumps got dried up making it difficult to replace the pumps. There are still other issues: some farmers re-bored their tube wells due to the depletion in ground water; some have abandoned the wells and shifted the pump set to a new bore well as the previous one had dried up; some of the bore wells are deeper than reported during baseline (thus require more energy to pump the water), and others have technical problems. These factors are likely to make the implementation of the project difficult and there are is a large possibility that the project will be unsustainable, at least under the present approach. However, the model can be replicated provided all the issues described in the above paragraphs are addressed within an integrated framework. Urban water and energy nexus intervention? The waste water treatment project was completed in Nagpur, Maharashtra, as a part of the WENEXA 2 project under task six .The main objective of this task is to “provide site-based support to improve urban energy/water accessibility, delivery, efficient use, and waste management.” The project was designed to treat the wastewater of Nagpur city by Nagpur Municipal Corporation (NMC) in order to provide non-potable water for use in the thermal power plant of Maharashtra state Power Generation Company (Mahagenco). The treated wastewater was used for cooling the thermal plant and cleaning fly ash from the plant. The project aimed to generate 245 million liters per day (MLD) of treated water, about 110 MLD of which is supplied to Mahagenco. Tetra Tech provided the technical assistance for preparation of the Detailed Project Report (DPR) of the project. The work was completed by NMC at a cost of INR1.3bilion ($28 million) in 2007. The costs were shared by the Government of India (50 percent through the Jawaharlal Nehru National Urban Renewal Mission known as JNNURM), Maharashtra state government (20 percent) and by NMC (30 percent). Mahagenco would bear the operation and maintenance cost for 30 years and pay for the water tariff at INR1.07per cubic meter to NMC. The project saves energy from the treated water which otherwise would have been pumped from ground water supplies. Thus the project aims to address savings in both the energy and water sectors. The project was started in 2005 and completed on time in 2007, gaining popularity in the state for which both USAID and Chief Minister of the state gave commendations. An agreement was signed between Mahagenco and NMC for operation of the plant for 30 years. The project also contributed to the formulation of a state policy on wastewater reuse. The project clearly has demonstrated the importance of recycling of wastewater through treatment, which contributes to savings in both the water and energy sectors. The project is also an example of collaboration among different agencies of the government and electric utilities. However it is not clear from the available quarterly reports whether the project has been replicated in the state or other places. Proper follow seems to have been lacking, as the project was completed at the beginning of the WENEXA 2 project. A Recommendation: Utilize the project as a model for replication in the state and elsewhere, while also using it a model policy advocacy on water and energy nexus issues in the urban sector. 38 EVALUATION OF DRUM AND WENEXA 5.10 Gender In the three project sites, explicit attention has not been given to energy service needs of women (especially poor women), nor to mainstreaming relevant gender issues in power distribution within the scope of the DRUM project. Gender considerations can inform policy makers and planners about the range of needs for electricity services among a population. As stakeholders with different roles in society, men and women may have particular expectations and views of quality power distribution. In particular, women view electricity as a means to save time and reduce drudgery, thereby freeing them for other income generating and household activities. Chief among women’s concerns in several DRUM sites is the use of electricity for distributing, pumping, and storing water for domestic use. In areas of unreliable power distribution, women are particularly dissatisfied with the utility’s services. Therefore, better coordination between water and energy utilities, particularly in urban areas, is important. In rural areas, this “water and energy nexus” is also important, as unreliable electricity supply increases women’s burden of work and time spent in collecting water. Women’s organizations and cooperatives may be helpful in promoting communications and outreach strategies to educate people about energy conservation and efficiency. In particular, women’s groups may be a strong force for building societal pressure to reduce theft. Perhaps most importantly, women’s SHGs and NGOs, to the extent that they are community-based organizations, may be effective in assisting utilities in billing and collecting fees. Training and capacity building is needed to build power sector policy makers’ and practitioners’ ability to integrate gender considerations in their current work and to identify issues that may arise in the future. With improved access to electricity, it is hoped that women and men will have improved opportunities to create income. It is hoped that women in particular will use these opportunities and the time saved because of access to electricity to contribute more productively to their families and communities. A gendered approach might have addressed: (a) Concerns particular to women and men in power distribution in the three project sites of Umreth, Gujarat and Aurangabad, Maharashtra, and Doddaballapur, Bangalore. (b) Opportunities for strengthening commercially viable models of power distribution based on improved customer satisfaction and revenue collection. (c) The role of women as a consumer and demand side management. (d) Capacity Building as part of the training component for utility employees. (e) Providing some direction on how to integrate gender concerns into planning procedures and program implementation within the DRUM project. As part of the purpose of this evaluation is as a learning tool, the evaluation team has undertaken a gender assessment of DRUM, which is attached at Annex 6. Recommendations: • Enlist women’s NGOs and cooperatives in franchise opportunities; • Encourage women’s participation in customer service and awareness campaigns; • Promote gender balanced hiring in utilities; • Include gender sensitization in capacity building for utilities. EVALUATION OF DRUM AND WENEXA 39 6.0 Conclusions When the DRUM program was initiated in 2005 the operating performance of the electric utility sector in India was significantly below U.S. standards. The state-owned power sector was in continued financial crisis with most individual electric utilities considered bankrupt by U.S. financial standards. The financial losses of this sector were estimated at over 1.5% of GDP. Electric tariffs are distorted with a high degree of cross subsidy and do not cover the cost of providing service. The political context of providing free (or very low cost) water and electric service to lower income consumers has a negative impact on the industry. Technical and distribution losses were estimated at 40 to 50% of generated electricity in most states in India or over $5 billion. This can represent over 10% of the states’ budget. The problems included: poor metering, inadequate revenue collection, inefficient billing, commercial losses, theft and poor management. Resistance to metering was particularly strong by agriculture consumers, which prevents cost recovery and distorts the market. The current situation has not shown a significant change in the financial viability of the electric power sector. For the 2008/9 year the deficit in power averaged 11.1 percent, and the deficit in peak demand was 12 percent36. For the 2007-8 year, the profit and loss data indicates that even with the inclusion of GOI subsidies, all major electric utilities in India have a negative cash flow and a continued financial loss for their operations3738. At the micro level, DRUM has not reversed the negative financial situation of India's electric utilities and much of the failure resides in the institutionalized system of subsidized electric power. Subsidized power and water will also affect the final outcome of WENEXA, as there is no incentive for farmers to save energy because electricity is provided free of charge. Future energy-water programs in India must take an integrated approach, which takes into account both energy and water (not just energy). The implementation of WENEXA lacked such an integrated approach, neglecting other aspects like ground water recharge, coordination with other agencies related to water and agriculture, and involvement of farmers’ organizations in planning and management of the project. Continued heavy subsidies on electricity and water could undermine the successful outcomes that have resulted from DRUM’s technological upgrades and training, and WENEXA’s public-private partnership for energy efficiency. Another factor that could undermine the success of DRUM is the failure to replicate the pilots, which may have at its root a flaw in the initial design and strategy of the DRUM program, specifically the lack of a coherent replication strategy. Additionally, training by itself cannot achieve the kind of reform or transformation that is needed. Change must come from the top and from the bottom, but unless top management and policy makers are committed to reform, the efforts will likely fall short. At the end of the day, the continued financial constraints on India’s electricity distribution system could discourage foreign direct investment in this crucial infrastructure, essential to power India into the competitive world market. 7.0 Recommendations 7.1 Process Recommendations • Future USAID training courses should include a systematic method of measuring knowledge and/or skills gained from training courses. Pre- and post-testing for key themes, skills, or concepts would be one way of measuring such progress. • Establish and consistently use a shared drive where each project stores important project documents (including pertinent email exchanges). • Establish regular managerial meetings for projects with multiple sites and multiple managers. 36 TERI: TEDDY, 2010, Table 6.11, 6.12 37 TERI: TEDDY, 2010, Table 6.19 38 PFC, 2009, Performance of State Power Utilities 40 EVALUATION OF DRUM AND WENEXA • In designing future training programs, establish a mechanism for determining level of learning, as well as a mechanism for follow up regarding effectiveness. • In designing future multi-year training programs, establish a mechanism for periodic updates of the material to reflect current technology. • If pilot projects are to be replicated, and as long as decision-making capacity lies ultimately with the government, it is crucial that the government (state and national) is an active partner in future pilot projects. • Enlist women’s NGOs and cooperatives in franchise opportunities • Encourage women’s participation in customer service and awareness campaigns. • Promote gender balanced hiring in utilities. • Include gender sensitization in capacity building for utilities. • Expand support for renewable energy. 7.2 Programmatic Recommendations • Replicate a revised DRUM platform for Smart Grid pilot projects, taking into consideration the requirement for replication. • Expand the use of installed smart meters to pilot time-of-day and remote metering. This must be done with the consent of the regulatory body. • Provide U.S. experts on smart grid development, climate change, and grid technology for periodic conferences and presentations targeted at electricity distribution experts throughout India. • For next year, develop and implement Smart Grid DRUM training. • Develop one or two high-level workshops on Smart Grid around the country. • Develop and implement DRUM training for Agricultural Demand Side Management. • Continue work in the water sector. The problems that DRUM was designed to mitigate in the electric power sector are apparent in the water sector, particularly in the urban distribution area. Although 30 percent of the country lacks electric power, everyone must consume water on a daily basis. The water sector has inherent problems of contamination, erratic supply, poor metering and intrinsic defects at the final distribution level. The lack of meaningful water tariffs discourages investment in this structure, insures that groundwater aquifers will be depleted, and the future availability of water will diminish. We suggest future water interventions in the form of pilot projects at several urban locations and that they be executed as part of a public private partnership. This could be part of a WENEXA III program. • Replicate and improve urban wastewater treatment projects in other parts of India, based on Nagpur experience. • Before DRUM officially ends, help the pilot sites develop a systematic maintenance plan. • Develop DRUM environmental training related to distribution reform. • Modify outdated DRUM materials. • Develop and implement a short, DSM awareness program for women in DRUM pilot sites. • Consider establishing an exchange program focused on Smart Grid and Renewable Energy Project Development. • Continue watershed work, including proper baseline studies, comprehensive communication program, capacity building at the grassroots level, and involvement of farmers in water and energy management program. 8.0 Lessons Learned • Large, multi-year projects warrant careful midterm reviews and a frank assessment of the program’s stated objectives. • Demonstration projects require (i) comprehensive communication strategies that target decision makers as well as managers, and (ii) periodic assessments of the strategies to determine their impact. EVALUATION OF DRUM AND WENEXA 41 • Develop a mechanism (maybe as part of USAID/India’s overall Monitoring and Evaluation activities) to measure the outcome and impact of WENEXA over time. • Continued training in modern technological and managerial concepts is essential for a dynamic workforce. • Economic viability is essential for the devolution of the electric power sector; however, this requires a political commitment and therefore, beyond the scope of USAID. 42 EVALUATION OF DRUM AND WENEXA Annex 1: DRUM/WENEXA Evaluation: Statement of Work I. BACKGROUND Strategic Program Objective USAID/India's energy and environment Strategic Objective (SO16) is: "Improved Access to Clean Energy and Water in Selected Indian States". S016 reflects the Mission's focus on power distribution reform and the nexus between water and energy to improve access to more reliable, quality power in a financially sustainable manner. The USAID strategy for addressing these critical issues focuses on the central role of distribution reform to improve the quality and reliability of electric power. USAID had designed the Distribution Reform, Upgrades and Management project (DRUM) with the purpose to demonstrate best commercial and technological practices that improve the quality and reliability of "last mile" power distribution in selected urban and rural distribution circles in the country. The objective of the DRUM project is to establish the framework, institutional capacity and development functions at the central and state levels and to enable implementation of several full-scale, commercially replicable initiatives in key reform states in India. The WENEXA II project was initiated as a separate program, closely coordinated with the DRUM project, emphasizing the end-use efficiency of agricultural pump sets, on-farm water management, and the sustainable use of groundwater resources. DRUM Overview Reform of power distribution is fundamental to improving commercial performance and financial viability of the power sector in India. USAID/India recognized this and developed the Distribution Reforms Upgrades and Management (DRUM) with the Ministry of Power (MOP). Its design is drawn from a detailed analysis titled “Study Report: India Electricity Distribution Reform Initiative”.39 DRUM is based on the hypothesis that a power distribution utility--the only direct link to consumers--can retain existing customers and increase its customer base only if it is commercially viable and is accountable to satisfied customers. DRUM showcases replicable models of commercially viable electricity distribution along with supporting institutional structures at the state and central level. DRUM was launched “to demonstrate best commercial and technological practices that improve the quality and reliability of last mile power distribution in selected urban and rural distribution circles in the country. The key objectives of the DRUM program are: (a) to establish the framework, institutional capacity and project development functions at the central and state levels, and (b) to enable implementation of several full-scale commercially replicable distribution initiatives in key reform states in India. WENEXA Overview 39 For further information, please refer to USAID/India website at www.usaid.gov/india. EVALUATION OF DRUM AND WENEXA 43 USAID/India implemented WENEXA I, an I8-month pilot initiative focusing on the Agriculture sector in Andhra Pradesh. The pilot project sought to advance policy initiatives in the water (e.g. groundwater legislation) and energy sectors with an emphasis on power distribution services to rural consumers (e.g. loss reduction). Using a case study approach, WENEXA I launched a series of tasks in the Maheswaram MandaI area outside Hyderabad and the Andhra Pradesh Central Power Distribution Company's (APCPDL) service area. The watershed covered 40 sq. km, 6 villages, 1200 agricultural connections, and 30, 000 people.40 Phase II of WENEXA seeks to improve co-management of energy and water resources in the agriculture, urban and industrial sectors through enhanced power distribution and end￾use efficiency, coupled with sound water management practices. The activity will contribute to the power distribution reform process and introduce commercial best practices, help conserve energy and water resources, reduce growth of GHG emissions, and improve the quality of life for participating farm and urban families. B. PROJECT IMPLEMENTATION B.1 DRUM Component 1: National Strategy and Alternative Financing At the national level, DRUM will support this component through the following two initiatives: 1.(a). Support for Distribution Reform & Alternative Financing Partners: Maharashtra Electricity Distribution Company Limited (MSEDCL) and Bangalore Electricity Supply Company (BESCOM) Implemented by: Rural Utility Services (RUS), United States Department of Agriculture (USDA) Type of instrument: PASA with RUS with USDA PACD: September 30, 2011. The purpose of this component is to plan, design, finance and implement distribution reform and rural electrification frameworks and projects. RUS, a division of US Department of Agriculture, is responsible for providing financing and management support to rural electrification effort in United States. RUS brings vast experience to create a network of rural utilities in India. The objectives for work under this PASA at the time of conception were envisaged as follows: • Design alternative lending schemes for distribution entities (beginning with public investors such as the Rural Electrification Corporation (REC)), based on commercial lending principles rather than third-party government guarantees; • Engage Indian private credit rating agencies in credit worthiness assessments; • Develop of alternative financing mechanisms for distribution entities that include long￾term debt, partial equity, grants and credit guarantees; 40 For further information please refer to the WENEXA I website at www.waterenergynexus.com. 44 EVALUATION OF DRUM AND WENEXA • Develop of term lending structures that include covenants (such as benchmark performance standards), pricing, and duration periods that match the life of distribution system assets; and, • Support the APDRP cell in the Ministry of Power to design and implement its distribution reform strategy; monitor APDRP’s commercial modernization projects for distribution entities in reforming states. The tasks identified in the scope of work included: 1. Strengthen REC capacity for loan portfolio design and management for rural electrification project financing and replication; 2. Strengthen the capacity of APDRP to finance, plan, design and monitor rural distribution system improvements in selected distribution circles; 3. Support the design of rural distribution reform regulatory frameworks including the design and evaluation of applicable participatory rural distribution models; 4. Provide support to USAID/India’s appointed DRUM Institutional Contractors to demonstrate best management, commercial and technological practices in rural distribution system planning, operation and maintenance; and 5. Provide general project support for activities (as determined by USAID/India) such as organizing training and exchange programs, participation of Indian officials in national, regional and international conferences and seminars. The activities under the PASA primarily comprised of providing technical assistance, training and sharing of best practices through exchange programs, study trips, workshops, one-to￾one discussions and training programs. Over the years, RUS has supported its partners Maharashtra State Electricity Distribution Company Limited (MSEDCL) in Maharashtra and Bangalore Supply Company Limited (BESCOM) in Karnataka to improve distribution operations and management practices, reduce losses, and improve efficiency and safety. Many distribution engineers, managers and linesmen were trained in various aspects of the distribution business and rural electrification. As a part of exchange visits, RUS gets its cooperative members from the United States to visit MSEDCL and BESCOM to share their experience. Teams from the Indian partners also visit the member cooperatives and their facility in U.S. to learn from their experiences. 1.b. Support for Functional Training Partners: Power Finance Corporation (PFC), 19 Indian training institutes including Management Development Institute (MDI), Gurgaon, The Energy Resource Institutes (TERI), etc. Implemented by: Institutional contractor (CORE) for course development and Power Finance Corporation (PFC). Type of instrument: Contract with CORE for course development and PIL with PFC for acting as financial intermediary. PACD: September 30, 2012. One of the key strategies under DRUM was to design and implement a program to build capacity of the distribution utility engineers, managers, personnel, regulatory commissions and staff of various states. EVALUATION OF DRUM AND WENEXA 45 This component was designed to meet the functional need of the six target groups, namely - policy makers and regulators; regulatory staff; senior managers and engineers; mid-career managers and engineers; entry-level managers and engineers; and distribution utility personnel including technicians, linesman, meter-readers, billing and collection staff, customer interface personnel etc. The activity essentially comprised of designing the training program as well as imparting the trainings. The training covered broadly three functional themes: 1. Technical. Operation and maintenance of distribution assets; standards for equipment, systems and practices, network planning, construction and optimization; reducing technical losses, fault prevention, detection and correction; distribution modernization; safety standards and practices; IT for improving distribution services, etc.) 2. Managerial/Business. Enhancing commercial orientation; developing consumer friendly work culture; project preparation and development for distribution upgrades; innovative models and management of rural distribution business; financial planning and risk management; Procurement planning and inventory control; Human resource development and management; Reducing commercial losses; Management information systems for improving efficiency; Regulatory responsiveness; Disaster management for distribution utilities, etc. 3. Supportive functions. Skills enhancement such as communication, negotiation, report writing; motivational and behavioral aspects; etc. An Institutional Contractor, CORE, was appointed to provide handholding support to selected Indian institutions in evolving strategy, developing curriculum, course material, training of trainers and designing and delivering training to distribution utility personnel. The contract concluded in 2006 after development of 22 training course modules. A unique Executive MBA program was also developed and launched in 2005, in partnership with one of India’s leading business schools (Management Development Institute) to develop future leaders for the sector. While the contract for developing the course material has concluded, the DRUM training is still being imparted. For wider reach, the courses are instructed through 19 different Indian training institutes (both private and public) engaged in power sector and distribution reform activities. The objective of the training programs is to build a cadre of distribution engineers and technician by sharing experience on national and international best practices, and innovations in community based participative distribution business models. Practical demonstration, case studies and field visits also formed an integral part of training activity. DRUM has already achieved the cumulative target of training 32000 staff (engineers and managers) from the electricity distribution companies. The DRUM training is considered the most successful component of the program. This is widely accepted within the Government of India. The GOI’s training program under R-APDRP is heavily influenced by DRUM program with adoption of more than 18 modules that was being covered under DRUM. While the partners for the training program are the training institutes, Power Finance Corporation a subsidiary of Government of India, is the key partner. It was appointed as the coordinator for the DRUM program by the MOP. PFC is engaged in supporting the DRUM Project in three key ways. First, it provides management support to the implementation of the Project by co-coordinating activities with all key stakeholders. Second, it acts as a financial intermediary and banker for controlling and directing funds to DRUM pilot projects. Third, it directs DRUM capacity building efforts by acting as a resource center, databank and financial intermediary for the training component. Component 2 and 3: State Planning and Design and Distribution Reform Pilot Projects 46 EVALUATION OF DRUM AND WENEXA Partners: Maharashtra Electricity Distribution Company Limited (MSEDCL), Madhya Gujarat Vidyut Corporation Ltd. (MGVCL), Bangalore Electricity Supply Company (BESCOM) and Power Finance Corporation (PFC) Implemented by: Tetra Tech (Formerly PA Consulting) Type of instrument: Contract PACD: March 31, 2011. The institutional contractor Tetra Tech (formerly known as PA Consulting) is responsible for addressing the other two DRUM components. The contractor assits the DRUM program through provision of technical assistance, training, commodity procurement and other support services. Component 2 essentially focuses on training and capacity building to advance distribution reforms in states and providing technical assistance for activities such as: • Developing regulatory and legal frameworks for reducing electrical theft; • Mapping of distribution networks in selected states, in part to identify high-loss points in power delivery systems; • Disseminating engineering standards to encourage better technical performance; and • Sharing “lessons learned” from distribution demonstration projects to encourage replications of successful interventions. The centerpiece of the DRUM program is Component 3 (Distribution Reform Pilot Project). This component essentially comprises of demonstration projects in rural and urban areas focusing on capacity building and improving technical efficiency, cost recovery and customer functions in local power distribution units that exhibit best institutional and commercial practices for wider replication. The institutional contractor was expected to support the planning, design and implementation (including commodity procurement, installation, and commissioning of improved distribution systems in select distribution circles) in reforming states through TA & training. The activity under component 3 provided the blue print for the planning, modernization and financing of select distribution circles in India. It was envisaged that these distribution circles will serve as models of excellence and showcase efficient technologies, systems, business values and practices. A special feature of the project focused on enhancing customer relations and the role of village level communities, co-operatives, and private entrepreneurs in managing the business of rural power distribution. Four pilot sites were selected in consultation with the Ministry of Power. These are as follows: 1. Doddaballapur in Karnataka, BESCOM (rural site) 2. Umreth in Gujarat, MGVCL (urban site) 3. Aurangabad in Maharashtra, MSEDCL (urban site) 4. New Delhi in Delhi, New Delhi Power Ltd. (urban site) NDPL dropped out from the project as they wanted to follow the activities on their own. Therefore, DRUM project is now working at three sites, all of them with public sector. EVALUATION OF DRUM AND WENEXA 47 While developing this component, MOP and USAID identified 15 indicators to monitor the performance of the pilot sites. Based on the objectives of the program and activities identified in the scope of work, the institutional contractor developed a work plan for each of the pilot sites to achieve the target as set by MOP for the 15 indicators through implementation of technical and breakthrough programs. Keeping pace with development in the power sector, new concepts were also introduced in the project to expose the partner DISCOMs. BESCOM is being supported to do a pilot project on a smart grid for the first time in India (sub contracted to Central Power Research Institute). In addition, to sensitize partner utilities on Smart Grid, several workshops were also organized for stakeholders. A new set of activities were added under the institutional contractor under the Asia Pacific Partnership (APP) Program. The Market Development of Renewable Energy (MDRE) component provided significant impetus to the renewable energy market in India by supporting several inter-related activities. A Detailed Model Project Report for a 5 MW Solar PV Project in Andaman & Nicobar Islands for NTPC (National Thermal Power Corporation) was prepared. In association with Ministry of New &Renewable Energy (MNRE), a study was conducted for strengthening of the State Nodal Agencies (SNA) by WISE, Pune (subcontract). Another study on “Design of a Policy Package ,Development of Incentive Schemes, and Identification of Intervention Areas for Up gradation and Promotion of Small Manufacturing Enterprises (SME) was conducted APTRI, Hyderabad (subcontract). Training on regulatory and market aspects to regulators, policy makers, utility staff as well as private sector is an important component of MRDE. In second phase, 11 Capacity Building workshops on “Market Development of Renewable Energy” (MDRE) for about 550 stakeholders were organized by WISE, Pune (Subcontract). Other than the three DISCOMs, the Power Finance Corporation (PFC) again is a key partner in DRUM. It was nominated as Principal Financial Intermediary and banker for controlling and directing funds to DRUM pilot projects. B.2 WENEXA II Partner: Bangalore Electricity Supply Company (BESCOM) Implemented by: Tetra Tech Type of instrument: Contract PACD: March 31, 2011 Component 1 -Policy Dialogue. The component consisted of undertaking dialogue, policy and institutional support to advance efficient energy and water use in agriculture, urban and industrial sectors. The activities included under this component included development of an energy/water policy framework; provision of central and state-level programmatic and institutional support and support institutional arrangements for sustainable water resources management in target areas. Component 2 - Sector Reform. This component consisted of site-based activities to enhance commercial power distribution and sustainable water management in the agriculture, urban and industrial sectors. Some of the activities under this component included providing site-based support for improving the reliability and quality of energy and water delivery in the agriculture sector; supporting basin-level hydrological information collection and analysis for target areas; providing site-based support to improve urban energy/water accessibility, delivery, efficient use and waste management and providing 48 EVALUATION OF DRUM AND WENEXA support to improve the water service delivery, reliability and pollution reduction in the industrial sector. Component 3 - Customer Service. This component comprised of user group participation and service delivery to foster improvements in energy and water supply and demand. Some of the activities under this component included development of communication and participation strategy for targeted rural, urban and industrial sectors; improve power end-use efficiency; and promote sustainable water resources management through grassroots institution strengthening. WENEXA II is being implemented in Doddaballapur in Karnataka under Bangalore Electricity Supply Company, which is also the DRUM pilot site. The WENEXA is implemented on portion of area under DRUM project. It has benefitted from the implementation of DRUM program by creation of basic infrastructure such as installation of automatic meters, separate feeders for agricultural pumps, etc. II. EVALUATION SCOPE i) Purpose and Objective USAID/India intends to carry out an in-depth and thorough evaluation of the DRUM and WENEXA II programs. Though WENEXA is a separate program with its own objective, it is implemented by a common partner (Tetra Tech) and also shares the project site (Doddaballapur in BESCOM). USAID envisions one comprehensive report that covers the following: • PASA with RUS (Component 1.a of DRUM); • PIL with PFC and contract with CORE (Component 1.b of DRUM); • Institutional contract with Tetra Tech (Component 2 and 3 of DRUM); and • WENEXA II activities implemented by Tetra Tech. The objectives of the review are to: • Determine the impact and achievements of each of the instruments/components relative to stated objectives and achievements. • Document the challenges faced in implementation and the lessons learned • Effectiveness of the program management structure and the methodology adopted for implementing each component. • Make suitable recommendations for the future direction of energy projects related to distribution reforms and water/ energy projects. ii) Statement of Work This scope of work is for a comprehensive impact evaluation, including an assessment of the appropriateness of the project activities in achieving the stated objectives, the level of impact, cost-effectiveness and future directions. Critical stakeholders will be involved during various stages of the review process as appropriate. The team will gather both qualitative and quantitative data based on the following specific objectives. 1. IMPACT: EVALUATION OF DRUM AND WENEXA 49 • The evaluation team will conduct a performance evaluation of each of the implementing partners in executing their respective scopes of work. Were they able to meet their responsibilities under their contracts or agreements? • The impact of the activities implemented. For example, for DRUM the impact can be verified by documenting the changes in the performance of the pilot site vis-à-vis the pan utility as well as with respect to other utilities in the country in the key indicators such as AT&C losses, accident rates, Transformer failures, etc. • The extent to which the projects have achieved the objectives and met targets as set for the different components. Identify the contributing factors and barriers to achievement of the objectives that were not fully met. • Document the main successes and lessons learned. • Provide recommendations for future programs in distribution reforms and water-energy efficiency. 2. RELEVANCE: • Evaluate if the original hypotheses on which the program was designed is still valid. • Assess if the DRUM and WENEXA programs were able to respond to the needs of the GOI. Also assess if it still continues to respond to the needs of the counterparts. • Review if the program was able to reinvent itself when faced with challenges in the implementation. 3. EFFECTIVENESS: • Cost effectiveness of different instruments in meeting the said targets and objectives. What has been the return on USAID’s investment? • Assess the effectiveness of the program management structure for DRUM and WENEXA. • Determine whether, and how effectively, the projects have coordinated and collaborated with the host government and stakeholders at all levels. • Effectiveness of the project in transferring organizational development skills and developing capacity of the partners and beneficiaries. • Appraise the quality of the technical expertise provided under the project to the pilot sites and for other components of the project. Evaluate if the program was able to make most of opportunities in introducing new and emerging technologies. • Evaluate whether the project has been able to strengthen public/private/civil society linkages. 4. EFFICIENCY: • Assess if the coordination of different implementation partners was effective to achieve the objectives of the overall DRUM program (e.g. determine how effectively the three components under DRUM collaborated with each other). Also ascertain how effectively the WENEXA project has collaborated with the DRUM project. 50 EVALUATION OF DRUM AND WENEXA • Appropriateness of the target group (i.e. have technical assistance, training, and partnerships been targeted at the appropriate beneficiaries to ensure the greatest impact). • To what extent have the outputs from technical assistance, training and other DRUM￾funded activities been utilized by targeted beneficiaries. Identify the best practices that have been adopted by the targeted beneficiary and if any additional organization who received information from targeted beneficiaries. • Determine the extent to which DRUM activities supported or complemented activities sponsored by other donor partners. • Evaluate if WENEXA has been able to promote collaboration between the providers of water and energy services for reliable and efficient delivery of energy and water to consumers. • Evaluate whether WENEXA has been able to establish an integrated approach to efficient and reliable energy and water management services and has an impacted GOI policies in Agricultural Demand Side Management. 5. SUSTAINABILITY: i. DRUM • Assess the sustainability of the DRUM program and its activities in terms of creating institutional capacity and filling gaps. Based on results to date, are these activities likely to engender sustainable development impacts after USAID funding has stopped? Would the GOI be able to support the activities undertaken by the project by itself in the future – in terms of funds, human resources? • Evaluate the extent to which the practices and results from DRUM have been institutionalized by the counterparts in their operations. • Investigate whether systems have been established internally for tracking, monitoring, and reporting on results attributable to DRUM activities and whether these systems utilize independently verifiable information. ii. WENEXA: • Is the public-private partnership model for Agricultural Demand Side Management on a performance contract basis sustainable? • Would the model be replicated by other GOI States or other utilities (e.g. Government of Karnataka)? iii) Methodology The evaluators should consider a range of possible methods and approaches for collecting and analyzing the information which is required to assess the evaluation objectives. Data collection methodologies will be discussed with, and approved by, USAID/India prior to the start of the assignment. 1. Desk review of documents: USAID/India will provide the team with all relevant country and project specific documents such as proposals, reports, scope of works, etc. The evaluation team is expected to go through the official websites of DRUM, WENEXA and USAID/India and collect other relevant documents, reports, and data. The evaluation team is expected to be aware of the background of the program before meeting the GOI partners. The Mission points of contact (Monali Zeya Hazra and Apurva Chaturvedi) will provide the EVALUATION OF DRUM AND WENEXA 51 evaluation team with project reports, and other documents needed for conducting this desk review. 2. Team Planning Meeting: A team planning meeting will be held by the evaluation team in India before the evaluation begins. This will be facilitated by the team leader, and will provide the Mission with an opportunity to present the purpose, expectations and agenda of the assignment. The evaluators shall come prepared with a draft set of tools and guidelines and preliminary itinerary for the proposed evaluations. TPM will also provide an opportunity to the evaluators to explain the strategy that will be adopted for the evaluation to USAID /India. In addition, the TPM will also: • Clarify team members’ roles and responsibilities • Establish the timeline, share experiences and thoughts on the evaluation methodology • Interview questionnaire to be prepared in advance and finalized during the TPM • Finalize the data collection tools and guidelines 2. Site Visits and Interviews: The evaluator will meet relevant implementation partners, GOI partners, and subcontractors. • Interviewees will include key members from MOP and all other stakeholder groups such as: o DRUM Component 1a (PASA with RUS): RUS, partner DISCOMs (MSEDCL and BESCOM), REC, Management Development Institute (MDI). o DRUM Component 1b (PIL with PFC) – PFC, some partner training institutes (such as Reliance Energy Management Institute, NPTI, Cenpeid, PMI, etc.), MDI, TERI, Chief of Party of the institutional contractor. o DRUM Component 2 and 3 – Tetra Tech, MSEDCL, BESCOM, MGVCL, PFC, NTPC, Tetra Tech, WISE, APITCO, CPRI. o WENEXA - Ministry of Power, BESCOM, ENZEN, farmers in the target area, MERC and other partners and beneficiaries under WENEXA program • Interview questions will be prepared in advance and finalized during the TPM • Site visits will be planned, taking into consideration factors like geographical diversity, representation of various beneficiary groups, and scale of interventions. • The Team will evaluate the periodic reports to take stock of the indicators • Site visits to pilot sites and select training institutes. 4. Preparation of draft report and its finalization with inputs from USAID/India. Based on the collected information, interviews and site visits, the evaluation team will make a draft report which will be finalized after soliciting feedback from USAID/India. On the completion of the site visits, and after meeting with the GOI counterparts and implementation partners, the evaluation team will make a presentation to the USAID team on key findings. iv) Timeline USAID/India anticipates that the period of performance of this review will be from late February 2011 to March end 2011, for about five weeks. 52 EVALUATION OF DRUM AND WENEXA The five person evaluation team is authorized a total of five weeks for this evaluation, as follows: 1) a week in Washington, DC to read program documents, obtain briefings from USAID and implementing partner staff, and draft the evaluation work plan; 2) one day of consultation in Delhi with USAID team; 3) two weeks working in the field that includes one week travelling within the country and 1 week in New Delhi including debrief to USAID/I; 4) 1½ weeks for final consultations and drafting the report in Delhi, including a presentation to USAID of the draft final report, prior to finalization, and 5) Two days to finalize the report. v) Evaluation team composition, technical qualifications and experience requirements All team members must have relevant prior experience in South Asia, and preferably in India; and familiarity with USAID’s objectives, approaches, and operations, and prior evaluation/assessment experience. In addition, individual team members should have the technical qualifications identified for their position below: Senior Energy, Water and Policy Analyst (Team Leader): This Senior Analyst should have an excellent understanding of the energy and water sector. S/he should have significant experience monitoring and evaluating energy and water programs throughout the world, in particular Asia. A minimum of 12 years of experience in the design, management and evaluation of energy and water programs, particularly with regard to sector reforms, institutional development and policy interventions is desired. S/he will serve as Team Leader and will be responsible for coordinating evaluation activities of the various components and ensuring the production and completion of a quality report, in conformance with this scope of work. These reports will become public documents for distribution among the program’s key stakeholders, including high-level U.S. government policy makers and officials, host country government officials, private sector and NGO leaders, and other audiences. S/He should have experience of handling evaluation programs of similar complexity. (LOE up to 33 days). Evaluation Methods Specialist: This expert will have deep knowledge of evaluation methodologies and their practical applications. (LOE up to 33 days). Two Distribution and Energy Experts (Local): In addition to the Team Leader and Evaluation Specialist, the contractor should propose two additional local energy sector specialists with extensive knowledge of working in India. These experts should have relevant educational qualification and good understanding of the utilities operations and networks. They should also have an extensive understanding of the distribution reforms in the country and the latest technological innovations and best practices in the distribution sector. These experts should also have the experience of evaluating energy projects and working with the GOI as well as the private sector. One of the experts will be assigned to investigate Component 1.a (Support for Distribution Reform & Alternative Financing) under RUS/USDA PASA and Component 1.b (Support for Functional Training) under PFC. The other specialist will be assigned to investigate the component 2 and 3 (demonstration pilot) under the Tetra Tech contract. (LOE 20 and 24 days respectively). Water-Energy expert/consultant (Local):The local consultant should have a minimum of 7 years experience in the design, management of water-energy projects. S/he should have relevant educational qualification and good understanding of the utilities operations and EVALUATION OF DRUM AND WENEXA 53 networks. S/he should also have some experience of evaluating energy or water related projects. This expert will be assigned to investigate the WENEXA component. (LOE up to 20 days). Summary Table: Labor Labor Category Maximum LOE Senior Energy, Water and Policy Analyst (Team Leader) 33 Evaluation Methods Specialist 33 DRUM: Distribution and Energy Expert (Local) – Component 1a (RUS/USDA PASA) and component 1.b (PIL with PFC) 20 DRUM: Distribution and Energy Expert (Local) – Component 2 and 3 (Contract with Tetra Tech) 24 WENEXA: Energy Expert (local) 20 Note: Assuming 6 days in 6 weeks In addition, each team member should have, at minimum, the following skills and experience: 1. An understanding of the country context. 2. An advanced degree in energy, water, natural resource management, or any other relevant course of study. 3. Demonstrated skill in written and oral communication. 4. Demonstrated knowledge of USAID policies and procedures. 5. Ability to work effectively in, and communicate with, a diverse set of professionals. vi) Relationships and Responsibilities • The contractor will work under the guidance and general direction of Mr. Jeremy Gustafson, Office Director, CLEEO, Monali Zeya Hazra, Program Officer – DRUM, and Apurva Chaturvedi – WENEXA in conjunction with the Program Support Office. • Contractor will be responsible for obtaining visas and country clearances for travel for consultants. • Contractor will be responsible for coordinating and facilitating assessment-related field trips, interviews, and meetings in conjunction with the USAID/India and the implementation partners. • Contractor will be responsible for submitting a budget for all estimated costs incurred in carrying out this review. The proposed cost may include, but not be limited to: (1) international and in-country travel; (2) lodging; (3) M&IE; (4) in-country transportation; and (5) other office supplies and logistical support services (i.e., laptop, communication costs, etc.) if needed. • In-country logistics to include transportation, accommodations, communications, office support, etc. viii) Reports and Deliverables • Draft Work Plan and Pre-Departure Briefings. The evaluation team will develop a draft work plan prior to departure from Washington, DC. The team will meet with USAID and other contractor staff for at least three working days prior to initiating the field trip and meeting with key counterparts. 54 EVALUATION OF DRUM AND WENEXA • Oral Presentation. The evaluation team will provide an oral briefing of its findings and recommendations to the CLEEO Director and relevant staff in the field as well as to the respective country coordinators and other USAID staff at the conclusion of the visit. • Draft Report. The evaluation team will present a draft report of its findings and recommendations to the CLEEO office Director before return to the United States. • Final Report. Ten paper copies of the final report as well as an electronic version in Word shall be submitted within five working days following receipt of comments from USAID and its implementing partners. Ten copies will be provided to USAID/India. The final report should include an executive summary of no more than three pages, a main report with conclusions and recommendations not to exceed 40 pages, a copy of this scope of work, evaluation questionnaires used to collect information on each of the program components, lists of persons and organizations contacted and minutes of the meeting with different stakeholders. EVALUATION OF DRUM AND WENEXA 55 Annex 2: Persons Contacted AFGHANISTAN Kabul Tetra Tech James Hogan INDIA Bangalore BESCOM B.N. Sathyaprema Kumar B.K. Udayakumar CETC Rajalakshmi Jayaraman CPRI V Arunachalam R Gavirangappa C.P. Jairam Dhiya Metikurke Vasuki Dhiya Parameshwar Hegde Tetra Tech B.V. Rudresh Gurgaon Tetra Tech Sumedh Agarwal Sachin George Rakesh Kumar Goyal Michael Hajny Mumbai MERC Vijay Sonavane MSEDCL R.B. Gowardhan Deepak Lad Reliance Energy Uday Lajmi 56 EVALUATION OF DRUM AND WENEXA New Delhi BSES Gopal Saxena DRUM Sudhir Vadehra NDPL M.C. Gurani Uday Mishra Ajai Nirula PFC Neeraj Singh USAID/India Apurva Chaturvedi Monali Zeya Hazra Chandan Samal Vadodara DRUM V.C. Shah ERDA U.C. Trivedi MGVCL S.M. Godkhindi UNITED STATES U.S. Department of Agriculture Lawrence Barbieri Gary Bojes Melissa Fraser Mark Hawthorne Jeffrey McWilliams EVALUATION OF DRUM AND WENEXA 57 Annex 3: Pilot Site KKPI Tracking BESCOM: DODDABALLAPUR Table 4: Key Performance Indicators for Doddaballapur pilot site 58 EVALUATION OF DRUM AND WENEXA MSEDCL: AURANGABAD Table 5: Key Performance Indicators for Aurangabad pilot site EVALUATION OF DRUM AND WENEXA 59 MGVCL: UMRETH Table 6: Key Performance Indicators for Umreth pilot site 60 EVALUATION OF DRUM AND WENEXA Annex 4: Interview Questions EVALUATION QUESTIONS These questions are to be used as a guide during the interviews with key informants. Because we are using a semi-structured interview format, feel free to encourage the interviewees to expand on their answers. DRUM General Questions 1. How is the “best practice model company” defined by DRUM? Best practices are based on what? 2. How is customer satisfaction measured? 3. How have customer satisfaction levels changed over the course of the project? 4. How has changed customer satisfaction been measured? 5. How long does it take to change defective meters compared to before DRUM? 6. Are meters replaced as a result of customer complaints, or through routine inspection by engineers or linemen? 7. Are trained engineers utilized in the distribution sector or elsewhere? Questions on DRUM Objectives: 1. Has a framework for commercially viable and replicable electricity distribution systems been established? 2. Has the institutional capacity to develop commercially viable, replicable electricity distribution systems been established? 3. Have the project development functions necessary to develop commercially viable, replicable electricity distribution systems been established? 4. Have commercially viable, replicable electricity distribution systems been implemented as a result of the DRUM project? a. How many? b. Where? 5. Has a cadre of trained experts in electricity distribution been developed? 6. Has a systematic program of training electricity distribution professionals been developed? Component 1: National Strategy and Alternative Financing (RUS) 1. Were alternative, commercially-based lending schemes for distribution entities designed? EVALUATION OF DRUM AND WENEXA 61 a. Were they implemented? 2. Were Indian private credit rating agencies involved in assessments of the credit worthiness of targeted distribution entities? 3. Were alternative financing mechanisms developed? a. Were they put in place? b. To what effect? 4. Were term lending structures developed? a. Were they put in place? b. To what effect? 5. How many distribution engineers, managers, and linesmen were trained in aspects of the distribution business and rural electrification? a. Who did the training/exchange visit? b. Documentation? c. How many women trained at different levels. What efforts made by DISCOM to ensure women participation? 6. Is there documentation regarding the benefits of the training/exchange? 7. Has the RUS partnership had an effect on the way the DISCOMs function? a. Specifically how? 8. Why did this component shift focus away from REC to the utilities? 9. Is there documentation concerning the change in focus of Component 1 away from REC and toward the utilities? Component 1b: Support for Functional Training (CORE) 1. Was the DRUM training effective? a. If so, how do you know? b. If not, why not? 2. As a result of the training component, has a cadre of distribution engineers and technicians been built? a. How do you know? 3. How many graduates of the Diploma in Business Management (EPGDBM): Energy Management with specialization in Power Distribution? 4. Are there pre- and post-tests, which would indicate knowledge enhancement? 5. Is there an annual listing or registration of trained and tested engineers? 6. Is there a periodic retraining mechanism [every 4 years?] to maintain knowledge and skills? 7. What effect has the DRUM training had on those who took the courses? a. Are those effects measurable? 8. How has DRUM affected the training institutes? a. In terms of qualified trainers? b. Yearly revenue? c. In terms of number of participants per year? d. Does supply meet demand? 9. Is the current curriculum (2004-6) still useful? a. Is there any process for upgrading the course material? 10. Why was CORE’s contract terminated early? Component 2 & 3: State Planning and Design/Distribution Reform Pilot Projects 1. Have regulatory and legal frameworks been developed for reducing electrical theft? 62 EVALUATION OF DRUM AND WENEXA a. If so, have these frameworks been put on place? b. To what effect? c. Has the level of enforcement improved? i. If so, can it be quantified? ii. What is the procedure for non-payment and cut-off of service? iii. Has this changed since the beginning of DRUM? 2. Have best-practice engineering standards been disseminated? a. How? b. To what effect? 3. Have “best-practices” been replicated in the demonstration projects? a. Has technical efficiency been improved, and can it be documented? b. Regarding safety, how many linemen died by accident during the baseline period and how many in the last year after training on best safety methods? c. Has cost recovery improved, and can it be documented? d. Has customer service improved, and can it be documented? i. Has the time taken to respond to repair requests improved? ii. Has the quality of the repairs improved? iii. Has the hours of available electricity per day improved? iv. Has the average level of load-shedding been reduced? v. Has voltage reliability been improved? vi. Has the amount of time to get a new electricity connection been reduced? 1.By how much? vii. Has the cost of a new connection changed as a result of DRUM? e. Have village communities, co-operatives, and private entrepreneurs participated in managing the business of rural power distribution? i. If so, how? Can this be documented? ii. If so, to what effect? Can this be documented? f. Are the demonstration project sites sustainable? i. How is sustainability measured? g. Have the demonstration project sites resulted in increased access (quality and quantity) of electricity for the service area? i. If so, how do you know? 4. Regarding the pilot projects carried out by the targeted “best-practice” DISCOMS, what role has Tetra Tech played in “supporting” BESCOM’s smart-grid project? 5. Regarding the studies and reports41 a. Were these studies useful? prepared under the APP (i.e. the Market Development of Renewable Energy component): b. How do you know? c. How were they put to use? 6. Regarding the MDRE courses, do we have pre- and post-tests to determine level of learning? 7. What does “support” to the APDRP (Accelerated Power Development Reform Program) cell within the MOP mean? 8. Did the APDRP implement its distribution reform strategy? What are the other programs implemented by the state like jyoti Gram etc? 41 Model project report for a 5 MW Solar PV Project in Andaman and Nicobar Islands; Strengthening for State Nodal Agencies; Design of Policy Package, Development of Incentive Schemes and Identification of Intervention Areas for APTRI, Hyderabad; EVALUATION OF DRUM AND WENEXA 63 WENEXA General: 1. Why was WENEXA –II chosen in Karnataka site? 2. Why was not continued in Andhraprdesh where first phase of the project implemented? 3. WENEXA –II was approved in 2004 and was implemented in 2010(Quarterly reports) 4. WENEXA-II End date : July, 2011? 5. Pending work, if any, as of now (e.g. Watershed work)? 6. Baseline electricity consumption data what are the results? 7. Is there any baseline data on farmers ,their crop practices, and their socio-economic 8. Conditions before and after the project DRUM/WENEXA? 9. What were the major challenges to the implementation of WENEXA? 10. What was the cause of the delays? Would you have done things differently knowing what you know now? 11. Could WENEXA be implemented in Maharashtra? Is MERC interested? Component 1: Policy Dialogue 1. Was an energy/water policy framework developed? a. If so, what has been done with it? 2. What does “support” for sustainable water resources management mean? a. What has taken place? b. What has changed as a result of the support? 3. Legal framework for the Water Energy Policy a. Is there a State water Policy and its relevance to energy / water nexus? b. Is there any legal backup for the new policy framework esp. ground water management ? c. What happened to the model ground water Act developed by the national government and sent to states for implementation? d. How will the WENEXA experience feed in to Ground Water Regulation Policy in the state? 4. Cost Recovery issues: a. Has the policy regarding the pricing of water, especially with regard to electricity charges, (and surface water charges) changed as a result of DRUM/WENEXA due to reliability in electricity supply b. Are the farmers are willing to go for metering of electricity? c. Estimate of overall savings in energy cost due to WENEXA? Component 2: Sector Reform 1. What does “site-based support” for improving the reliability and quality of energy and water delivery in the agriculture sector mean? a. What specific activities have been accomplished? b. What has changed as a result of the support? c. Has reliability and quality of energy delivery improved? d. Number of Hours of electricity supply before and after WENEXA? Has reliability and quality of water delivery improved? 2. Has electricity efficiency improved? Specifics? 64 EVALUATION OF DRUM AND WENEXA a. Is there any difference in electricity supply between in Head and Tail end of feeder line? b. Reduction in the pump set burning? c. Has water use efficiency improved? Specifics? d. Changes in cropping pattern/productivity and income of farmers 3. Has basin-level hydrological information been collected and analyzed for target areas? a. If so, how was it useful and its impact - what has changed? 4. Impact of Watershed works in the area on ground water recharge and reduction in electricity demand/consumption a. Level of Ground Water increased ? 5. Progress and benefits in replacement of energy efficiency pump sets for old pump sets? 6. Have waste management practices changed as a result of the project? a. If so, how? 7. Have pollution reduction actions been taken in the industrial sector? a. If so, what? b. To what effect? Component 3: Customer Service 1. Has a communication and participation strategy been developed? a. If so, what has been the outcome? 2. What does “grassroots institution building” mean? a. Did these institutions promote sustainable water resources management? b. If so, how? c. Role of NGOs in the building of gross root institutions and promoting sustainable water resources management? d. What kind of institutions and their role in changing the farmers behavior in water management e. Results of customer survey in Agriculture sector in BESCOM EVALUATION OF DRUM AND WENEXA 65 Annex 5: Training Survey DRUM Training Questionnaire Exit this survey * 1. Social Impact, a US-based consulting firm, has been hired to evaluate the Distribution Reform and Utility Management (DRUM) project. As one of the people who received training under DRUM, we would like to ask you a few questions about your experience. We will not ask you your name, and your answers will remain anonymous. However, your honest assessment of the program may be useful in designing future electricity sector capacity building programs. Company: * 2. When you took the DRUM course, were you in a management position? Yes No * 3. Are you currently in a management position? Yes No * 4. What course, or courses, did you take under the DRUM program? * 5. Under which training institution(s) did you take the course? APTRANSCO BBMB CENPEID/NDPL CIRE CPRI ESCI FBVPL IEEMA IGNOU IIT MDI 66 EVALUATION OF DRUM AND WENEXA MSEDCL NPC NPTI PMI REMI TERI TNEB YASHADA YMPL Other (please specify) * 6. I what year(s) did you take the course(s)? 2004 2005 2006 2007 2008 2009 2010 * 7. How would you rate the quality of the course? Sub-standard Good Excellent * 8. Do you use what you learned in the course From time to time Rarely Always Frequently * 9. What was the single most important thing you learned from the DRUM course(s)? EVALUATION OF DRUM AND WENEXA 67 * 10. Specifically, how do you use what you learned in the course in your day-to-day work? Powered by SurveyMonkey 68 EVALUATION OF DRUM AND WENEXA INTRODUCTION The Internet survey was carried out Social Impact to evaluate impact of training conducted under the Distribution Reform, Upgrades and Management (DRUM) project. The questionnaire was sent to 350 people, out of which 76 contacts were invalid and 37 people have responded (13.5 percent). The survey was carried out from March 12-24, 2011. The total number of people trained under DRUM is 33,135. This survey is not scientifically valid because the sample size is too small and because it is limited to those participants with email addresses. It is therefore, not statistically representative; however, given the resource limitations and time constraints of the evaluation, the survey does provide useful anecdotal evidence of the impact of the DRUM training. RESULT ANALYSIS: • The majority of respondents (about 70%) were in management positions, which is not surprising as this segment is the most likely to have email addresses. The number of people attending the training increased every year from 2005-2010. Year No. of People attended the training 2005 1 2006 4 2007 3 2008 8 2009 12 2010 19 Table 7: Year and number of people attending DRUM training • The respondents attended the training at a number of training centers, with NDPL/CENPEID and MDI accounting for the largest number of respondents. Training Center No. of People attended the training CENPEID/NDPL 18 CIRE 4 NPTI 1 PMI 1 REMI 2 SURVEY CONDUCTED FOR THE ASSESSMENT OF TRAINING PROGRAM CONDUCTED UNDER THE PROJECT DISTRIBUTION REFORM, UPGRADE AND MANAGEMENT PROGRAM (DRUM) EVALUATION OF DRUM AND WENEXA 69 ESCI 1 IEEMA 1 MDI 14 IGNOU 1 Table 8: Location and number of people attending DRUM training • According to the respondents, the most important topics covered included: Methods for Distribution Loss Reduction, Distribution System Management, TATA Code of conduct, billing processes, metering processes, ethics, customer care, commercial best practices, GIS information in distribution, supervisory control and data acquisition (SCADA), and Automatic Metering System. • 94.5 percent of the respondents felt the quality of the training was good or excellent, and nearly one-third of use what they’ve learned “all the time,” while another third use what they learned “from time-to-time.” Figure 8: Quality of DRUM training • Out of the total 38 responses, 31 strongly expressed that they use the knowledge they have gained in the training in their day today work. Figure 9: Usefulness of DRUM training 0 5 10 15 20 Sub standard Good Excellent 0 10 20 30 40 Total no. of people attended training Total no. of people found taining useful in their day today work 70 EVALUATION OF DRUM AND WENEXA • Respondents were able to list specific things they learned in the DRUM training that they used in their jobs, such as: o Adaption of Automation & GIS in Distribution sector. o New technologies o How you help a person in an accident within 3 min. o Visionary leadership o Smart Grid and reforms process of DISCOMS o Expansion of thinking o Global exposure of power distribution business o Personal development o Change of perception and ability to grasp the industry environment at national level o Management concepts o The present scenario in electricity and the steps that should be taken for energy conservation and reduction of energy intensity o Process of Distribution Loss Reduction and Assessment o Hands on training to handle a safety situation under difficult conditions o In depth understanding of general management aspects of business along with exposure of specific issues and policy matters related to energy sector. o For sustainable growth of the power utility you need to reduce distribution losses and maximize revenue collection o Change Management o Customer Care Processes o Management & Leadership in Business Organizations o Confidence building in project appraisal and project report building and dealing in legal complications in contracts o Loss reduction in terms of efficiency, and loss reduction in terms of consumer satisfaction. • One outlier noted: “I don’t think that there is useful material is available to learn in DRUM program.” • In terms of using what they’ve learned, trainees responded: o “I facilitated the EAG group for calculating the AT & C Loss calculations using GIS based DT-Wise consumer Indexing.” o “GIS training is helps a lot in planning of the electrical network and safety helps how lived in the world with caution against accidents” o “Consulting of Sick DISCOMS” o “Positive and innovative approach for problem solving” o “Management methods and concepts learned are being used in day to day work” o “I use the Principles of Quality Management in day to day work.” o “Electrical safety procedure most frequently used by me in daily work” o “Very useful in day to day working, specially non technical matter such as finance and HR etc.” o “Practically implementing the various methods used for loss reduction as well as maintaining good quality supply to the consumers” EVALUATION OF DRUM AND WENEXA 71 o “I study the difference the change will bring and benefits associated with it, and try to implement those changes by taking everyone into confidence and support.” o “Presently I got posted in vigilance dept so the lecture on theft detection is very much usable for day to day use.” o “How ICT will help in reforming the electric value chain system” o “Project evaluation, Management planning, Corporate meetings, business analysis of various scheme, strategy for upcoming projects, etc.” o “During fault analysis and maintenance work.” • However, not everyone felt what they learned was useful, either because they aren’t in a decision making role, or because they are working for a state-owned enterprise and therefore change comes from the outside. 72 EVALUATION OF DRUM AND WENEXA Annex 6: Gender Assessment ASSESSMENT OF ISSUES AND OPPORTUNITIES TO MAINSTREAM GENDER IN THE DISTRIBUTION, REFORMS, UPGRADES, AND MANAGEMENT (DRUM) PROJECT In the three project sites explicit attention has not been given to energy service needs of women (especially poor women) as well as the requirements of men, and to mainstream relevant gender issues in power distribution within the scope of the DRUM project. Specifically, as: (f) Flags concerns particular to women and men in power distribution in the three project sites of Umreth, Gujarat and Aurangabad, Maharashtra, Bangalore. (g) Identifies opportunities for strengthening commercially viable models of power distribution based on improved customer satisfaction and revenue collection (h) Identify role of women as a consumer-Demand side management (i) Focus on Capacity Building as part of the training component-Utility Employee (j) Provide some direction on how to integrate gender concerns into planning procedures and program implementation within the DRUM project. The gender assessment is based on a literature review that informed visits to two DRUM sites in Umreth, Gujarat and Aurangabad, Maharashtra. The purpose of these trips was to conduct a baseline assessment at one urban and one rural DRUM pilot site and formulate recommendations to mainstream gender considerations that will help boost overall project effectiveness and sustainability. 2. BACKGROUND 2.1 Gender and poverty issues relevant to power supply Like race, ethnicity, and class, gender42 is a social category that may largely establish one's life chances and participation in the economy and society. Men and women’s particular responsibilities in society are often the result of their different roles. In many societies, men’s activities center around productive and community functions. By contrast, women’s functions include productive and community as well as reproductive work.43 42 “Gender is a system of socially defined roles, privileges, attributes and relationships between women and men, which are not determined by biology, but by social, cultural, political and economic expectations”. 43 Kunst and Kruse, 2001. EVALUATION OF DRUM AND WENEXA 73 Box 1: Definitions of Productive, Reproductive, and Community Activities Productive: That work which is remunerated in the market economy or in other systems of trade and bartering within the community. Reproductive: That work which centers on caring for the family and domestic work. These activities, though they enable families to engage in productive activities, are largely unremunerated and therefore often overlooked / underestimated. Community: These activities reinforce community relations and life, such as weddings and festivals. In rural communities, this may also involve labor or other goods exchange. Labor arrangements, such as working on neighboring fields in exchange for similar help on one’s own field, are particularly common among women. In addition, resource sharing among household is also practiced, such as with excess fodder or drinking water during periods of shortage. Some ways in which electricity can contribute towards women performing their practical, productive and community functions better are as follows: Box 2: Uses of electricity for different purposes Practical Productive Community Pumping water: reducing need to haul and carry Mills for grinding Lighting improves working conditions at home Increase possibility of activities during evening hours Power for specialized enterprises such as hairdressing and internet cafes Make streets safer: allowing participation in other activities (e.g. evening classes and women’s group meetings) Open horizons through radio, TV and internet Table 9: Women’s uses of electricity Several consequences arise from men and women’s different roles in society. First, men’s greater involvement in the market economy may give them the ability to have greater access to and control of income. This may in turn boost their decision-making power within the household. Men’s ability to control income, power, and decisions in the household can both support and be supported by prevailing cultural norms. In many households, men therefore control decisions on investments made for improving energy services (e.g. inverters, solar lamps, etc.). This is particularly relevant when it comes to investments that are likely to assist specifically women in performing their daily chores, such as kitchen appliances, and light points in kitchens. Second, women’s labor and work contributions are often unpaid and unrecognized. This means that women’s interests are often in reducing drudgery and saving time. Resultant time-savings, though important in and of themselves to give some respite from heavy workloads, may also be used for productive activities and boosting family income. Third, women and men may have different uses for energy within the household. In both urban and rural areas, the mechanization of household tasks and water collection can save time, with positive implications for women’s health, their ability to look after their families and participate more fully in decision-making in their communities, and their potential to earn income. In rural areas, energizing productive activities for which women are typically 74 EVALUATION OF DRUM AND WENEXA responsible, such as agricultural processing or water collection, can also free women from drudgery. It is not surprising, therefore, that women see the primary benefits of energy as reducing workload and improving health while men perceive increased access to energy in terms of better education for their children, leisure, and quality of life.44 Finally, the provision of electricity may have different impacts on men and women. The evaluation of a rural electrification project in Tamil Nadu showed that men benefited more than women since the electricity was used to run irrigation pumps substituting for oxen￾drawn water and the care of the oxen was traditionally a task for men. They gained more free time when the number of draft animals decreased, which they used for involvement in politics and improving their agricultural methods, thereby increasing their social and human capital. However, electricity did not substitute for any of women’s tasks. Sometimes, access to modern forms of energy can have unforeseen positive social benefits. For example, once electricity is introduced, women have been found to benefit in terms of their self-esteem from access to television. In addition to gender, poverty and economic class are also important variables in determining access to energy supply. The poor, unable to invest in inverters, solar lamps, or battery powered lanterns, or other technologies to mitigate poor quality power supply may be particularly hard hit. With the additional convenience afforded by modern energy options, there are strong incentives to invest in such technologies. However, the poor’s ability to do so is circumscribed by their purchasing power. The consequences for the poor are that precious cash resources are used on low quality fuels, such as kerosene, oil, or dung, which are then used at low efficiency, reducing poor families’ ability to accumulate the financial resources they need to invest in strategies for improving their livelihoods. The implications of reduced energy consumption for poor women imply an increase in drudgery and their burden of work. When considering improvements in the distribution of power and customer satisfaction that can form the basis of commercially viable utilities, it is useful to ask specifically what men and women mean by better quality service from the utility. Different consumer groups may place importance on various different aspect of power distribution and supply, whether timing, regularity, duration, or frequency of power cuts and voltage fluctuations. Women and men, with particular uses for energy, may well have differing views on what constitutes improved power distribution and how to better it. A gender assessment can help provide a comprehensive picture of different users’ needs and view to inform project priorities and activities. Site visits to both Aurangabad and Umreth show that women and men do have different uses for electricity. Women currently use electricity for pumping and storing water for domestic purposes, household appliances, extending their productive hours (through lighting in the evenings), and running small businesses. Men use electricity primarily for productive uses such as pumping water for irrigation, agricultural processing and farming, small businesses, and also entertainment. 4.a Aurangabad 44 Clancy, J., Skutsch, M., and Batchelor, S. EVALUATION OF DRUM AND WENEXA 75 Aurangabad, like many Indian cities is rapidly growing in both population and size. In the 1980s, Aurangabad was one of Asia’s fastest growing cities due primarily to industrial development. There are approximately 160 slums and unauthorized settlements that contribute 40 percent of the city’s population. The 109 identified unauthorized settlements mushroomed with increased industrialization. Many do not have access to basic services such as piped water supply. The Aurangabad Municipal Corporation (AMC), Maharashtra Industrial Development Corporation (MIDC), and the City and Industrial Development Corporation are responsible for water supply, sewage and waste management, roads, streetlights, and overall urban planning. The Maharashtra State Electricity Distribution Company Limited (MSEDCL) is responsible for providing electricity to Aurangabad. The situation of power distribution is characterized by load shedding and unscheduled power cuts, voltage fluctuations, and energy infrastructure in need of upgrades. Distribution losses are at about 55 percent, with 15 percent due to technical losses and 40 percent due to theft. Therefore, billing, metering, penalizing theft, and customer relations and awareness are particularly important in Aurangabad. Load shedding and power cuts Maharashtra’s state policy of load shedding is publicized through local media to assist people in planning around interruptions in power supply. However, there are several problems with the load shedding schedule and timing that pose a particular hardship for women. First, load shedding during critical morning hours conflicts with women’s peak time for electricity use to accomplish household work. These interruptions in power supply are especially difficult for women, who are juggling both household and employment or business responsibilities. This results in many of these women working longer hours each day to make up for the time lost in the morning. With continuous power supply, women believe they could engage in income generating activities such as tuition classes, beauty parlor, sari pikko, stitching, papad making, ironing, masala grinding, etc. Second, load shedding time sometimes conflict with municipal water delivery. Municipal water supply occurs every other day for approximately one hour. During this critical time, women collect and store water for two days’ domestic use. However, breaks in the power supply lead to interruptions in the municipal water supply as well. Those families who are well off can make provisions to buy water from private tankers for Rs. 1,000 per month. However, poor women who cannot afford this convenience must forage for water from nearby bore wells to gather water for their family. To collect two days’ water for a family from bore wells, women spend at minimum 2 -3 hours. This is an example of how poor women bear a disproportionate burden of interruptions to power supply. With no capital to invest in services to surmount the inconveniences of power and water supply disruptions, poor women are forced to spend their time foraging for and hauling water back to their families, an extension of their already over-burdened days. Apart from load shedding, there are also unscheduled power cuts for which it is impossible to plan. Poor families in general do not have the wherewithal to invest in technologies such as inverters and generators that can make up for regular or unscheduled lapses in power supply. A common complaint of mothers from these families focus on the negative impact of power cuts on their children’s education and ability to use evenings to study. Again, the 76 EVALUATION OF DRUM AND WENEXA problem of power supply disruptions disproportionately affects children in poorer families, who cannot afford to invest in auxiliary technology. Small businesses also suffer because of breaks in power supply. Women small business owners interviewed say that their business is operating at 30 – 40 percent below capacity because of inadequate, unreliable electricity supply. Although both women and men own small businesses, it may be that women are at an added disadvantage for several reasons. First, they lack the ability to take decisions to spend family savings on technologies such as inverters or generators that could reduce interruptions to their business operations. Second, women face time constraints that men do not have. Unlike women, men are able to work into the evening and night to make up for time lost to breaks in power supply. Women, with household and childcare responsibilities, are less able to work late into the evening. Voltage fluctuations Women’s primary complaint with voltage fluctuations was that they burn their household appliances and motors for pumping water. This adds an additional economic burden. The degree of this problem’s severity, however, is not clear. It may be that the cost of burnt out appliances and motors is more than the cost of paying for electricity that is of reliable quality. Customer care and service In Aurangabad, bill payment is usually done at the utility’s customer service centers. However, some centers are not open a full business day, with some closing as early as 2 pm. This can be inconvenient for customers, particularly for women who work and have household responsibilities. This system of bill collection may be made more convenient by longer hours during which customer service centers are open, providing different outlets where bills may be settled, or billing through post. Power theft is a major problem throughout the system, with people tapping into to low￾tension lines. In addition, power failures are commonplace. Women and men both lodge complaints about power failures over the phone. Usually problems were attended to within 2 – 3 hours; however, it can take much longer for more serious power failures. Men will sometimes stage public demonstrations when complaints are not attended to in a timely manner. MSEDCL argues that it faces the problem of lack of staff to monitor and punish theft. In addition, it does not have sufficient supplies to maintain the infrastructure properly. Even though this particular issue does not have a direct gender implication, it affects the overall quality of supply, which affects the women significantly. Outsourcing meter reading and bill collection Women’s NGOs in Aurangabad expressed interest in being involved in meter reading, bill collection, spot billing, and also for reporting theft. They believe that with proper training, they can help improve MSEDCL’s billing and collection system. Gender balance within the utility EVALUATION OF DRUM AND WENEXA 77 MSEDCL reports that it has no formal policy on gender balanced hiring. It estimates that 5 – 10 percent of employees are women, mostly concentrated in lower paying administrative jobs. 4.b Umreth Agriculture and dairying are the primary sources of income for people in Umreth. Although the area’s power distribution, with a mixed feeder system and a high incidence of theft did not always function well, recently electricity supply has greatly improved with the Jyoti Gram Scheme (JGS). Feeder lines for agriculture and residences are now separate, with residences receiving electricity 24 hours a day. Agricultural electricity customers, who use energy primarily for pumping irrigation water, receive two eight hours shifts of power, and pay at subsidized rates per horsepower. A local lineman serves as a liaison between the community and utility as well as responds to customer complaints and disruptions in the power supply. Madhya Gujurat Vij Company Ltd. (MGVCL) is responsible for the area’s electricity supply. Women’s NGOs, SHGs and cooperatives are active in the area and engage in dairying as a major income generating activity. These groups are becoming an increasingly powerful and potentially an effective vehicle for women’s income generation and empowerment. Impact of increased energy access In Umreth, women are increasingly relying on household appliances, such as refrigerators (60 percent) and washing machines (10 percent). In addition, agricultural processing for which women are traditionally responsible is becoming increasingly mechanized. In the 6 months that JGS has been implemented, women have been able to avail of regular electricity supply and saved time on household tasks. Increased productive hours through lighting also give women more time to spend on both income generating and household responsibilities. Now, women report that they have more time for income generation activities such as dairying, child and elder care, and rest. Women’s SHGs are exploring opportunities in the area to open small business such as tuition or tutorial centers or other small shops. One complaint that women had, however, is that streetlights are not properly maintained. For safety reasons, women value community and street lighting. The Gram Panchayat does not maintain streetlights properly. However, this is not a consequence of power distribution, but rather local government priorities. Suggestions for improving MGVCL operations through women’s participation MGVCL has a program to boost energy conservation and efficiency among the local community. Utility officials propose that women’s SHGs take the lead in raising awareness on energy conservation and efficiency in schools and other community venues. In addition, officials proposed that women’s cooperatives be at the forefront of campaigns to discourage electricity theft and create social pressure not to steal power. Currently in Umreth, men are primarily responsible for paying electricity bills and making complaints to the local lineman, whereas women report that they do not know as much about metering and billing. However, MGVCL experience shows that, when women are involved in mobile teams to collect fees and arrears, mixed gender teams are more successful 78 EVALUATION OF DRUM AND WENEXA than men only teams. This, they report, is because female utility representatives are more able to reach out to women in the community. Women, the utility reports, are more likely to pay arrears and other fees than their husbands. Utility officials also suggested that women’s SHG members be trained to read meters, collects fees, do spot billing, and control theft. This, they believe, would lead to better collection rates of fees and revenues. As part of the JGS, consumer committees have been created to inform MGVCL functioning and policy. However, women do not participate in those committees. Given women’s different needs and uses for energy, women’s NGOs in the area recommend that women also be encouraged to participate in consumer committees. Gender balance within the utility In Umreth, MGVCL reports that there is no policy on gender balanced hiring. It reports that the number of female utility employees is a small percentage of the total, and that these workers are concentrated primarily in lower paying administrative jobs. V. SYNTHESIS AND RECOMMENDATIONS Table 1: Commonalities and differences between Aurangabad and Umreth Aurangabad Umreth Urban site Rural site Scheduled load shedding and frequent power cuts Uninterrupted power supply under the Jyoti Gram Scheme Interrupted water supply as a result of power cuts Reliable water supply Time lost and burden of work increased due to power cuts Increased time and efforts towards women’s income generation • Women’s NGOs and SHGs willing to work on O&M of last mile infrastructure and billing • Women’s NGOs and SHGs at forefront of conservation and awareness raising • High use of domestic appliances • No institutional policy on gender in both utilities A few women in low administrative posts. No women in decision-making positions. • Power thefts a serious problem in both locations Table 10: Commonalities and differences between Aurangabad and Umreth One of the objectives of the gender assessment was to define how women and men define “quality” power supply. In both the locations, it emerged clearly that women value the following elements of quality: -Steady voltage -Load shedding is acceptable, but must be pre-announced and predictable. -Reliable power supply during the times when women’s workload is the highest -Reliable power supply for water supply. EVALUATION OF DRUM AND WENEXA 79 The primary benefits that quality power supply can offer women include time saving, flexibility to organize household chores by extending working hours and income generation. Unfortunately, there is little empirical evidence on what women and men actually do with time saving that electricity is known to bring about. In general, it appears that when time￾savings do take place because of introduction of energy saving devices, men use these savings primarily for recreation and leisure, whereas women redirect them to other household chores. On the whole, time-savings from electricity do not reduce the overall workload of women, although they make work easier. In Sri Lanka, women consider that lighting gives them about two extra hours of useful time, which are invested not only in better housework and care of the children, but also in time to rest, socialize, and watch television, and sometimes to develop income-generating activities (Matly 2003), significant benefits. Surprisingly, most recommendations are common to both Umreth, a rural site, and Aurangabad, an urban site. However, concerns and recommendations about coordination between different municipal utilities are particular to Aurangabad and an urban setting. 5.a Aurangabad Better coordination between utilities One of the most important findings from the gender assessment underscores women’s need for timely power to supply them with water for household use and consumption. Many women’s major complaint is that power cuts and the time for municipal water supply often coincide, often leaving women without access to water for four days at a stretch. Their customer satisfaction would therefore depend heavily on whether power is available during times of municipal water supply so that they may save time and drudgery in collecting and storing water. It is recommended that, in Aurangabad, agencies responsible for water supply and MSEDCL improve coordination so as to avoid load shedding or power failures during municipal water supply delivery. 5.b Overall recommendations Enlist women’s NGOs and cooperatives in franchise opportunities In both Aurangabad and Umreth, women’s NGOs and cooperatives expressed interest in assisting electric utilities in meter reading, billing, and spot billing. In Umreth, utility officials themselves suggested that women’s NGOs could be successful in improving collection rates. Involving women’s NGOs and cooperatives in infrastructure projects is a common practice in India, with many involved in the operation and maintenance of water supply infrastructure and systems. It is therefore anticipated that women’s NGOs would be able partners in boosting utilities’ ability to recover arrears and fees. NGOs and cooperatives, as community based organizations, would be perhaps more effective and efficient than an outside entity such as a utility or other private company. Women’s cooperatives and Self Help Groups (SHGs) are particularly strong and well￾established in South India. Annex 1 describes the history and function of women’s SHGs. At the Doddaballapur site, there are commonly upwards of 3 SHGs per village. Most have collected savings ranging between Rs. 50,000 – Rs. 1,00,000 and are active in income generation activities. Some SHGs are holders of government contracts to provide school lunches, while others focus on dairying and agricultural activities. All SHGs in the area are 80 EVALUATION OF DRUM AND WENEXA linked to banks and maintain proper records and accounts. Therefore, perhaps the Doddaballapur site could be considered in piloting an initiative to involve a women’s SHG in billing and fee collection for electricity. Encourage women’s participation in customer service and awareness campaigns Men and women, as has been discussed earlier, may have different needs and uses for electricity. Consequently, men and women often have different perceptions of quality energy supply and customer satisfaction. Involving women in customer service committees could help address issues in power distribution more comprehensively. Women’s NGOs and SHGs typically command strong networks for outreach and raising awareness. These NGOs and SHGs often have links to schools, health care facilities, religious groups, and other community based institutions. Therefore, Umreth utility officials’’ suggestion that women’s organizations be involved in campaigns to raise awareness about energy conservation and efficiency is a sensible one. With the proper training and materials, women’s NGOs and SHGs could be effective vehicles for communications and outreach on energy conservation and efficiency. Organizations such as Janarth in Aurangabad, JagritiMahilaSangathan in Umreth may be good candidates to assist in awareness raising campaigns. Promote gender balanced hiring in utilities Within utilities, efforts should be made to boost women’s representation in utilities, particularly at the officer level where decision-making occurs. Notwithstanding the fact that inclusion of women at decision making levels does not guarantee that institutions will become gender-sensitive, inclusion of women in decision-making may open space for different ideas and concerns to be addressed, particularly those relevant to social protections and women’s burden of work. A critical minimum mass of women also is a starting point for engendering organizations and their policies. In practical terms, what this means is not relaxing the criteria for selection, but strongly encouraging women to apply, by creating setting in place definite policies and mechanisms that help women to operate in this sector. Participatory research in Asia, a Delhi-based NGO has 50% women as staff. This has been achieved after years of providing support to women through personnel policies, such as providing office vehicle after late work-hours or arrangements for picking up and dropping off at bus- and railway stations etc. Women staff feel that these actions have been extremely successful in providing real support where it is needed and enhancing the confidence levels of women staff. Gender sensitization in capacity building for utilities Mainstreaming gender within capacity building programs underway for policy makers, regulatory staff, engineers, and utility staff can address the impact of electricity supply on women and men. In addition, training to enable energy sector policy makers and practitioners to identify future issues in power distribution will build local capacity to address poverty and gender concerns that could arise in the future. 6. CONCLUSION EVALUATION OF DRUM AND WENEXA 81 Gender considerations can inform policy makers and planners about the range of needs for electricity services among a population. As stakeholders with different roles in society, men and women may have particular expectations and views of quality power distribution. In particular, women view electricity as a means to save time and reduce drudgery, thereby freeing them for other income generating and household activities. Chief among women’s concerns in several DRUM sites is the use of electricity for distributing, pumping, and storing water for domestic use. In areas of unreliable power distribution, women are particularly dissatisfied with the utility’s services. Therefore, better coordination between water and energy utilities, particularly in urban areas, is important. In rural areas, this “water and energy nexus” is also important, as unreliable electricity supply increases women’s burden of work and time spent in collecting water. Women’s organizations and cooperatives may be helpful in promoting communications and outreach strategies to educate people about energy conservation and efficiency. In particular, women’s groups may be a strong force for building societal pressure to reduce theft. Perhaps most importantly, women’s SHGs and NGOs, to the extent that they are community-based organizations, may be effective in assisting utilities in billing and collecting fees. Training and capacity building is needed to build power sector policy makers’ and practitioners’ ability to integrate gender considerations in their current work and identify issues that may arise in the future. With improved access to electricity, it is hoped that women and men will have improved opportunities to create income. It is hoped that women in particular will use these opportunities and the time saved because of access to electricity to contribute more productively to their families and communities. ANNEX A Key questions for baseline gender assessment related to disruptions in electricity services for the project sites: The gender assessment will consist of informal interviews/ meetings with partners, members of the DRUM team (local level), counterpart companies, local NGOs working with women issues, community members and others, as appropriate to flag issues of particular concern to women in the provision of electricity services. Through discussion with both utility officials and customers, it is hoped that this gender assessment will identify whether women have a different understanding of 'quality' and expectations from high quality electricity supply than men. In addition, it is hoped that avenues for involving gender concerns will contribute to increased project effectiveness. For example, women may be willing to pay a premium if high quality power supply can be assured. In addition, involving women in specific consumer-end tasks (complaints) may enhance quality of service and be a mode for women's empowerment. Key questions at the community level include: What are women and men’s electricity consumption patterns? How does access to electrification /improved electricity services (which ever is the case) influence women? What are the current and potential uses of electricity from the point of view of women users? What are men’s views regarding these applications? 82 EVALUATION OF DRUM AND WENEXA What would women do with time saved (if any and how) from reliable, constant electricity supply? How are women, who rely on energy to increase productivity of domestic and income generating work, affected if they are subject to frequent residential power cuts? What income generating activities are women and men involved in? From which activities do women have some control over income (farm based, handicraft, food-processing, cottage? What do women do in case of power cuts? What is their level of awareness, for lodging complaints dealing with the utility, etc? Is there an economic burden of disruption in power supply on the women in terms of availability of water, health, work days lost etc. Do disruptions in electricity supply affect children’s (especially girl children’s) education and their school attendance? Do disruptions affect the health of family members or their access to health care services? How does quality improvement w.t.o. electricity supply affect men and women’s daily activities? How do changes to the structure of user fees and for electricity affect the household? . What local level institutions are women involved in and why? What are opportunities for strengthening and enhancing women’s participation in at the local level and entry points into the community to improve electricity services. What are local level suggestions about what role women can play to improve energy services? Do they need training and support of local NGOs for improved participation? Are there existing village level institutions that can play a more active role (e.g. women’s SHGs)? What is the present paying mechanism for electricity bill collection? What is the role of women (housewife etc) in this case? About how much is your monthly bill, and how regularly is it paid? Are there opportunities to empower women through energy entrepreneurship? If women had improved access to electricity, what are the potential applications that women see as beneficial? For what activities do they prefer to use electricity vs. other energy sources (cooking, agricultural processing, lighting, income generation)? Do disruptions affect / aggravate the problem of willingness to pay: • To what extent people (men and women) are willing to pay higher tariffs and installing of meters (especially in rural areas where the power cuts are longer) if better quality of service can be assured? If yes, how much and for what quality of service? EVALUATION OF DRUM AND WENEXA 83 • What other improved distribution systems can be made feasible on sustainable basis? Empowerment and decision-making • What is the role of women in decision-making within the household and outside? • Do women contribute in decision making about payment of bills, etc. • What is the role of women in decision making about purchase of electrical appliances or other assets – TV, radio, etc? • Does the improved access to electricity lead to social/economic empowerment of women? What is the benefit of access to electricity for • Productivity (and income) gains • Agriculture • Education of children (girl child) • Entertainment • Health • Leisure • Socialization • Involvement and decision making in community activities What are women’s priorities for electricity use (Note done from 1- 6, with 1 being of highest priority and 6 being of lowest.) Activity Ranking Energy for pumping water (irrigation or drinking) Lighting for longer productive hours Lighting for safety Power for mechanized agricultural processing Electricity for household appliances Energy for leisure or information (TV, radio) Table 11: Women’s priorities for electricity use How important is timing of energy supply, particularly for women that have severe constraints on time and may depend on electricity supply during night and very early morning hours to increase the time available for them to complete their work. Key questions for utility officials and other partners: 1. Do they see a potential role for rural women (existing customers) in any aspect of distribution? If yes, in which aspect? Do women have the necessary skills required to handle these proposed tasks? What are the capacity building needs that will have to be addressed? 2. What are the organizational policies on recruitment of women? How many women are in the utility, in what capacity? Are there any special provisions for women workers? 3. What are their views on the impact of tariff rates and structures on the poor and vulnerable women￾headed households? 4. Do they understand that regularity and reliability of electricity is often important to ensure the reliability of drinking water supply? Do they understand that disruptions in electricity service and drinking water supply may be particularly difficult for women, who are primarily responsible for procuring water for domestic use? 84 EVALUATION OF DRUM AND WENEXA 5. How do they maintain a relationship with the community or customers? How long does it take to attend a complaint? Do women make complaints? 6. The power supplies options. The quality of supply may be important, for example, three phase power may be needed in areas where water distribution is dependant on electricity supply; however, the utility may supply more single phase electricity, meaning that water is not as readily available. 7. Do they understand the importance of reliable timing for electricity, particularly for women that have severe constraints on time and may depend on electricity supply during night and very early morning hours to increase the time available for them to complete their work or to gather water for the day?. 8. Do they value community lighting at night for safety reasons? 9. Do these see ways in which community members can contribute to the upkeep and maintenance of electricity infrastructure? EVALUATION OF DRUM AND WENEXA 85 Annex 7: References Barnett 2000, Energy and the Fight against Poverty: Energy and the "Empowerment" of Livelihoods, 3 August 2000 Elizabeth Cecelski 2002. Enabling Equitable Access to Rural Electrification: Current Thinking on Energy, Poverty and Gender, Briefing Paper. Asia Alternative Energy Policy and Project Development Support: Emphasis on Poverty Alleviation and Women Asia Alternative Energy Unit, The World Bank, Washington, DC, January 2000, Revised July 2002. Elizabeth Cecelski 2000. The Role of Women in Sustainable Energy Development. NREL/SR-550-26889, June 2000 Clancy, Joy. Gender and Household Energy - An Introduction to the Key Issues, Blowing the Smoke out of the Kitchen: Core Issues in Household Energy and Gender Clancy, Oparaocha and Roehr 2004. Gender Equity and Renewable Energies: Thematic Background Paper for renewable 2004, Bonn June 2004. Clancy J S, Skutsch M and Batchelor S (2003), The Gender - Energy- Poverty Nexus: Finding the energy to address gender concerns in development. Paper prepared for the UK Department for International Development (DFID), London. DFID Project CNTR998521. http://www.utwente.nl/tdg DFID 2002. “Energy for the Poor. Underpinning the Millennium, Development Goals” Department for International Development, August 2002 Barbara C. Farhar , Gender and Renewable Energy: Policy, Analysis and Market Implications. National Renewable Energy Laboratory, USA SadekaHalim 2005. Gender and Rural Electrification: A Case from Bangladesh. Paper presented in “Asian Regional Workshop on Electricity and Development”, AIT, Thailand. Organized by United Nations Environment Programme (UNEP), United Nations Development Programme (UNDP), and The International Energy Agency (IEA), 28-29 April 2005 Ines Havet 2003. Linking Women and Energy at the Local Level to Global Goals and Targets, in Energy for Sustainable Development, Volume VII No. 3, September 2003. Kunst, S. and Kruse, T. Integrating Gender Perspectives: Realising New Options for Improved Water Management. Cross-cutting Thematic Background Paper. Secretariat of the International Conference on Freshwater – Bonn. 2001. Michel Matly, M., France (2003). Rural Electrification in Indonesia and Sri Lanka: From Social Analysis to Reform of the Power Sector, The International Bank for Reconstruction and Development / The World Bank. Ramani, K., V. and Heijndermans, E. (2003) “Energy, Poverty, and Gender: A Synthesis”. Report submitted to The International Bank for Reconstruction and Development / The World Bank, Washington DC 86 EVALUATION OF DRUM AND WENEXA Srivastava, P. and Basu, P. Scaling-up Access to Finance for India’s Rural Poor. (A case study from Reducing Poverty, Sustaining Growth—What Works, What Doesn’t, and Why A Global Exchange for Scaling Up Success Scaling Up Poverty Reduction: A Global Learning Process and Conference Shanghai, World Bank. May 25–27, 2004. UNDP 2003, Gender and Energy: How is Gender Relevant to Sustainable Energy Policies, in Sustainable Energy Strategies: Materials for Decision Makers, United National Development Programme UNDP/ ESMAP 2003. Monitoring and Evaluation in Rural Electrification Projects: A Demand-Oriented Approach, July 2003 Woroniuk and J. Schalkwyk, 1998. Information Sheet on Mainstreaming Equality between Women and Men: Handbook on Gender Perspectives in Energy Sector Development. Published by the Infrastructure Division, SIDA, June 1998. Social Impact, Inc. 2300 Clarendon Boulevard Suite 300 Tel: (703) 465-1884 Fax: (703) 465-1888 www.socilaimpact.com