April 13, 2016 This report was produced for review by the United States Agency for International Development (USAID). It was prepared by International Business & Technical Consultants, Inc. (IBTCI) under Contract No.: AID-388- C-14-00001. MID-TERM PERFORMANCE EVALUATION FINAL REPORT Catalyzing Clean Energy in Bangladesh (CCEB) Accelerating Capacity for Monitoring and Evaluation (ACME) Activity MID-TERM PERFORMANCE EVALUATION FINAL REPORT Catalyzing Clean Energy in Bangladesh (CCEB) April 13, 2016 Previous drafts submitted on: April 5, 2016 March 19, 2016 Contract No.: AID-388-C-14-00001 (USAID/Bangladesh Accelerating Capacity for Monitoring and Evaluation – ACME) International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive, Suite 400 Vienna, VA, 22182 USA DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT TABLE OF CONTENTS ACRONYMS............................................................................................................................................................... iv EXECUTIVE SUMMARY.......................................................................................................................................... vi Purpose, Objectives, and Evaluation Questions............................................................................................ vi Methodology and Conceptual Approach ........................................................................................................ vi Reponses to the Evaluation Questions........................................................................................................... vii Evaluation Question 1: To what extent is the CCEB project on track, in terms of progress and outcomes, to meet its overall goals for the five tasks under Components A and B?......................... vii Evaluation Question 2: What plausible opportunities exist to enhance the project’s programmatic approach and effectiveness within the stipulations of the contract? ....................................................... vii Evaluation Question 3: What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability?....... ix Evaluation Question 4: How effectively has gender been integrated or incorporated in the interventions? What have been the challenges and opportunities, if any?............................................... x Lessons Learned..................................................................................................................................................... x 1. INTRODUCTION.............................................................................................................................................. 1 2. PROJECT BACKGROUND: THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE.......... 2 3. PURPOSE OF THE EVALUATION................................................................................................................ 3 3.1. Intended Audience ................................................................................................................................... 3 3.2. Evaluation Questions............................................................................................................................... 3 4. EVALUATION METHODOLOGY AND LIMITATIONS......................................................................... 4 4.1. Planning and Document Review ........................................................................................................... 4 4.2. Data Collection and Analysis................................................................................................................. 4 4.3. Biases and Limitations............................................................................................................................. 4 5. FINDINGS AND CONCLUSIONS................................................................................................................ 5 5.1. Findings - CCEB Overall Performance................................................................................................ 5 CCEB Risk Assessment Framework (RAF) and Risk Mitigation Options................................................ 6 CCEB Risk Mitigation Assessment..................................................................................................................... 7 5.2. Evaluation Question 1........................................................................................................................... 13 5.3. Evaluation Question 2........................................................................................................................... 15 5.4. Evaluation Question 3........................................................................................................................... 17 5.5. Evaluation Question 4........................................................................................................................... 18 5.6. Findings – Overall................................................................................................................................... 19 5.7. Findings – Task 1: Improve Regulatory Environment for Clean Energy Development......... 19 5.8. Findings – Task 2: Strengthen Analytical Capacity for Energy Sector Planning and Policymaking .......................................................................................................................................................... 20 5.9. Findings – Task 3: Promote Industrial Energy Efficiency and Conservation............................. 20 5.10. Findings Task 4: Adopt DSM Programs for Electric Utilities....................................................... 21 5.11. Findings Task 5: Conduct Market Analysis and Development for Improved Cook Stoves . 21 i CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Findings – CCEB support to market development ................................................................................. 21 Findings – ICS Manufacturers....................................................................................................................... 21 Findings – ICS Suppliers and Distributors................................................................................................. 22 Findings – ICS End Users .............................................................................................................................. 22 5.12. Conclusions – Overall........................................................................................................................... 23 6. RECOMMENDATIONS.................................................................................................................................. 25 6.1. Overall Recommendations................................................................................................................... 25 6.2. Task 1 Recommendations.................................................................................................................... 26 6.3. Task 2 Recommendations.................................................................................................................... 27 6.4. Task 3 Recommendations.................................................................................................................... 27 6.5. Task 4 Recommendations.................................................................................................................... 28 6.6. Task 5 Recommendations.................................................................................................................... 28 7. LESSONS LEARNED........................................................................................................................................ 30 7.1. Effectiveness: ............................................................................................................................................ 30 7.2. Sustainability: ........................................................................................................................................... 30 7.3. Program Improvement:......................................................................................................................... 30 Annex A: Gantt Chart of Activities...................................................................................................................... 31 Annex B: Evaluation Design Matrix...................................................................................................................... 36 Annex C: Data Collection Schedule and Data Analysis Methodologies...................................................... 38 Annex D: Key Informant Interview Questions.................................................................................................. 41 Annex E: Focus Group Discussion Guides......................................................................................................... 48 Annex F: ICS Survey Instruments......................................................................................................................... 50 ICS Survey Instrument – Manufacturers ............................................................................................................. 51 ICS Survey Instrument – Suppliers/NGO Community.................................................................................... 53 ICS Survey Instrument – Other Stakeholders................................................................................................... 55 ICS Survey Instrument – End Users..................................................................................................................... 57 Annex G: Quantitative Results Charts................................................................................................................ 60 Annex H: Document and Information Sources................................................................................................. 64 Annex I: Evaluation Methodology – Risk Assessment Framework (RAF) .................................................. 69 CCEB Risk Assessment Framework................................................................................................................ 69 Annex J: Interview and Contact Listing............................................................................................................... 72 Annex K: Final Report on ICS Survey ................................................................................................................. 77 ABBREVIATIONS..................................................................................................................................................... 78 TABLE OF CONTENTS......................................................................................................................................... 79 Preface......................................................................................................................................................................... 82 Executive Summary .................................................................................................................................................. 83 1. Introduction ........................................................................................................................................................ 86 2. Survey Methodology......................................................................................................................................... 86 2.1 Sampling.................................................................................................................................................... 86 ii CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 2.2 Training for Enumerators by DPC and ACME................................................................................ 87 2.3 Survey Instruments................................................................................................................................ 88 2.4 Survey Staffing Plan ................................................................................................................................ 88 2.5 Program used for data entry operation and analysis..................................................................... 88 3. Constraints and Limitations of the survey .................................................................................................. 88 4. Findings of the survey....................................................................................................................................... 88 4.1 Specific findings based on data analysis............................................................................................. 88 5. Recommendations............................................................................................................................................. 90 6. Lessons Learned ................................................................................................................................................ 91 7. Conclusions ........................................................................................................................................................ 91 References Tables..................................................................................................................................................... 93 List of Annexures: .................................................................................................................................................. 106 Annex L: Mid-Term Evaluation SOW............................................................................................................... 107 iii Addendum to the approved Catalyzing Clean Energy in Bangladesh (CCEB) Mid-term Performance Evaluation Report The Evaluation Question # 4 “To date, how effectively has gender been integrated or incorporated in the interventions? What have been the challenges and opportunities, if any?” under the CCEB Mid-term Evaluation Report could not completely and clearly capture how gender was effectively integrated and incorporated. Apart from citing M&E training data, which can also be gathered from the CCEB’s quarterly and annual M&E reports, the evaluation report did not provide sufficient information and evidence to back the statement: "There is informed evidence that CCEB has promoted gender integration within its activities." Furthermore, the response to gender integration appeared to be limited to only personnel trained. Also, a meager reference has been made on CCEB gender strategy call for support to the Council on Women in Energy and Environmental Leadership, without providing enough substantial information. Based on this review, USAID finds responses on question 4 to be inadequate. CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT ACRONYMS ACME Accelerating Capacity for Monitoring and Evaluation AEE Association of Energy Engineers BB Bangladesh Bank BERC Bangladesh Energy Regulatory Commission BPDB Bangladesh Power Development Board CAP Country Action Plan CCEB Catalyzing Clean Energy in Bangladesh CDCS Country Development Cooperation Strategy COR Contracting Officer’s Representative COP Chief of Party DCOP Deputy Chief of Party DESCO Dhaka Electric Supply Company DISCO Distribution Company DO Development Objective DO4 DO for Climate Change, Environmental & Natural Resources projects DPDC Dhaka Power Development Company DSM Demand Side Management EE Energy Expert EG Economic Growth Office FGD Focus Group Discussion GCC Global Climate Change GHG Greenhouse Gas GOB Government of Bangladesh IBTCI International Business and Technical Consultants, Inc. ICS Improved Cook Stoves ID Identifier/Identity IDCOL Infrastructure Development Company Limited IP Implementing Partner IR Intermediate Result KII Key Informant Interview KPI Key Performance Indicator M&E Monitoring and Evaluation NEEAP National Energy Efficiency Action Plan NGO Nongovernmental Organization NRM Natural Resource Management PMP Performance Management Plan iv CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT PRA Participatory Rural Appraisal PSPAM Power Sector Planning Analysis Model RAF Risk Assessment Framework SOW Statement/Scope of Work SPSS Statistical Package for Social Sciences SREDA Sustainable and Renewable Energy Development Authority TOU Time of Use US United States USAID U.S. Agency for International Development USG United States Government v CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT EXECUTIVE SUMMARY International Business & Technical Consultants, Inc. (IBTCI), under U.S. Agency for International Development (USAID)/Bangladesh’s Accelerating Capacity for Monitoring and Evaluation (ACME) activity,1 conducted a mid-term performance evaluation of USAID/Bangladesh’s Catalyzing Clean Energy in Bangladesh (CCEB) project.2,3 The CCEB project is intended to support energy sector development for energy security, contribute to economic growth and climate change mitigation, build capacity to design and implement supportive policies and regulations, and increase utilization of clean energy approaches and technologies for energy sector development on a low carbon trajectory. The CCEB components and the tasks are as follows: • Component A: Improve Enabling Environment for Low Emissions Development o Task 1: Improve Regulatory Environment for Clean Energy Development o Task 2: Strengthen Analytical Capacity for Energy Sector Planning and Policymaking • Component B: Increase Energy Efficiency and Conservation o Task 3: Promote Industrial Energy Efficiency and Conservation o Task 4: Adopt Demand Side Management Programs for Electric Utilities o Task 5: Conduct Market Analysis and Development for Improved Cook Stoves Purpose, Objectives, and Evaluation Questions The purpose of the mid-term performance evaluation was to assess the extent to which the CCEB project is on track to meet its overall goals and inform management of any challenges or opportunities that warrant adjustments to the project to ensure the achievement of the project’s intended results. The findings and recommendations from the evaluation are expected to be used to improve CCEB’s implementation and to inform the design of other relevant climate change, environmental, and natural resources projects. The mid-term evaluation aimed to answer the following four questions: (i) To what extent is the CCEB project on track, in terms of progress and outcomes, to meet its overall goals for the five tasks under Components A and B? (ii) What plausible opportunities exist to enhance the project’s programmatic approach and effectiveness within the stipulations of the contract? (iii) What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? (iv) How effectively has gender been integrated or incorporated into the interventions? What have been the challenges and opportunities, if any? Methodology and Conceptual Approach IBTCI used a utilization-focused, mixed-method evaluation methodology that included desk review of program-produced documents and related materials, key informant interviews (KIIs), focus group discussions (FGD), and a field survey of improved cook stoves (ICS) end users, which included visits to the Bagerhat, Chapai Nawabganj, and Gaibandha districts of Bangladesh. For each evaluation question, the team used multiple data sources to triangulate evidence to minimize biases and other limitations, (including recall, response, and selection biases). By synthesizing the information from multiple sources, no one singular data point could skew the analysis. 1 Contract number: AID-388-C-14-00001 2 On October 10, 2012, USAID/Bangladesh awarded a $14,990,150 contract to Deloitte Consulting LLP for implementation of the five-year CCEB program. 3 The scope of work for the mid-term evaluation is presented in Annex K of this document. vi CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT The evaluation team conducted 17 KIIs with 39 people, three FGDs with 15 industry representatives and members of the financial community, and surveyed 134 ICS end users.4 The informants were key CCEB stakeholders, including central government entities, private sector organizations, the CCEB implementing partner and local subcontractors, and program beneficiaries. A list of organizations can be found in Annex I. The evaluation team collated and analyzed the quantitative and qualitative information collected through the KIIs, FGDs, and survey to address the questions and inform evaluation findings, recommendations and lessons learned. Reponses to the Evaluation Questions Evaluation Question 1: To what extent is the CCEB project on track, in terms of progress and outcomes, to meet its overall goals for the five tasks under Components A and B? To evaluate the project’s progress and outcomes, the team asked key counterparts in each task to provide responses against a Likert scale to a series of questions.5 The majority of respondents believe CCEB is making progress and is on track to achieve its goals related to Tasks 2, 3, and 5. Progress in Tasks 4 and 1 is less clear. The counterparts associated with Task 4, Dhaka Electric Supply Company (DESCO) and the Dhaka Power Development Company (DPDC) indicated they see the prime driving force for widespread adoption of DSM to be a fully functioning BERC and while both organizations spoke highly of efforts taken by CCEB to try to facilitate DSM activities, they consider it may be difficult for the CCEB to achieve its goal of increasing use of demand side management (Task 4) given the lack of sector regulation and supportive government policies. Similarly, the Task 1 work with the Bangladesh Energy Regulatory Commission (BERC), has made some progress but is still a long way from providing sector regulation to an internationally acceptable standard. Evaluation Question 2: What plausible opportunities exist to enhance the project’s programmatic approach and effectiveness within the stipulations of the contract? The evaluation team found a number of opportunities to enhance programmatic approach. They include: • Overall: CCEB should continue to align all activities with relevant national goals and objectives, especially the 2013 National Energy Efficiency Action Plan (NEEAP) and the Country Action Plan (CAP) for improved cook stoves for (ICS) as SREDA becomes effectively operational. • Capacity Building: CCEB could follow-up with counterparts to evaluate their capacity to perform their role and/or function and overall usefulness and effectiveness of the training. For example, staff of the Bangladesh Power Development Board (BPDB) indicated to the evaluation team that they are not able to fully utilize the models developed by CCEB and require further hands-on training. • Monitoring and Evaluation (M&E) of BERC’s Performance: It is not clear whether current progress relative to the organizational maturity model reflects how BERC is performing relative to its legally mandated functions. As such, it may not be an effective measure of progress as a result of CCEB support • M&E of Training: Capacity-building efforts aimed at enhancing the country’s ability to design and implement supportive policies and regulations, and increase utilization of clean energy approaches and technologies should be evaluated after an extended period post- training delivery to assess the initiative’s long-term efficacy and benefits. 4 The survey was limited to direct beneficiaries of CCEB support under Component 2, Task 5 Conduct Market Analysis and Development for Improved Cook Stoves. 5 The Likert scale is defined as a method of ascribing quantitative value to qualitative data, to make it amenable to statistical analysis. vii CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT • M&E of ICS: o Most of the ICS users surveyed had been using the ICS for less than one year. A follow-up survey is recommended to determine the long-term durability and sustainability of tools. o The ICS end users surveyed were from middle to low income communities and they did not indicate the purchase of the ICS was a major issue. However, a survey of low￾income rural communities may be considered to determine whether the pricing would be a significant barrier to future deployment on a large scale. o CCEB should revise the Task 5 M&E reporting methodology to include information on distributor sales rather than collecting information only on sales by manufacturers, which provides a limited picture of the ICS supply chain and does not provide sufficient, feedback to CCEB and USAID/Bangladesh on how successfully the supply chain is developing. Expanding the M&E reporting system to include information from ICS distributors can provide valuable information for CCEB and help inform other key ICS CAP stakeholders. • Support to BERC: Support to BERC should focus on enhancing the organization’s function with respect to its legal mandate and less on resolving short-term issues. • Energy Planning Models: User reference manuals should be developed for the two CCEB models to improve energy sector planning and analysis. • Sustainability of Industrial Energy Efficiency: CCEB should work closely with the Sustainable and Renewable Energy Development Authority (SREDA) to accelerate adoption of national standards for auditor certification and energy audit protocols, as well as improve coordination with industry associations and the financial community to enhance awareness of the benefits of energy efficiency. • Development of ICS market supply chain: o Findings indicate that there are lags in end-user sales relative to distribution from the manufacturers. o Use female ICS end users as advocates in CCEB promotion activities. The ICS end users survey indicates end users discuss their experiences not only with family members but also with their local community. CCEB should use some of the end users as advocates to facilitate and accelerate village level acceptance of ICS. The evaluation team found a number of opportunities to improve effectiveness. They include: • Overall: o CCEB and key stakeholders should develop a stakeholder-driven strategic plan for sustainability. o CCEB activities should emphasize outcomes over outputs to achieve its development objective. o CCEB could work to facilitate strategic alliances between SREDA, the financial community, and industry and trade associations to foster local ownership and engender greater awareness throughout the industrial sector. This would catalyze implementation in accordance with the goals of the 2013 NEEAP. o Capacity building should be performance-based and support skills building in M&E to ensure effectiveness. • Task 3: o CCEB should assist key counterparts and stakeholders in the development of financing mechanisms. Findings indicate there is willingness to pay for energy efficiency services delivered by the CCEB subcontractors and that the financial community views the energy efficiency market as an opportunity to expand its lending portfolios. CCEB can assist these key stakeholders and key counterparts such as SREDA in the establishment of new financing mechanisms for energy efficiency and clean energy. The project should consider the feasibility of transitioning the existing grant program to a sustainable financing mechanism, simplifying the application process, and reducing the time for financier’s approval. viii CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT o CCEB should establish a formal mechanism for matchmaking between industry and the financial community to promote implementation of clean energy and energy efficiency measures. Findings indicate there is a nascent market for energy efficiency; CCEB should work with industry (through trade associations and the chamber of commerce) and the financial community to develop a formal mechanism that can accelerate “match-making” between the two sectors, stimulate, and expand the fledging market. • Task 4: o Utility-based DSM programs targeting energy efficiency and conservation are facilitated by a supportive regulatory framework to ensure widespread adoption. However, to date, favorable regulations promoting/requiring utility-based DSM are not in place. Therefore, CCEB should not initiate any additional demand side management work and should redistribute resources to other tasks until a supportive regulatory framework is in place. • ICS: o CCEB should support the ICS manufacturers in working more closely with distributors and vendors to ensure balance of production and demand along the supply chain so that there is not oversupply or unavailability in the market. o CCEB should keep working closely with the manufacturers to ensure the validity of the design and construction issues raised by some ICS end users. o CCEB should work with policy-makers to revisit the duties imposed on imported cook stoves given the limited local manufacturing capacity and the substantive number of ICS needed to meet the CAP targets. Evaluation Question 3: What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? The team found evidence of the following constraints to sustainability: • Regulatory environment is not supportive of clean energy o BERC is still a weak organization that lacks autonomy, requires restructuring, and lacks resources, including sufficient qualified permanent staff to carry out its mandated functions and develop an enabling environment conducive to widespread adoption of clean energy. • Insufficient skilled personnel o There are limited organizations offering training for clean energy professionals. o A national standard for certification, in line with international practices, of energy efficiency professionals is anticipated to develop qualified persons to work as consultants or as energy managers. o Key organizations such as BERC and SREDA do not have enough qualified professionals. o Capacity building must be scaled up to develop and sustain the critical mass of energy efficiency professionals needed to reach national targets for industrial energy efficiency. • Market for clean energy and energy efficiency goods and services is still immature o Although there are some examples of private companies paying for energy efficient goods and services, there is limited demand due to a lack of awareness among most of industry. o CCEB should examine whether it is appropriate to begin phasing-out its current support to ensure they are not subsidizing services that private sector companies are willing to buy. If appropriate, the project should develop a transition strategy in conjunction with key counterparts and stakeholders to facilitate market expansion. ix CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT o The design and construction of the ICS needs to be further evaluated. Over 27% of the ICS responders expressed concerns with the ICS designs and attributes. Concerns included a ‘better fan/charger’, ‘enlarged fuel feed entrance space’, ‘enlarged combustion space’, ‘increased stove height’, ‘better quality handle’ and ‘making the body heat resistant’. Other requests for improvements included: increased ease of refueling, reduced cooking time, less smoke, and more energy savings. These should be further investigated by the manufacturers. o There are insufficient financing mechanisms for clean energy and energy efficiency. CCEB should work with the financial community to transition its grant facility to a sustainable mechanism. The following opportunities to enhance sustainability exist: • CCEB activities are consistent with national policies and the program can work with key GoB counterparts to ensure they can leverage CCEB program activities to establish national standards. • CCEB can enhance its outreach efforts and leverage existing communication networks such as trade and industry associations to increase awareness of the benefits of energy efficiency. • CCEB can promote the additional benefits of job creation and employment from adoption of clean energy to facilitate political support. • CCEB should work to establish a formal mechanism for matchmaking between industry and the financial community to promote implementation of clean energy and energy efficiency measures. • CCEB should direct more support on ICS toward the point of sales between distributors and end users and leverage existing end-user experiences, to accelerate sales. Evaluation Question 4: How effectively has gender been integrated or incorporated in the interventions? What have been the challenges and opportunities, if any? In Bangladesh, 94.8%6,7 of the energy sector workforce is male. CCEB has effectively successfully mainstreamed gender integration, exceeding program targets for gender participation by the end of the third year. Although CCEB has been able to increase gender participation in its activities, the team did encounter some negative attitudes towards women, notably regarding their participation in energy audits. Lessons Learned The following key lessons can be learned from CCEB: • Capacity building needs to be strategic and evaluated for effectiveness, targeting performance improvement. For example, CCEB should work closely with SREDA to build its capacity to establish national standards for energy audit protocols and certification of energy efficiency professionals. (Lesson Learned (LL)4 and LL5) • Communication with all key actors and stakeholders is essential for scalability and sustainability. For example, CCEB should improve and enhance its outreach on industrial energy efficiency by using the existing networks of trade and industry associations. (LL3) • Counterpart ownership is crucial for long-term sustainability. For example, it is important for BERC to secure funding for changes in its organizational structure and for an e-docketing system. (LL2) • Project implementation should emphasize strategy over activity to achieve its development objective. For example, having demonstrated the value of using professional certified 6 Source: “Bangladesh Bureau of Statistics, Ministry of Planning, Report on Labor Force Survey 2010” 7 In Pakistan the number is 92%. x CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT auditors and standardized approaches to audits, long-term sustainability will be enhanced if the CCEB approaches are used to develop national standards. (LL1) ii CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 1. INTRODUCTION International Business and Technical Consultants, Inc. (IBTCI) under the Accelerating Capacity for Monitoring and Evaluation (ACME) activity, U.S. Agency for International Development (USAID) contract AID-388-C-14-00001, conducted a mid-term performance evaluation of the five-year Catalyzing Clean Energy in Bangladesh (CCEB) project. Initiated in October 2012,8 CCEB is intended to: • Support energy sector development for energy security, economic growth, and climate change mitigation in Bangladesh. • Build the Government of Bangladesh’s (GOB) capacity to design and implement supportive policies and regulations, and increase utilization of clean energy approaches and technologies for energy sector development on a low carbon trajectory. The primary purpose of the evaluation is to determine the extent to which the CCEB is on track (including processes and outcomes) to meet its overall goals across the project’s two major components: • Component A: Improve Enabling Environment for Low Emissions Development • Component B: Increase Energy Efficiency and Conservation The evaluation team carried out fieldwork in Bangladesh from November 9, 2015 through December 31, 2015. Data were collected in Dhaka and several rural locations.9 The team analyzed the data and wrote the report between December 31, 2015 and February 9, 2016. 8 The evaluation scope of work is included in Annex K of this document. 9 Task 1-4 information was gathered in Dhaka. Data pertaining to Task 5, ICS, were collected from the following districts: Bagerhat, Chapai Nawabganj and Gaibandha. 1 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 2. PROJECT BACKGROUND: THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE Economic growth in Bangladesh has been constrained in part due to a flawed energy sector. More than 50 percent of the population lacks access to electricity. The sector governance framework lacks a strong regulatory environment and energy prices are too low to sustain commercial services. Several studies show that energy efficiency can be enhanced in the industrial sector. However, the country faces challenges to grow its economy using a low carbon development trajectory. USAID/Bangladesh is committed to providing support to reduce and sequester greenhouse gasses (GHGs). The 2011-2016 Country Development Cooperation Strategy (CDCS) is oriented around four development objectives (DOs). DO4: Responsiveness to Climate Change Improved aims to mitigate climate change and increase the nation’s responsiveness and adaptation to challenges that occur due to climate change through three interrelated intermediate results (IRs):1) improved management of natural resources; 2) enhanced adaptation capacity and resilience to shocks; and 3) strengthened capacity for low emissions development, focusing on mitigation. CCEB contributes to the achievement of CDCS IR4.3: Strengthened Capacity to Reduce Emissions and the corresponding sub-IRs 4.3.1: Improved Enabling Environment for Low Emissions Strategies; 4.3.2: Increased Adoption of Renewable Energy; and 4.3.3: Improved Energy Efficiency and Conservation. The CCEB’s development hypothesis is that “USAID support for enhanced planning capacity for energy sector and low-emissions development, increased investment in energy efficiency and renewable energy, and an improved energy sector regulatory framework will result in strengthened Bangladesh capacity to reduce emissions.” 2 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 3. PURPOSE OF THE EVALUATION As required by the scope of work (see Annex K), the mid-term evaluation assessed the extent to which the CCEB project is on track to meet its overall goals and recommended evidence-based adjustments to programmatic approaches that could enhance the potential for achievement of the intended results. Evidence-based constraints are presented with specific recommendations to enhance effectiveness and impact, taking into consideration the project’s existing scope of work and contract. Finally, the evaluation team determined the extent to which USAID investments are likely to be sustained after the life of the project and provided viable recommendations for future (post-CCEB) focus areas. 3.1. Intended Audience While the primary intended user of this evaluation is USAID/Bangladesh, particularly the Economic Growth (EG) Office and Mission management, the evaluators anticipate that other key stakeholders may be interested secondary audiences. These include local institutions, including the GOB, other donors, and other actors in the clean energy space, including the leading institutions involved in energy efficiency and clean energy in Bangladesh. The USAID EG office should be able to use the findings and recommendations presented in this report to plan for the remaining period of the project, as the evaluation provides an informed evidence-based assessment of the project’s progress and identifies needed adjustments. 3.2. Evaluation Questions The evaluation provided evidence-based answers to the following questions, as specified in the scope of work: 1. To what extent is the CCEB program on track, in terms of progress and outcomes, to meet its overall goals for the five tasks under Components 1 and 2? 2. What plausible opportunities exist to enhance the project’s programmatic approach and effectiveness within the stipulations of the contract? 3. What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? 4. To date, how effectively has gender been integrated or incorporated in the interventions? What have been the challenges and opportunities, if any? 3 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 4. EVALUATION METHODOLOGY AND LIMITATIONS The evaluation used a utilization-focused, mixed-method methodology. The steps included planning; document review; data collection through key informant interviews (KIIs), focus group discussions, (FGDs) and surveys; data analysis; and report preparation.10 4.1. Planning and Document Review Throughout the evaluation, the team worked closely with USAID/Bangladesh’s EG Office, especially the evaluation point of contact and the ACME Contracting Officer’s Representative (COR), ensuring the evaluation remained on schedule and issues were addressed quickly. This was pertinent to the improved cook stoves (ICS) survey, given that limited end user information was readily available. The work plan was submitted on November 15, 2015. It was revised and resubmitted on November 17, 2015, November 24, 2015, and December 5, 2015 and was approved by USAID on December 7, 2015. The team reported progress every week and held progress briefings with USAID/Bangladesh bi￾weekly. The team relied on CCEB-produced materials but also reviewed a range of secondary information sources (see Annex G for details). 4.2. Data Collection and Analysis The evaluation team collected both quantitative and qualitative data through 17 KIIs with 39 people, 3 FGDs with 15 industry representatives and members of the financial community, and a survey of 134 ICS end users.11,12 The informants were key CCEB stakeholders, including central government entities, private sector organizations, the CCEB implementing partner and local subcontractors, and program beneficiaries. A list of organizations can be found in Annex I. Every two days, the team held internal debriefs to review and crosscheck the evidence collected, look for patterns, identify any discrepancies, and make any required adjustments to the schedule. The team analyzed the data based on its respective source and compared to related data sources. For example, the team developed preliminary findings by first analyzing interviews with key informants, and then developed complementary preliminary findings from key documents and other secondary materials. Finally, the team used a risk assessment framework (RAF) matrix to assess the extent of progress toward the desired development outcome. See Annex H for a full description of the RAF methodology. 4.3. Biases and Limitations Recall, response, and selection biases and other limitations were considered and accounted for within the methodology used for data collection and subsequent analyses. The team sought to minimize these by triangulating data using multiple sources for each issue. By synthesizing the information found in documents and/or interviews from multiple sources, the evaluation team reduced the likelihood of a singular data point skewing end results. 10 Documents reviewed are included in Annex G. Instruments used for KIIs, FGDs and the ICS survey are included in Annexes D, E, and F respectively. Data analysis methodologies are discussed in Annex C. The Evaluation Design Matrix used to inform the answers to each of the evaluation’s questions is included in Annex B. 11The evaluation team found that neither the CCEB nor the ICS manufacturers CCEB is supporting collect contact information for end users. As such, the team carried out a phone survey of some of the major ICS distributors to identify end users. 12 The survey was limited to direct beneficiaries of CCEB support under Component 2, Task 5 Conduct Market Analysis and Development for Improved Cook Stoves. 4 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 5. FINDINGS AND CONCLUSIONS Evidence-based findings and conclusions drawn from the document review, KIIs, FGDs, and ICS survey are presented in the following sections of the report. 5.1. Findings - CCEB Overall Performance The team analyzed responses to the four questions to determine the extent of the project’s progress. Responses are summarized below with an indication of the responses’ consistency with the views of implementing partner and USAID.13 Table 3: Qualitative Questions Summary Findings Item Number of Stakeholder Interviewee Responses Highest # 2nd Highest # 3rd Highest # Expectations Promote & motivate industry to implement14 Targeted capacity building to address skill gaps15 Accomplishments BERC e-docketing system 16 Grants Software models Challenges Industry awareness Lack of access to energy experts (EE) and green technologies Residential energy use Challenges – demand side management (DSM) only Distribution Companies (DISCOs) not policy setters Residential energy use Load shedding no longer an issue Strengths Financial support to industry17 Improved BERC Activities outside of Dhaka18 Constraints Lack of awareness and knowledge19 Lack of primary data Lack of trained manpower Constraints – DSM only Government policy does not promote DSM20 Lessons learned Targeted capacity building to address skill gaps is needed Promotion and awareness can facilitate positive behavior change Private sector approach helped facilitate development of ICS manufacturing 13 It is imperative to learn if the comments received are consistent across all of the key actors involved in CCEB’s implementation. This provided an objective vision of the program in totality as well as providing a method of identifying where there are disconnects that may well hamper progress towards meeting outcomes. 14 Consistent with USAID and or IP comments 15 Consistent with USAID and or IP comments 16 Consistent with USAID and or IP comments 17 Consistent with USAID and or IP comments 18 Consistent with USAID and or IP comments 19 Consistent with USAID and or IP comments 20 Consistent with USAID and or IP comments 5 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT CCEB Risk Assessment Framework (RAF) and Risk Mitigation Options CCEB support includes activities related to the clean energy enabling environment (Tasks 1 and 2), clean energy deployment delivery mechanisms (Tasks 3, 4, and 5), and the financing environment (Task 3). As such, it is important to consider any interdependences and linkages between the different activities when assessing the project’s progress toward its development objective. To do this, the team assessed risks using the quantitative and qualitative information collected through the KIIs and FGDs, with consideration for potential impact if any tasks did not achieve the desired outcome. The team also considered the probability and potential severity of impact of an occurrence. The evaluation team ranked against the following four impact levels: negligible, marginal, serious, and catastrophic. The probability of failure was ranked as remote, occasional, probable, or frequent. (See Annex H for quantitative and qualitative explanations of each of these impact and probability levels.) Table 4 shows the relative rankings of each of the main CCEB activities. The relative rankings for each CCEB activity were entered into a RAF matrix using the following probability ranges: (i) remote. less than 0.25 probability; (ii) occasional, 0.26 to 0.50 probability: (iii) probable range, 0.51 to 0.75 probability; and (iv) frequent, 0.76 to 0.99 probability. Table 5 shows the CCEB Risk Mitigation Assessment Matrix. Review of the RAF shows there is less than 50 percent probability that tasks 2 and 3 would fail to achieve their expected objectives and the impact would be marginal. With Tasks 4 and 5, there is slightly higher probability of failure, but the impact would be similar. Given the crucial role regulation has on creating an enabling environment for widespread adoption of clean energy, there will be major impact if BERC fails to function at or close to international best practice standards. Failure in this area could potentially undo much of what CCEB has accomplished thus far in establishing foundations for success. Unfortunately, it is also difficult for CCEB to address BERC’s systemic issues, which require sustained commitment from the GOB to be addressed appropriately. If that commitment is forthcoming, there is a high probability that CCEB will be very close to achieving its development objective by the end of the program. 6 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 4: Summary of RAF Analyses Impact Catastrophic High Critical Critical Critical Serious High High Critical Critical Marginal Medium Medium High High Negligible Low Low Medium Medium Remote Occasional Probable Frequent Probability CCEB Risk Mitigation Assessment Identifying a risk is only a first step. It is imperative to identify concrete actions to mitigate identified risks. Table 6: CCEB Risk Mitigation Assessment Matrix presents mitigation options for the risks identified with the various CCEB activities. Tasks 2 & 3 Task 1 Task 5 Task 4 7 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 5: Summary of Probability and Impact Determinations for CCEB Tasks CCEB Component ID # CCEB Activities Probability Rating Impact Impact & Probability Deter￾mination Ranking Comments Component 1 Task 1: BERC 1.1.1 (i) Maturity Model ODA Occasional Marginal Medium BERCs continuing staffing issues and GOB’s lack of commitment to independent regulation and lack of funding means BERC does not progress as an organization and the impact is marginal (M). Probability is considered occasional (O) and impact and probability determination (I&PD) is medium (M). 1.1.2 (ii) e-docket system Probable Serious Critical Best practice regulatory bodies use e-docketing systems to enhance their performance. Developing a request for proposal (RFP) for an e-docketing. System will accelerate adoption if BERC can obtain funds from the GOB to implement. The probability of failure is considered probable (P); the impact serious (S) and the I&PD is critical (C). 1.1.3 (iii) Legal department Frequent Serious Critical A legal department is needed for appropriate regulatory authority. BERC has no legal department and lacks funds to implement. The probability of failure is considered frequent (F); the impact serious (S) and the I&PD is critical (C). 1.1.4 (iv) Organizational review Probable Serious High The new organizational structure requires resources to implement. BERC has no resources and lacks funds to implement. The probability of failure is considered probable (P); the impact serious (S) and the I&PD is critical (H). 1.1.5 (v) Advisory services Occasional Marginal Medium Ad hoc support to BERC addresses problems but does not resolve systemic issues, including continuing staffing issues coupled with the lack of commitment to independent regulation and lack of funding. Impact is marginal (M) as there is no capacity being built, probability is considered occasional (O) as CCEB can provide support to address ad hoc issues and the I&PD is medium (M). 1.1.6 (vi) Power plant regulatory audit procedure Occasional Marginal Medium Limited number of trained staff. Impact is marginal (M) given limited trained staff but as there is some trained staff the probability is considered occasional (O) and the I&PD is medium (M). Average Ranking Task 1 Occasional to Probable Marginal to Serious High to Critical 8 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Component 1 Task 2: Analytical Capacity 1.2.1 Developed GHG emission and Power Sector Planning and Analysis models and built capacity of counterpart personnel Occasional Marginal Medium Models provide good analyzes but need trained staff to use. Impact is marginal (M) due to limited training. As there is some trained staff the probability is considered occasional (O) and the I&PD is medium (M). Average Ranking Task 2 Occasional Marginal Medium Component 2 Task 3: Industrial Energy Efficiency 2.3 • Built capacity of service organizations • Introduced certification standards and audit protocols • Provided grant financing • Implemented energy efficiency measures • Raised awareness among stakeholders, notably the financial community Occasional Marginal Medium Limited number of auditors trained, audit protocols demonstrated but greater awareness of EE by industry needs to be developed as well as many more trained auditors are needed to implement on a national scale. Impact is considered marginal (M) due to limited number of trained staff but as there is some trained staff the probability is considered occasional (O) and the I&PD is medium (M). Component 2 Task 4: DSM 2.4 • DSM Roadmap • Time of Use (TOU) tariffs • Interruptible tariff • Load control • Pre-paid meters Probable Serious High DSM is highly dependent on strong regulation of electricity usage and is driven by a regulatory framework in which pricing reflects cost of service. BERC is not performing as an independent regulator due to lack of autonomy, lack of sufficient qualified permanent staff and the lack of an organizational structure that reflects international best practice there is high probability (P) of failure and the impact of this will be serious (S). The I&PD is high (H) Component 2 Task 5: Market Development 2.5 Developed local manufacturing capacity and sales of Tier 2 ICS Probable Serious Medium Local manufacturing capacity has been established and ICS produced locally are apparently gaining acceptance, but ICS targets are very high. There is high probability (P) of failure unless manufacturing is increased and sales and marketing is improved. Failure to expand capacity would have a serious impact (S). Expansion requires funding so the I&PD is medium (M). 9 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 6: CCEB Risk Mitigation Assessment Matrix Desired Outcome CCEB Component ID # CCEB Activities Key Observations Risk Mitigation Options Enabling Environment in place that promotes clean energy development Component 1 Task 1: BERC 1.1.1 (i) Maturity Model ODA KPIs should reflect BERC’s mandated performance. Necessary changes in the organization structure that include: creation of a Legal Department to support legal, litigation, arbitration, investigation, enforcement and other legal functions; incorporates M&E requirements and introduces other functional responsibilities as provided by BERC Act 2003; must be implemented rapidly and require appropriate commitment and support for BERC to become a fully￾functioning regulator consistent with international best practices. Significant development support could facilitate this but should be conditional on demonstrated GOB commitment to implementing regulatory autonomy. 1.1.2 (ii) e-docket system The proposed system is consistent with BERC operating as per its legislated mandate. Funding should be allocated by GOB to implement. 1.1.3 (iii) Legal department The proposed legal department is consistent with BERC operating as per its legislated mandate. The GOB needs to commit to funding the necessary changes in the organization structure and implement them rapidly. This will enable BERC to function in accordance with the 2003 BERC Act and help establish a regulatory framework that promotes clean energy deployment. This in turn will lead to consistent pricing which in turn will stimulate energy efficiency and DSM. 1.1.4 (iv) Organizational review CCEB carried out this key input as a step towards needed changes in BERC’s structure. Implementing the changes will facilitate BERC’s The GOB needs to commit to funding the necessary changes in the organization structure and implement them rapidly This will enable BERC to function in accordance 10 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT functioning in accordance with its legislated mandate. with the 2003 BERC Act and help establish a regulatory framework that promotes clean energy deployment. This in turn will lead to consistent pricing which in turn will stimulate energy efficiency and DSM. 1.1.5 (v) Advisory services Can be key inputs to catalyze BERC functioning in accordance with its legislated mandate. Services should be conditional on demonstrated GOB commitment to implementing regulatory autonomy 1.1.6 (vi) Power plant regulatory audit procedure This is consistent with BERC functioning in accordance with its legislated mandate. Regulation should be adopted rapidly as it is consistent with international best practices and reduce emissions of greenhouse gases. GHG analytical capability exists in key government entities Component 1 Task 2: Analytical Capacity 1.2.2 Built capacity of counterpart personnel Counterparts stated they need hands-on training. Provide hands-on training to staff at Power Cell, BPDB, and staff at other key stakeholders. Industry reduces energy use by cost￾effective reduction projects Component 2 Task 3: Industrial Energy Efficiency 2.3 • Built capacity of service organizations • Introduced EA certification standards and audit protocols • Provided grant financing • Implemented energy efficiency measures • Raised awareness among stakeholders, notably the financial community Energy Audit certification and audit protocols positively recognized by industry and financial community CCEB engagement with financial community very positive. Financial community ready to lend for EE; CCEB Investment Grade Audit facilitates financier’s due diligence. Approach (audit/certification) scalable consistent with national policy for EE, and can Limited signs that EE services market is developing. Awareness among end users limited, particularly on accessing finance. • Work with SREDA and leverage the CCEB introduced auditor certification curricula and audit process and protocols to establish national standards. • Develop financing mechanisms with SREDA, and the financial community. • Develop a match-making mechanism between industry and the financial community. • Leverage the existing trade associations and chambers of commerce/industry to promote greater awareness of energy efficiency benefits. 11 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Utilities implement DSM programs to improve effective use of available energy supply Component 2 Task 4: DSM 2.4 • DSM roadmap • TOU tariffs • Interruptible tariff • Load control • Pre-paid meters Both Dhaka Electric Supply Company (DESCO) and the Dhaka Power Development Company (DPDC) see the prime driving force for DSM to be a fully functioning BERC. Both spoke highly of efforts taken by CCEB to try to facilitate DSM activities. Both looking for funding for pilots. Both still very technical energy supply organizations and yet to transition into commercial entities. Supporting BERC’s efforts to become a fully functioning regulator will facilitate DSM initiatives if it is accompanied by reduction/removal of subsidies currently provided to electricity consumers. Supporting and strengthening consumer/end user advocacy organizations can help catalyze DSM initiatives. Scale up use of ICS in rural communities Component 2 Task 5: ICS Market Development 2.5 Developed local manufacturing capacity and sales of Tier 2 ICS The third year has seen a significant increase in sales of the ICS from the five manufacturers supported by CCEB. The ICS survey revealed over 70% of the ICS sold by the manufacturers have been distributed. Develop a match-making financing mechanism between the ICS manufacturers and the financial community to access funds for expansion of manufacturing capacity. Provide training to point-of-sale/interface between end users and vendors/suppliers. Use existing ICS end users in promotion activities, leveraging their experiences to date with the ICS. 12 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 5.2. Evaluation Question 1 The findings with respect to the four specific evaluation questions included in the Evaluation Design Matrix (Annex B) are presented in the following sections. For each question, we present qualitative data collected from KIIs, organized around the individual questions. Quantitative findings are shown in graphic form. Question 1: To what extent is the CCEB program on track, in terms of progress and outcomes, to meet its overall goals (for the five tasks under the components 1 and 2)? The findings show that most stakeholders believe CCEB is making positive progress, and is having positive impact in most areas, with the exception of the regulatory framework. These issues have been described and mitigation options are provided above. KII Question 1.1 (a) How effective has the program’s technical assistance approach been in strengthening institutional capacity? The evaluation team found the program has been partially successful to date in strengthening institutional capacity. BERC is a stronger organization thanks to CCEB support, notably having developed the power plant regulatory audit protocol and strengthened the legal department. BERC has a long way to go, however, to become a regulatory body consistent with international best practices. Analytical capabilities for energy planning have been enhanced within key counterpart organizations (i.e., Power Cell and Bangladesh Power Development Board (BPDB)) due to the availability of the program-developed analytical models (Power Sector Planning Analysis Model (PSPAM) and the GHG model). However, their utility is limited as some of the counterpart staff indicated during KIIs they felt the training they received was not sufficient for them to be fully capable of using the models. Consequently, they do not use them in their day-to-day activities. As such, the models are not used to help inform national policy decisions. The evaluation team found evidence that the CCEB has contributed to the establishment of a market for energy efficiency services and technology through Task 3 and 5 activities. Under Task 3, the program has succeeded in mobilizing the financial community, helping to reduce transaction costs for the banks by developing a small pool of internationally-certified energy auditors who carry out technical due diligence of energy efficiency measures using standardized procedures (walk-through audits and investment grade audits). They have used a grant financing mechanism to “buy-down” part of the implementation costs for identified energy efficiency opportunities, establishing a proof of concept for certain elements of the industrial sector. As part of Task 5, CCEB has helped increase local manufacturing capacity for ICS and has supported development of testing protocols. KII Question 1.1 (b) How effective has the project’s technical assistance approach been in building capacity? The evaluation team found evidence the project has built capacity across all major task areas, but further capacity building is needed. For example, in Task 2 there is limited evidence that the professionals who received training on the analytical models developed by CCEB can fully use them, nor can they adapt or develop them. There is evidence that the analytical tools are valued but their long-term utility is at risk. There is a small cadre of certified audit professionals whose are appreciated and valued by the financial community. However, this group is only sufficient to have provided qualified personnel to carry out the activities expected to be delivered by CCEB. Strategically, having proven the conceptual approach, the program can enhance the likelihood of sustainability by transitioning towards assisting the country in establishing mechanisms to develop a critical mass of qualified professionals to support the 2013 NEEAP goals for energy efficiency improvements. How effective has the project’s technical assistance approach been in catalyzing clean energy development? 13 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT The evaluation team found evidence that the project’s approach has been somewhat effective in catalyzing clean energy. The evidence includes the implementation of energy-efficient technologies in industry, introducing the concepts of certified energy efficiency professionals, developing standardized auditing protocols that are gaining acceptance by industry, and the financial community’s willingness to invest in clean energy. However, there is much to be done to ensure the foundations that the program has established are leveraged to expand the market and create sustainable mechanisms for clean energy development. How effective has the program’s technical assistance approach been in increasing access to financing? The evaluation team found evidence that one of CCEB’s major accomplishments has been increasing access to financing. The project has mobilized the financial community, resulting in the initiation of limited financing of industrial energy efficiency activities without financial support from CCEB (audit services and energy efficiency project implementation). While these are positive outputs, the evaluation team found evidence from KIIs with CCEB subcontractors that it is currently difficult to sell services for energy efficiency and that due to the lack of industry awareness, the market for energy efficiency goods and services is not yet mature.21 In addition, CCEB has been instrumental in getting Bangladesh Bank to accept that energy efficiency activities can be eligible for “Green Financing.”22 How effective has the CCEB program been in achieving outputs or results? There are findings from KIIs and FGDs that the project has been effective in achieving outputs and results in all major task areas and the CCEB outputs are consistently considered to be of high quality. Examples include: the development of the Power Plant Regulatory Audit protocol (Task 1); the PSPAM and GHG models (Task 2); the certification from an internationally recognized body of more than 80 energy auditors; the introduction and acceptance by local banks of standardized energy audit protocols as technical due diligence tools for investment decision making; saving of energy from deployment of energy efficient technologies (Task 3); and assisting the development of the market supply chain for ICS (Task 5). However, the challenge exists to turn these outputs into outcomes as envisioned in the CCEB results framework. To what extent has participant training under CCEB been able to build capacity of partner institutions, including capacity of women members, to enhance market development? The evaluation team found evidence that the country’s analytical and energy planning capacity has been somewhat enhanced, but additional training is needed. Some of the participants trained on the two analytical models expressed their concerns that the training afforded by CCEB did not include any “hands-on” sessions that would have enabled them to learn to use the models under the supervision of the training instructor. In addition, they were not provided with a user manual to be used post training as a reference. International best practices for instruction typically includes “learning by doing,” and provides trainees 21 The estimated market for industrial energy efficiency goods and services is between $3 to 4 billion. The exact amount of funding spent by CCEB for industrial energy efficiency activities is not known but thought to be less than 0.1% of the overall market potential. 22 “Green Financing” products/activities are eligible for funding at a discounted financing rate, currently around 9%, as opposed to the prevailing commercial rate that is between 15% and 16%. 14 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT access to the tools needed (in this case laptops and the model) during the classroom sessions as well as simulation exercises to solidify participants’ understanding and capabilities. 5.3. Evaluation Question 2 Question 2: What are the plausible opportunities to enhance programmatic approach and effectiveness within the stipulations of the contract? • Overall: CCEB should continue to align all activities with national goals and objectives, especially the 2013 NEEAP and the CAP for ICS. • Capacity Building: CCEB should ensure their counterparts’ capacity to perform their role and/or function. For example, staff of the Bangladesh Power Development Board (BPDB) indicated they are not able to fully utilize the models developed by CCEB and require further hands-on training. • Monitoring and Evaluation (M&E) of BERC’s Performance: It is not clear whether current progress relative to the organizational maturity model reflects how BERC is performing relative to its legally mandated functions. As such, it may not be an effective measure of progress as a result of CCEB support • M&E of Training: Capacity-building activities should be evaluated after an extended period post- training delivery to assess the initiative’s long-term efficacy and benefits. • M&E of ICS: o Most of the ICS users surveyed had been using the ICS for less than one year. A follow-up survey is recommended to determine the long￾term durability and sustainability of tools. o The ICS end users surveyed were from middle to low income communities and they did not indicate the purchase of the ICS was a major issue. However, a survey of low-income rural communities should be done to determine whether the pricing would be a significant barrier to future deployment on a large scale. o CCEB should revise the Task 5 M&E reporting methodology to include information on distributor sales rather than collecting information only on sales by manufacturers, which provides a limited picture of the ICS supply chain and does not provide sufficient, feedback to CCEB and USAID/Bangladesh on how successfully the supply chain is developing. Expanding the M&E reporting system to include information from ICS distributors can provide valuable information for CCEB and help inform other key ICS CAP stakeholders. • Support to BERC: Support to BERC should focus on enhancing the organization’s function with respect to its legal mandate and not on resolving short-term issues. • Energy Planning Models: User reference manuals should be developed for the two CCEB models to improve energy sector planning and analysis. • Sustainability of Industrial Energy Efficiency: CCEB should work closely with the Sustainable and Renewable Energy Development Authority (SREDA) to accelerate adoption of national standards for auditor certification and energy audit protocols, as well as improve coordination with industry associations and the financial community to enhance awareness of the benefits of energy efficiency. • Development of ICS market supply chain: o Findings indicate that there are lags in end-user sales relative to distribution from the manufacturers. o Use female ICS end users as advocates in CCEB promotion activities. The ICS end users survey indicates end users discuss their experiences not only with family members but also with their local community. CCEB should use some of the end users as advocates to facilitate and 15 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT accelerate village level acceptance of ICS. The evaluation team found a number of opportunities to improve effectiveness. They include: • Overall: o CCEB and key stakeholders should develop a stakeholder-driven strategic plan for sustainability. o CCEB activities should emphasize outcomes over outputs to achieve its development objective. o CCEB should work to facilitate strategic alliances between SREDA, the financial community, and industry and trade associations to foster local ownership and engender greater awareness throughout the industrial sector, thus catalyzing implementation in accordance with the goals of the 2013 NEEAP. o Capacity building should be performance-based and support skills building in M&E to ensure effectiveness. • Task 3: o CCEB should assist key counterparts and stakeholders in the development of financing mechanisms. Findings indicate there is willingness to pay for energy efficiency services delivered by the CCEB subcontractors and that the financial community views the energy efficiency market as an opportunity to expand its lending portfolios. CCEB can assist these key stakeholders and key counterparts such as SREDA in the establishment of new financing mechanisms for energy efficiency and clean energy. The project should consider the feasibility of transitioning the existing grant program to a sustainable financing mechanism, simplifying the application process, and reducing the time for approval. o CCEB should establish a formal mechanism for matchmaking between industry and the financial community to promote implementation of clean energy and energy efficiency measures. Findings indicate there is a nascent market for energy efficiency. CCEB should work with industry (through trade associations and the chamber of commerce) and the financial community to develop a formal mechanism that can accelerate matchmaking between the two sectors, stimulate, and expand the fledging market. • Task 4: Task 4 activities require a functional regulatory framework for widespread adoption. Therefore, CCEB should not initiate any additional demand side management work and should redistribute resources to other tasks until the Government of Bangladesh demonstrates tangible commitment to systemic change at BERC. • ICS: o CCEB should support the ICS manufacturers in working more closely with distributors and vendors to ensure production and demand along the supply chain is well managed and there is not oversupply or unavailability in the market for end users. o CCEB should work closely with the manufacturers to ensure the validity of the design and construction issues raised by some ICS end users. o CCEB should work with policy-makers to revisit the duties imposed on imported cook stoves given the limited local manufacturing capacity and the substantive number of ICS needed to meet the CAP targets. 16 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 5.4. Evaluation Question 3 Question: 3. What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? The evaluation team found evidence that CCEB is addressing some of the major constraints to sustainability of industrial energy efficiency activities through Tasks 3 and 5. However, much work remains to be done to enhance the likelihood of sustainability and scalability. The major constraints identified in this area through the KIIs and FGDs include lack of awareness about the benefits of energy efficiency among the industrial sector, a lack of skilled workers, and a lack of primary data on energy efficiency technologies and energy utilization and management practices. In addition, counterparts indicated they do not have certified energy managers working for them, which is not currently addressed by CCEB. The utility DSM counterparts, DPDC and DESCO, stated that current government policy does not promote DSM. Respondents stated that DSM will only be viable when the BERC is fully functional and develops policies conducive to DSM implementation. They were aware of CCEB’s support to BERC but felt the support was having little or no impact on policy and expressed concern that the impact of CCEB support is limited at BERC. The counterparts at BERC also indicated there were initial challenges with the implementing partner’s assistance, including selection of personnel and activities. While these choices impacted timing when the program could disengage, the assistance in the early phase did not address BERC’s systemic challenges. USAID decided to modify the project contract in the third year to continue support to BERC throughout the life of the project. Project support to BERC has evolved in the third year with more focus on addressing shortcomings in BERC’s organizational structure.23 The KIIs and FGD indicated that moving forward, CCEB should focus on addressing the lack of awareness of EE technology and practices in industry. It was felt an awareness campaign using the industrial trade associations’ networks championed by the financial community and industries who have benefitted from CCEB support could accelerate understanding. Review of background documents and the findings from the KIIS and FGDs with key counterparts, stakeholders, and beneficiaries across all task areas indicate activities are very scalable and sustainable. The challenge will be to leverage opportunities to enhance scalability and sustainability. Enhancing sustainability will require greater local ownership of many of the CCEB activities. For example, SREDA needs to develop national standards for industrial energy audits and certification of energy efficiency professionals. Currently, however, there is no plan for CCEB to support SREDA directly in this area. In addition, key counterparts are like SREDA—nascent organizations with limited staff and resources, or are like BERC—struggling to assume their leadership role in the sector. BERC has a critical role in market transformation and adoption of clean energy. However, a weak, underperforming BERC will stall progress in Tasks 3, 4, and 5. Despite donor support, BERC is still a long way from being a fully autonomous energy sector regulator operating in accordance with international best practices. 23 It should be noted that the quality of the work completed in support of BERC in the first phase was appreciated by the counterparts. By the end of the third year, the CCEB developed a new organizational structure, prepared an RFP for an e-docketing system, outlined capacity-building requirements, and drafted a Power Plant Audit regulation. 17 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Sustainability of Task 3 and 5 activities is supported by the 2013 NEEAP and the 2009 CAP. 3.1 Has your involvement in CCEB helped your business activities? There is some evidence that CCEB has enhanced the businesses it has worked with directly (notably the ICS manufacturers, the private sector organizations providing services under Task 3, the companies who received grant support to implement energy efficiency measures, under Task 3, and the financial community). However, several interviewees expressed concern that the enabling environment is not developing (notably independent regulation) despite CCEB support. Most organizations—including all of the service companies interviewed, the Power Cell and BPDB, and the four representatives from the financial community—indicated that the work of CCEB is part of their long-term business plans. 5.5. Evaluation Question 4 Question: 4: To date, how effectively has gender been integrated or incorporated in the interventions? What have been the challenges and opportunities, if any? There is informed evidence that CCEB has promoted gender integration within its activities. The program has a gender strategy and reports participation disaggregated by gender in its annual progress report, quarterly reports, and in its project monitoring and evaluation plan (PMEP). However, there are still challenges and opportunities with respect to gender integration that the project can help address. These are discussed in the paragraphs below. 4.1 To what extent has CCEB promoted and integrated gender into its activities? The evaluation team found evidence the CCEB gender strategy has been successful. CCEB’s PMEP includes two indicators with gender integration information, indicator #3 Number of people trained in energy, technical, business, and/or regulatory practices, and indicator #10 number of person hours of training completed in climate change as a result of U.S. Government (USG) assistance. Table 7 shows the percentage change in female participation from program inception until the end of FY15. Table 7: CCEB and Gender Integration PMEP Indicator # % Female Participation Actual FY 13 Actual FY 14 Actual FY 15 Target FY 16 Target FY 17 3 7 16.9 38 35 35 10 6.8 16.1 22 35 35 By the end of FY 15 (approximately the end of the third year of CCEB) female participation had already achieved the life of project target. 4.2 Are any CCEB activities specifically targeted toward gender integration? 18 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT The evaluation team found evidence that CCEB’s strategy of mainstreaming gender integration rather than targeting gender integration in specific activities has been successful. For example, one of the selection criteria for industry participation in the CCEB-supported energy audits under Task 3 stresses female participation in the ownership of the company, but this is not the only criteria for participation. 4.3 What challenges has CCEB encountered in integrating gender into its activities? The program does encounter challenges with gender integration in the conduct of its activities. For example, evidence shows that only 5 percent of energy sector employees are female, though women make up 30 percent of Bangladesh’s workforce.24 The GOB does have a gender policy with a target of 10% female participation in the energy sector, but clearly this policy does not appear to be effective in promoting this. In other sectors that CCEB interacts with, the situation is different. For example, in the manufacturing sector women represent 28 percent of the labor force25. Legacy biases also remain (notably in the service sector) that women are unable to perform technical activities. These persist despite relatively high female participation in the manufacturing industries (a current driver of the economy). 4.4 In your opinion, do you see CCEB taking existing opportunities to facilitate and promote gender integration in its activities? The team found evidence that CCEB is using existing opportunities to facilitate and promote gender in its activities, as demonstrated by the results included in the program’s M&E plan and their last quarterly report. The CCEB gender strategy called for support to the Council on Women in Energy and Environmental Leadership, however, there are additional opportunities the program can take, notably in Task 5 by using female end users to promote ICS (78 percent of people not in the labor force are females). 5.6. Findings – Overall Project M&E and associated key performance indicators (KPIs) should help inform project priorities: A number of M&E indicators for should reflect the desired outputs/outcomes from program activities. For example, a protocol that provides for monitoring information at key nodes along the market delivery chain for ICS (manufacturing, supply and sales, and end use) would allow CCEB to evaluate where to prioritize interventions, rather than monitoring only at one nodal point, as is currently done (sales by manufacturers). Similarly, KPIs for BERC-related support focused on their delivery of regulatory functions may provide more information on where CCEB support should be provided. 5.7. Findings – Task 1: Improve Regulatory Environment for Clean Energy Development The team identified the following key findings with respect to Task 1 activities: 24 Source: “Bangladesh Bureau of Statistics, Ministry of Planning, Report on Labor Force Survey 2010”. 25 Ibid 19 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT • Support to BERC has resulted in enhancing the organization somewhat. Examples include the proposed e-docketing system, the Power Plant Regulatory Audit, and the Organizational Development Assessment, and proposed new organizational structure. However, BERC is still a long way from functioning as an independent regulatory body and carrying out all its legally mandated regulatory functions at a level consistent with international best practices. • Given BERC’s critical role in sector regulation, a weak, underperforming BERC will adversely impact the progress toward adoption of clean energy in the country. • BERC has yet to fully address long-term systemic issues such as the ability to attract appropriately qualified permanent staff, development of an appropriate organizational structure, and sufficient resources to put in place to enhance its performance. • The current KPI used to measure BERC’s development (based on the maturity model) does not appropriately reflect how BERC is progressing toward being an international class regulator. KPIs should be based on BERC’s mandated responsibilities. • Support to BERC should be strategically focused not ad-hoc in nature. 5.8. Findings – Task 2: Strengthen Analytical Capacity for Energy Sector Planning and Policymaking The findings with respect to Task 2 are: • CCEB-developed models are recognized as good analytical tools. • Use of the models by the trained counterparts is limited due to incomplete training and lack of reference materials and resources. 5.9. Findings – Task 3: Promote Industrial Energy Efficiency and Conservation The following are the evaluation findings with respect to Task 3: • Market potential for industrial energy efficiency is high (between $3-4 billion). • There are indications that a market for energy efficiency goods and services is developing. The financial community is already carrying out direct financing of energy efficiency outside of the CCEB. • The financial community is well aware of the benefits of energy efficiency and is looking to finance energy efficiency projects. • Accessing financing, either through CCEB or the Bangladesh Bank (BB) “Green Finance,” mechanism is procedurally challenging and time consuming. • Financing transaction costs have been reduced thanks to the acceptance by the financial community of certified energy efficiency professionals and a standardized energy audit methodology. • Although industries working directly with CCEB have knowledge of clean energy and the benefits of energy efficiency, broader awareness of EE is relatively limited throughout industry. • By working more closely with trade and industry on outreach activities, CCEB can strengthen the awareness and help catalyze the development of the market for energy efficiency. • National counterparts need to assume a greater role in leading activities and leveraging CCEB’s activities. 20 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 5.10. Findings Task 4: Adopt DSM Programs for Electric Utilities The evaluation team found evidence that CCEB support has been appreciated by the distribution companies; however, activities to expand DSM efforts by the distribution companies will continue to stall without appropriate sector regulation and pricing signals. 5.11. Findings Task 5: Conduct Market Analysis and Development for Improved Cook Stoves Although the team considered all CCEB’s efforts since inception, the evaluation focused on market development activities since the program changed direction to emphasize local manufacture of the ICS rather than imported ICS. In addition to highlighting the findings from key informant interviews with stakeholders along the ICS supply chain to assess the development of the market, the evaluators interviewed five manufacturers, 21 suppliers and 134 end users about the overall program efforts. Major findings are presented below by each stakeholder group and for the market development activities as a whole. Findings – CCEB support to market development The following are the key findings with respect to CCEB’s supply chain development activities: • The decision to shift to support local manufacturing of the ICS appears to have been successful based on the number of ICS produced since CCEB began supporting the local manufacturers. • The decision to help develop testing protocols and support the local manufacturers to produce well-rated ICS products was commendable as it helped catalyze the introduction of the locally manufactured ICS. • There is not a comprehensive information system across the supply chain to adequately assess the development of the market. Currently, CCEB tracks only one KPI to assess market development, sales by the local manufacturers. This provides limited information on a critical part of the supply chain but does not adequately indicate how the overall market is developing. • There were some early issues reported related to product quality and design issues that resulted in some stoves being replaced at no cost and the introduction by some manufacturers of warranties. The end user survey also highlighted some of these issues. • The CCEB intends to conduct an end user survey in the near future. • ICS end users can potentially be strong advocates for ICS use and have a positive role in creating awareness among other end users. Findings – ICS Manufacturers The major findings from the KIIs with the manufacturers were: • All five manufactures remain committed to producing ICS for the local market. • Sales by manufacturer range from 400 (Ecostories) to over 10,000 (Luxor Global/Life Engineering). • The overwhelming sales channel is through distributors, Only 152 have been sold directly to end users. • The manufacturers do not collect or maintain a database of ICS end users. 21 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT • There is limited interaction between the end users and the manufacturers, even though end users can provide significant feedback on design enhancements and product quality. • Given the projected market size (estimated at about $450 million), there are opportunities for the CCEB-supported manufacturers to scale up production. • The five manufacturers said they have only limited ability to prepare a bankable business plan to submit and secure financing for any future expansion. Findings – ICS Suppliers and Distributors The evaluation team limited its engagement to interacting with 21 suppliers and vendors to obtain information on sales and ICS end user contact details. The following are the findings: • Suppliers/distributors indicate they do not maintain a comprehensive sales/customer database with contact information, even for customers with warranties. • At the time of the evaluation, the suppliers/vendors had sold approximately 71 percent of the ICS purchased. Findings – ICS End Users 26 The following are the principal findings of the end-user survey: • The survey provided informative information but has limitations due to the following: o The pool of ICS users surveyed may not be representative, as participants were selected based on having contact information (cell phone number) obtained from a limited number of distributors (21). o The time for the survey was compressed in part due to local elections. o The coverage was limited to three main geographic regions. o The participants were mainly from low-middle income communities. • The majority of persons interviewed were female (69 percent). • 94 percent reported benefits from using the ICS including energy and cost savings, labor and time savings, improved cooking, health, and less pollution. • A similar number have shared their experiences with family and community members. 26 Annex J is the final report of the ICS end user survey undertaken by ACME subcontractor DPC. It provides comprehensive information on ICS end user experiences. 22 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT • About 90 percent said they had no problems when purchasing the ICS. The following issues were cited by the 10% who stated there were problems: no installment plan, facility for purchase, no subsidy, scarcity of ICS in market, quality of construction, and high cost. o While a limited number of people indicated ICS were not always available in the market, this situation may not be representative, as distributors appear to have about 30 percent of sales in stock currently. • Issues highlighted included the design (notably the issue of refueling) and construction quality, including the fan shutting off suddenly. • ICS end users indicated wood pellets, one of the users preferred fuels, were not always readily available in the market. This resulted in the end user having to spend time preparing fuel wood to a specific size for use in the ICS due to the design of the fuel feed. 5.12. Conclusions – Overall After analyzing and assessing the collected information, the mid-term evaluation team can conclude the program has consistently delivered quality outputs. However, the extent of overall progress to meet its overall goals is hindered by the relative inability of BERC to carry out its mandated functions as the national regulator. The project’s outcomes require a more strategic focus if there is to be scalability and sustainability. It should be noted that the BERC issue is not a new or recent phenomena. Since its creation in 2003, BERC has struggled to provide the anticipated regulatory authority as mandated by law, which has had a detrimental effect on the enabling environment for clean energy. While CCEB support has contributed to BERC’s growth, systemic issues remain, notably in BERC’s ability to attract, recruit and retain staff with appropriate skill sets. CCEB is facing similar challenges to those experienced by other donor initiatives that supported BERC. A flawed enabling environment is a threat to the program achieving its objective and ultimately limits the country’s overall ability to respond to climate change. The adoption and utility of demand side management (Task 4) activities are the most impacted and the program has appropriately limited Task 4 activities recently. The least impacted are the energy planning (Task 2) and the development of the ICS market development stream (Task 5). The industrial energy efficiency work is impacted. But the program’s work to facilitate partnerships with the financial community, in establishing standardized audit protocols and an internationally-recognized certification process for energy efficiency professionals, and the use of grants for energy efficient technologies has meant the program has demonstrated that energy efficiency can benefit industry under the current enabling environment. Work is still necessary to ensure scalability and sustainability post-CCEB. While CCEB activities align with national policy for clean energy (energy efficiency and ICS market development), they need to be better integrated. For example, in Task 3, sufficient capacity to develop the critical mass of audit professionals will not be created as a result of the project and although the certification process introduced through CCEB is internationally recognized, it is not clear that SREDA will leverage the processes and protocols introduced by CCEB when determining the respective standards. Similarly, the project’s grant program has shown that energy savings are achieved when appropriate technologies are deployed, but funding through this program is limited. The financial community recognizes that energy efficiency represents an opportunity for them to expand their lending portfolio. However, feedback from FGDs and several KIIs indicate that CCEB is not successfully addressing one of the main constraints highlighted—awareness of the benefits energy efficiency practices and technologies by industry at large. If efforts are not made to improve the limited awareness, then wide-scale adoption of energy efficiency will occur over an extended period. CCEB can remedy this relatively easily by facilitating key strategic alliances between SREDA, the financial community, energy￾23 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT efficiency service providers and equipment vendors, and industry at large and use industry trade association networks to accelerate awareness creation and facilitate partnerships between industry and investors. Implementing the above requires the program to operate more strategically and should move the program toward outcomes rather than outputs. This should enhance the likelihood of CCEB achieving its strategic objective—strengthened capacity to reduce emissions—and in doing this will increase the program’s contribution to USAID/Bangladesh’s DO4. These adjustments are relatively minor and can be done without modification of the existing contract. 24 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 6. RECOMMENDATIONS 6.1. Overall Recommendations Recommendation 1: Develop a Stakeholder-driven Strategic Plan for Sustainability CCEB has laid the foundation for leaving a positive legacy. Given its accomplishments to date, the project can evolve and begin to prepare for the sustainability of certain programmatic activities. The project has also established strong working relationships that can be leveraged. To this end, the project should look to work with key counterparts and stakeholders and assist them in developing a stakeholder-driven strategic plan focused on sustainability and scalability. Recommendation 2: Emphasize outcomes over outputs to achieve development objectives Although the overall DO is yet to be achieved; CCEB is on track to deliver a meaningful contribution. In its initial stages, CCEB has delivered some quality outputs. Notable examples include the models delivered under Task 2, the certification training for energy auditors and energy audits conducted under Task 3, and the development of local manufacturing capacity under Task 5. With these output accomplishments in place, the program can reorient itself toward development outcomes in the remainder of the life of the program by focusing on sustainability. For example, a national certification standard for energy efficiency auditors, developed by SREDA with CCEB assistance (based on the Association of Energy Engineers (AEE) certification methodology, but adapted to Bangladesh) can help develop a national critical mass of auditors. Similarly, a national energy audit protocol following that currently used by CCEB-trained subcontractors in carrying out investment grade audits will facilitate achievement of the development objective. Recommendation 3: USAID/Bangladesh should engage with GOB on the future development of the BERC: Although CCEB’s activities in Task 1 faced initial challenges; the program has delivered some very sound work that BERC appreciates. BERC, however, still has systemic problems that the program cannot resolve including autonomy and the ability to recruit permanent full-time staffing. A fully functioning BERC that performs its mandated regulatory functions in an effective manner will ultimately be a major driver toward achieving the overall DO, and will help establish a sustainable clean energy market and facilitate the implementation of the 2013 NEEAP and the CAP. Given BERC’s critical role in the energy sector, it is recommended that USAID/Bangladesh engage in a dialogue with GOB to understand if the GOB can establish a time-bound commitment to a fully functional regulator. Recommendation 4: Revise the CCEB M&E Plan to help improve programmatic effectiveness The KPIs relating to CCEB work in Tasks 1, 2, and 5 in the CCEB M&E Plan can be enhanced and provide better information on the outputs and outcomes of CCEB activities. For example, KPI #9, number of cook stoves installed (Task 5), implies information should be collected along the delivery channel. However, currently data collection focuses on the number of sales by ICS manufacturers with no information collected on sales by distributors or on end users. A more robust reporting mechanism could provide a better overview of the development of the market delivery channel. The KPI used for BERC, progress against the maturity model, does not fully reflect how CCEB assistance is enabling BERC to function as an independent, autonomous regulator consistent with international best practices. KPIs that benchmark BERC’s performance relative to its legal mandate would be a better reflection on how effective CCEB assistance is helping with the organization’s development, as well as the development of an enabling governance framework for sector regulation, a prerequisite for implementing clean energy and energy efficiency at the national level. Recommendation 5: Leverage outputs strategically to facilitate scalability and sustainability CCEB’s activities in Task 3 (certification of 88 energy auditors and energy audits of some 100 industrial companies) are perceived as distinct CCEB activities. However, they are consistent with the 2013 NEEAP 25 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT that anticipates a national certification process for energy efficiency professionals and standard protocols for conducting industrial energy audits. CCEB should work closely with SREDA and use the CCEB outputs as the basis for future national standards, thereby facilitating scalability and sustainability and catalyzing the development of an energy efficiency service sector, estimated to be worth $3 to 4 billion. Recommendation 6: Improve and expand stakeholder partnerships There are several existing mechanisms such as the CAP for ICS and the NEEAP that can be leveraged to enhance the program’s potential outcomes. With respect to energy efficiency, CCEB should improve the coordination of its activities with SREDA, GOB in leading the implementation of the action plan to accelerate the adoption of national standards for certification of auditors and energy audit protocols. CCEB should also improve its coordination with national industry and trade associations and the financial community to ensure that these key stakeholders have greater awareness of the benefits of energy efficiency. The relationships within the CAP for ICS and those created with the financial community can be leveraged to ensure sufficient capacity is developed along the ICS supply chain. For example, assisting the manufacturers, as appropriate, to access financing to expand their business operations can help create sustainable local manufacturing capability that can compete with (potentially) imported cook stoves. The CCEB should also work with policy-makers to revisit the duties imposed on imported cook stoves, given the substantive number of ICS needed to meet the ambitious targets of the CAP. Recommendation 7: Make capacity building performance-based and strengthen M&E of training activities Capacity building should be strategic with clearly defined goals. It should be tied to performance objectives for the individuals participating. In addition, the effectiveness of capacity building should be assessed to ensure knowledge transfer. 6.2. Task 1 Recommendations Specific recommendations relating to Task 1 are presented and discussed below. The first recommendation has been discussed in the previous section above and will not be discussed again here. Sustainability Recommendation 8: Task 1: USAID/Bangladesh should engage in dialogue with the GOB to establish a “time-bound” commitment to a fully functioning regulator Although CCEB support has contributed to BERC’s growth, long-standing, systemic issues remain. These issues include a flawed organizational structure, lack of essential systems and processes such as an “e-docketing” system, and an inability to attract, recruit, and retain staff with appropriate skill sets. Unless the systemic issues are addressed, the impact of the support provided by CCEB, will be marginalized, as has been the situation with prior donor initiatives in supporting BERC. Resolving the systemic issues requires appropriate GOB commitment and actions and it is recommended USAID/Bangladesh engage with GOB to understand the expectations and timeline for BERC becoming a fully functioning regulator as envisioned in its legal mandate. Programmatic Improvement Recommendation 9: Task 1: Emphasize regulator function rather than short-term response CCEB’s support to BERC during the remaining life of the program should focus on enhancing the organization’s capacity to function in accordance with international best practices and minimize responding to ad-hoc assistance requests aimed at resolving short-term problems. 26 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Recommendation 10: Task 1: Develop appropriate KPIs for BERC activities The current KPI (progress relative to improvement against maturity model KPIs) does not reflect how the organization is performing relative to its legally-mandated functions. As such, it may not be an ideal measure of how the effective CCEB support has been or how well BERC is progressing toward delivering on its mandated responsibilities. 6.3. Task 2 Recommendations Specific recommendations relating to Task 2 are presented and discussed below. Recommendation 11: CCEB models should be widely disseminated The two models developed by CCEB may well have utility throughout the energy sector and line ministries. Should this be the case, CCEB may wish to consider facilitating distribution to other organizations and providing training in their use. Recommendation 12: Provide additional capacity building to ensure utility of the models Additional capacity building of Power Cell and BPDB is needed to facilitate use of the models and should include hands-on computer modeling training. Recommendation 13: Provide a user manual for the models Findings show that trainees did not receive a user manual for either of the models. Without a reference, the long-term utility of the models may be diminished. 6.4. Task 3 Recommendations Specific recommendations relating to Task 3 are presented and discussed below. Recommendation 14: Establish a formal mechanism for matchmaking between industry and the financial community to promote implementation of clean energy and energy efficiency measures: Findings indicate there is a nascent market for energy efficiency. In addition, there is evidence that the financial community views the energy efficiency market as an opportunity to expand its lending portfolios. CCEB should work with industry (through trade associations and the chamber of commerce) and the financial community to develop a formal mechanism that can accelerate matchmaking between the two sectors and stimulate and expand the fledging market. Recommendation 15: CCEB should support SREDA in developing national standards for certification of energy efficiency professionals and audit protocols Findings indicate there is acceptance by industry and the financial community of the audit protocols and the work of certified local energy efficiency professionals. This can be leveraged by working with SREDA, the GOB entity charged with establishing national standards for energy efficiency professionals and auditing protocols. CCEB can support SREDA and help accelerate the development of appropriate national standards. Recommendation 16: Assist key counterparts and stakeholders to develop financing mechanisms Findings indicate there is a willingness to pay for energy efficiency services as delivered by the CCEB subcontractors and that the financial community views the energy efficiency market as an opportunity to expand its lending portfolios. CCEB can assist these key stakeholders, as well as counterparts such as SREDA, to establish new financing mechanisms for energy efficiency and clean energy. These mechanisms should seek to simplify the application process and reduce time for approval. Recommendation 17: Assess feasibility of transitioning the CCEB grant program to a sustainable financing mechanism CCEB should consider 27 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT the feasibility of transitioning the existing grant program to a sustainable financing mechanism. Recommendation 18: Transition subcontractor activities to fee-for-service to promote sustainability Findings indicate there is a willingness to pay for energy efficiency services as delivered by the CCEB subcontractors. Given that there is a nascent market for these services, CCEB should examine whether it is appropriate to begin phasing-out its current support to ensure they are not subsidizing services that private sector companies are willing to buy. If appropriate, the project should develop a transition strategy in conjunction with key counterparts and stakeholders to facilitate market expansion. Recommendation 19: Emphasize market development rather than financing subcontractor energy audit services Given willingness to pay, CCEB should focus on market development rather than just funding a limited number of audits. Recommendation 20: Facilitate strategic alliances between key counterparts and stakeholders The CCEB-supported activities have fostered acceptance by industry and the financial community and are recognized by key government counterparts. In this next phase of the program, the early successes can be leveraged and used to help engender greater EE awareness through the industrial sector to catalyze implementation in accordance with the goals of the 2013 NEEAP. CCEB can facilitate the development of formal strategic alliances between these key actors as a step to fostering local ownership. 6.5. Task 4 Recommendations There is one specific recommendation relating to Task 4. Recommendation 21: Reprogram resources to other tasks Given the findings that indicate the critical role of BERC in facilitating DSM, all activities relating to Task 4 should be put on hold and resources redistributed to other tasks until the GOB demonstrates its commitment to resolving the systemic challenges at BERC. 6.6. Task 5 Recommendations Specific recommendations relating to Task 5 are presented in the following paragraphs. Recommendation 22: Establish a formal mechanism for match-making between ICS manufacturers and the financial community to accelerate production and product improvement Findings indicate that the local ICS manufacturers have established credibility and acceptance in the marketplace and the financial is are looking for opportunities to invest in clean energy. The production capacity of the CCEB-supported ICS manufacturers can potentially be expanded given the market potential (estimated at $450 million). As such, CCEB should investigate the feasibility of establishing a formal mechanism to facilitate match-making between the manufacturers and the financial community. Recommendation 23: Conduct a follow-up survey to determine long term benefits The majority of ICS users had been using the ICS for less than 12 months. A follow-up survey should be completed to determine the long-term benefits and satisfaction with the ICS. Recommendation 24: Revise the Task 5 M&E reporting to include information on distributor sales: The reporting for the Task 5 component focuses on the manufacturing element of the ICS supply chain only. This provides an incomplete picture of the supply chain and does not provide sufficient, timely feedback to CCEB (and USAID/Bangladesh) on the development of the supply chain. This, in turn, affects program decision-making about where CCEB should be focus its support along the ICS value chain. Expanding the M&E reporting system to include information from ICS distributors with respect to sales can provide valuable information not only for CCEB but also can help inform key stakeholders involved in the CAP for ICS. 28 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Recommendation 25: Use ICS end users as advocates in CCEB promotion activities: Findings from the ICS end users survey indicate that end users discuss their experiences with the ICS not only with direct family members but also with their local community. CCEB should look to use a number of the end users as advocates in their outreach initiatives to facilitate and accelerate village level acceptance of the ICS. Recommendation 26: ICS use in low-income communities needs to be evaluated The majority of users surveyed were from medium income and medium-low income communities, and did not indicate the purchase of the ICS was a major issue. However, a survey of low-income rural poor communities should be done to determine whether the pricing is appropriate and not a significant barrier to future deployment on a large scale. Recommendation 27: The design and construction of the ICS needs to be further evaluate A number of users expressed concerns with the ICS designs and construction quality. Requests for improvements included: increased ease of refueling, reduced cooking time, less smoke, and more energy savings. These considerations need further investigation by the respective manufacturers. Recommendation 28: Increase support toward point of sale to accelerate sales and deployment Findings indicate that there are lags in end user sales relative to distribution from the manufacturers. A review of the sales and marketing should be done to identify improvements. By increasing support to sellers, CEB can accelerate sales and use of the ICS. Recommendation 29: Try to ensure ICS supply and availability is closely aligned with demand The ICS manufacturers should work more closely with distributors and vendors to ensure production and demand along the supply chain is well managed, mitigating either over- or undersupply. 29 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 7. LESSONS LEARNED Evidence-based lessons learned gleaned from the document review, KIIs, FGDs and the ICS survey are presented below. 7.1. Effectiveness: Lesson Learned 1: Use a strategic management approach rather than activity-focus to achieve development DO4 The implementation approach to CCEB has emphasized activity over strategy. CCEB may have benefitted from a strategic situational review as an input into their annual work plans. 7.2. Sustainability: Lesson Learned 2: Counterpart ownership is critical for success and sustainability The main tasks being implemented by CCEB are consistent with major policy initiatives, notably CCEB Task 3 and the 2013 NEEAP and Task 5 and the CAP for ICS. Given that CCEB tasks have emphasized the private sector, there is a risk that the project’s achievements are not properly leveraged by key government counterparts. There is limited evidence that GOB policymakers feel ownership of CCEB activities. Within the time remaining, it will be imperative for the project to strengthen its relationship with key counterparts to ensure they can leverage the CCEB activities to enhance sustainability. Lesson Learned 3: Communication with key stakeholders is essential to scalability and sustainability Key sectors of the economy have limited understanding of the benefits of clean and efficient energy. For example, the majority of the focus group discussion participants cited a “lack of awareness and information on energy efficiency technologies and practices and access to financing.” CCEB has done a very good job of demonstrating clean and efficient energy’s benefits to direct beneficiaries in industry and the financial sector. However, CCEB has not yet utilized the results to promote long-term sustainability and scalability. 7.3. Program Improvement: Lesson Learned 4: Capacity building should be strategic To facilitate sustainability, CCEB was expected build the capacity of key counterparts, notably within BERC and the GOB entities, to develop evidence-based policies on low emissions development strategies. However, the outcome of these efforts is somewhat limited. Capacity building at BERC centered on some of BERC’s mandated functional responsibilities but success was hindered by BERC’s limited human resources. Positions are often filled by seconding staff from GOBGOB entities for limited periods of time. In addition, these personnel do not see a clear career path, as is present in other GOB entities. SREDA is a key counterpart with respect to long-term scalability and sustainability of several of CCEB’s activities (Task 3, 4 and 5). CCEB should ensure it coordinates very closely with SREDA during the remainder of the project. However, SREDA is a young organization and needs to develop its capacity. Lesson Learned 5: Capacity building should be evaluated for effectiveness While the project’s M&E reporting includes a capacity building indicator, there is no evidence that CCEB is evaluating the long-term effectiveness of capacity building. 30 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex A: Gantt Chart of Activities Activity* Resources Activity Location Week 1 2 3 4 5 6 7 8 9 10 11 12 10/31- 11/6 11/7- 11/13 11/14- 11/20 11/21- 11/27 11/28- 12/4 12/5- 12/11 12/12- 12/18 12/19- 12/25 12/26- 1/1 1/2- 1/7 1/9- 1/15 1/16- 1/22 Inception Work plan preparation, document review and situational analysis Conference call with ACME COP and team leader John Dalton (ACME COP) ; Richard P. Smith (TL) Richard P. Smith (USA); John Dalton (Bangladesh) Travel to Bangladesh Richard P. Smith (TL) Travel Conduct document review Richard P. Smith (TL); Sattya Bhattacharjee (EE) Bangladesh Develop Evaluation Design Matrix Richard P. Smith (TL) Bangladesh Develop Evaluation schedule Richard P. Smith (TL); Sattya Bhattacharjee (EE) Bangladesh Develop survey instruments based on SOW and evaluation questions Richard P. Smith (TL); Sattya Bhattacharjee (EE) Bangladesh Develop draft work plan Richard P. Smith (TL); Sattya Bhattacharjee (EE) Bangladesh Submit draft work plan Richard P. Smith (TL); Bangladesh 11/18 11/24 12/5 Schedule key informant interviews Tasmin Akanada Ethu (PA) Bangladesh 31 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Activity* Resources Activity Location Week 1 2 3 4 5 6 7 8 9 10 11 12 10/31 -11/6 11/7- 11/13 11/14 - 11/20 11/21 - 11/27 11/28- 12/4 12/ 5- 12/ 11 12/12 - 12/18 12/19 - 12/25 12/26 -1/1 1/2 - 1/7 1/9- 1/15 1/16- 1/22 Field Work USAID briefings, key informant interviews, data synthesis, analysis, and follow-up CCEB evaluation team planning meeting, Dhaka John Dalton, (ACME COP); Richard P. Smith ( TL); Sattya Bhattacharjee (local EE ) Bangladesh 11/10 In-brief meeting with USAID/Bangladesh to review draft work plan and finalize schedule of field activities) John Dalton (ACME COP); Richard P. Smith (TL); Sattya Bhattacharjee (EE) Bangladesh 11/17 2nd In-brief meeting with USAID/Bangladesh John Dalton (ACME COP); Richard P. Smith (TL); Sattya Bhattacharjee (EE) Bangladesh 11/22 Finalize work plan Richard P. Smith (TL); Sattya Bhattacharjee (EE) Bangladesh 12/4 Submit final work plan Richard P. Smith (TL); Sattya Bhattacharjee (EE) Bangladesh 12/ 5 Conduct key informant interviews, focus group discussions, Richard P. Smith (TL); Sattya Bhattacharjee (EE); Naba Krishna Muni, (CBES); Tasmin Akanada Ethu, (PA) Bangladesh KIIs will be conducted on an intermittent basis Complete Preliminary Observations presentation Richard P. Smith (TL); Sattya Bhattacharjee (EE) Bangladesh 12/3 32 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Activity* Resources Activity Location Week 1 2 3 4 5 6 7 8 9 10 11 12 10/31- 11/6 11/7- 11/13 11/14- 11/20 11/21- 11/27 11/28- 12/4 12/5- 12/11 12/12- 12/18 12/19- 12/25 12/26- 1/1 1/2-1/7 1/9- 1/15 1/16- 1/22 Field Work USAID briefings, key informant interviews, data synthesis, analysis, and follow-up Meet with USAID/Bangladesh to present Summary of preliminary observations Richard P. Smith (CCEB evaluation TL); Engr. Sattya Bhattacharjee (National Energy Expert); John Dalton (ACME COP) Bangladesh 12/3 Depart from Bangladesh (December 4) Richard P. Smith (CCEB evaluation TL); 12/4 Conduct surveys of direct beneficiaries ACME consultants, DPC Group Bangladesh Return to Bangladesh (December 23) Richard P. Smith (TL); 12/23 Data Synthesis Richard P. Smith (TL); Sattya Bhattacharjee (local EE) Bangladesh Presentation of Findings and Debriefing to USAID/Bangladesh senior management 27 Richard P. Smith (TL); Sattya Bhattacharjee (local EE) Bangladesh 1/7 Depart from Bangladesh (January 8, 2016) Richard P. Smith (TL); 1/8 27 To be confirmed by USAID/Bangladesh 33 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Activity* Resources Activity Location Week 1 2 3 4 5 6 7 8 9 10 11 12 10/31- 11/6 11/7- 11/13 11/14- 11/20 11/21- 11/27 11/28- 12/4 12/5- 12/11 12/12- 12/18 12/19- 12/25 12/26- 1/1 1/2- 1/7 1/9- 1/15 1/16- 1/22 Produce Evaluation Report Conduct follow-up interviews, as needed Richard P. Smith (TL); Sattya Bhattacharjee (local EE) Bangladesh Develop proposed outline of Evaluation Report Richard P. Smith (TL); Sattya Bhattacharjee (local EE) Bangladesh 1/2 Draft Evaluation Report. due on January 5, 2016 Richard P. Smith (TL); Sattya Bhattacharjee (local EE) USA; Bangladesh 2/8 QA/QC draft evaluation report Richard P. Smith (TL); Sattya Bhattacharjee (local EE) Bangladesh 2/9 Submit Draft Evaluation Report Richard P. Smith (TL); Sattya Bhattacharjee (local EE) John Dalton (ACME COP) Bangladesh 2/10 Incorporate comments and feedback from USAID/Bangladesh Richard P. Smith (TL); Sattya Bhattacharjee (local EE) USA; Bangladesh 2/21 QA/QC final evaluation report John Dalton (ACME COP) Bangladesh 2/22 Submit Final Evaluation Report John Dalton (ACME COP) and IBTCI HQ Bangladesh 2/23 34 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Activity* Resources Activity Location Week 1 2 3 4 5 6 7 8 9 10 11 12 10/31- 11/6 11/7- 11/13 11/14- 11/20 11/21- 11/27 11/28- 12/4 12/5- 12/11 12/12- 12/18 12/19- 12/25 12/26- 1/1 1/2- 1/7 1/9- 1/15 1/16- 1/22 Program Management USAID briefings, progress reporting Submit weekly progress report to USAID/Bangladesh Richard P. Smith (TL); Bangladesh Meet with USAID/Bangladesh to review draft work plan and finalize schedule of field activities Richard P. Smith (TL); Sattya Bhattacharjee (local EE) Bangladesh 11/15 Debriefing presentation to USAID/Bangladesh senior management Richard P. Smith (TL); Sattya Bhattacharjee (local EE) John Dalton (ACME COP) Bangladesh 1/6 Debriefing presentation to GOB officials (if requested/required by USAID/Bangladesh) Richard P. Smith (TL); Sattya Bhattacharjee (local EE) John Dalton (ACME COP) Bangladesh 1/6 *Deliverables are in bold. Timing assumes 5-day work in the in US and 6-day workweek in Bangladesh. Abbreviations: TL Team Leader EE Energy Expert COP Chief of Party 35 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex B: Evaluation Design Matrix Evaluation Question Proposed Data Collection Methodology Proposed Questions for Data Collection Instrument Component 1 Component 2 Respondent Group – Task 1 Respondent Group – Task 2 Respondent Group – Task 3 Respondent Group – Task 4 Respondent Group – Task 5 1. To what extent is the CCEB project on track, in terms of progress and outcomes, to meet its overall goals for the five tasks under the components A and B? SDR; KII, FGD and PRA survey (Task 5 only) 1.1 How effective has the program’s technical assistance approach been in institutional strengthening, capacity building, and catalyzing clean energy and increasing access to finance? 1.2 How effective has the CCEB program been in achieving results considering the resources expended? 1.3 How has participant training under CCEB built capacity of partner institutions, including capacity of women members, to enhance market development? 1.4 What is the likelihood of achieving the expected results under each task? USAID; BERC; IP USAID; IP BPDB USAID; IP; Subcontractors; Direct Beneficiaries; Financial Community USAID; IP; BERC; Utilities USAID; IP; SREDA; donors;; Direct Beneficiaries: Manufacturers, vendors; end users; IDCOL 2. What are the opportunities to enhance programmatic approach and effectiveness plausible under the stipulation of the contract? DR; KII, FGD and PRA survey (Task 5 only) 2.1 What constraints are not currently being addressed by CCEB? 2.2 What can CCEB do to improve its efforts to catalyze clean energy considering the progress to date and remaining timeline? BERC; IP; USAID USAID; IP BPDB USAID; IP; Subcontractors; Direct Beneficiaries; Financial Community USAID; IP; BERC; Utilities IP; SREDA; Direct Beneficiaries: Manufacturers and vendors; end users; IDCOL 3. What have been the major constraints and opportunities with DR; KII; FGD and PRA survey (Task 5 only) 3.1 What constraints are being addressed by CCEB? USAID; BERC; IP USAID; IP BPDB USAID; IP; Subcontractors; Direct Beneficiaries; USAID; IP; BERC; Utilities IP; SREDA; Direct Beneficiaries: Manufacturers 36 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Evaluation Question Proposed Data Collection Methodology Proposed Questions for Data Collection Instrument Component 1 Component 2 Respondent Group – Task 1 Respondent Group – Task 2 Respondent Group – Task 3 Respondent Group – Task 4 Respondent Group – Task 5 respect to sustainability of the interventions? What measures should be taken to enhance sustainability? 3.2 How sustainable and scalable are the activities being undertaken by CCEB? 3.3 How can CCEB enhance the likelihood of sustainability post￾USAID assistance? 3.4 Are you aware of a sustainability plan post USAID assistance? 3.5 What increase in business activity have subs experienced? 3.6 What sustainability plan do the subs have? Financial Community and vendors; end users; IDCOL 4. To date, how effectively has gender been integrated or incorporated in the interventions? What have been the challenges and opportunities, if any? DR; KII; FGD and PRA survey (Task 5 only) 4.1 To what extent has CCEB promoted and integrated gender into its activities? 4.2 Are any CCEB activities specifically targeted toward gender integration? 4.3 What challenges has CCEB encountered in integrating gender into its activities? 4.4 Are there opportunities that CCEB is using to facilitate and promote gender integration? USAID; BERC IP USAID; IP BPDB USAID; IP; Subcontractors; Direct Beneficiaries; Financial Community USAID; IP; BERC; Utilities USAID; IP; SREDA; Direct Beneficiaries: Manufacturers and vendors; NGOs, end users; IDCOL 37 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex C: Data Collection Schedule and Data Analysis Methodologies CCEB Mid-Term Evaluation Field Work Schedule Date Activity Time Frame Location Activity 4-Nov Onboarding USA DR 5-Nov Team call USA/Bangladesh ETGA 6-Nov Off day 7-Nov Travel to Bangladesh All Day 8-Nov Travel to Bangladesh All Day 9-Nov Document review All Day ACME office DR 10-Nov Document review All Day ACME office DR 10-Nov Team Planning meeting All Day ACME office ETGA 11-Nov Work plan development All Day ACME office DR 12-Nov Work plan development All Day ACME office DR 13-Nov Off day 14-Nov Work plan development All Day ACME office ETGA 15-Nov Work plan development All Day ACME office ETGA 16-Nov Team Planning meeting AM ACME office ETGA 16-Nov Deloitte CCEB meeting PM CCEB office KII 17-Nov USAID/Bangladesh in-brief AM USAID Mission Briefing 17-Nov Work plan finalization PM ACME office ETGA 18-Nov Deloitte follow-up meeting AM CCEB office KII 18-Nov Sodev (Task 3 subcontractor) PM Dhaka KII 19-Nov Document review/Field work planning AM ACME office DR 19-Nov DPC meeting on ICS survey PM ACME office ETGA 20-Nov Off day 21-Nov Wellmake (Task 3 subcontractor) AM Dhaka 1000 KII 21-Nov BD Technology (Task 3 subcontractor) PM Dhaka 1000 KII 21 Nov DPC Group PM Dhaka SOW for ICS survey 22-Nov USAID/Bangladesh in-brief 2 AM USAID Mission Briefing 22-Nov ICS survey SOW PM ACME ETGA 23-Nov ICS survey SOW update AM Dhaka ETGA 23-Nov DPC Group PM Dhaka ETGA 24-Nov Deloitte follow-up meeting - ICS AM CCEB office KII 24-Nov Finalize work plan PM Dhaka ETGA 25-Nov Preparation for Focus group meetings Task 3 All day Dhaka FGD 25-Nov Schedule Focus Group Discussions PM Dhaka FGD 25 - Nov Finalize ICS field survey with DPC Group PM Dhaka ETGA 26-Nov Revision of ICS survey instruments AM Dhaka KIIs 26-Nov Meeting with Deloitte PM Dhaka KIIs 27-Nov Off day 38 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT CCEB Mid-Term Evaluation Field Work Schedule Date Activity Time Frame Location Activity 28 Nov DPC meeting re: ICS survey AM Dhaka ETGA 28 Nov Finalize schedule for fieldwork All day Dhaka ETGA 29-Nov SREDA AM Dhaka FGD 29-Nov BDPC Task 2 PM Dhaka KII 29-Nov Power Cell (Task 2) PM Dhaka FGD 30-Nov Task 3 Steel Industries AM Dhaka FGD 30-Nov BERC Task 1, Task 2 and Task 4 AM Dhaka KII 30-Nov Task 3 Financial Community PM Dhaka FGD 1-Dec Task 3 Industry group 1 AM Dhaka FGD 1-Dec Task 3 Industry group 1 PM Dhaka FGD 2-Dec Observation of DPC training and briefing on CCEB/ICS for ICS surveyors and enumerators AM Dhaka Briefing 2-Dec Preparation for ICS surveyor briefing and preliminary observations briefing PM Dhaka ETGA 3-Dec ACME briefing on ACME quality standards for ICS survey enumerators and surveyors AM ACME office ETGA 3-Dec USAID/Bangladesh progress briefing PM USAID Mission Briefing 5-Dec Off day Dec 6-10 Task 5 ICS Survey - Manufacturers Dec 13 - 17 Task 5 ICS Survey - Vendors Dec 20 - 24 Task 5 ICS Survey - End users and NGO Dec 6 – 31 Additional KIIs with key stakeholders, IP and USAID/Bangladesh Dec 26 - 31 SDR and Data Synthesis Jan 2-4 Draft Evaluation report 5-Jan Presentation for debriefing All Day ACME office ETGA 6-Jan USAID/Bangladesh debriefing on Evaluation report AM USAID Mission Briefing 6-Jan GOB debriefing on Evaluation report PM TBD Briefing Legend Survey Off day Abbreviations Key Informant Interviews KII Focus Group Discussion FGD Document review DR Secondary data review SDR Evaluation team group activity ETGA Structured Group Discussion SGD To Be Determined TBD 39 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Data Analysis Methodologies The data collected on the CCEB included information from KIIS and FGDs with beneficiaries (tasks 1, 2, 3, 4, and 5), key stakeholders (tasks 3 and 4) and key actors (CCEB program task leads for Tasks 1 through 5, the CCEB COP, USAID/Bangladesh COR and Task 3 subcontractors). The names of the responders from each task are included in Annex B Evaluation Design Matrix. The information included both qualitative and quantitative data. During the KIIs, key informants were asked to provide responses to a series of quantitative questions using a Likert-type scale. The number of responses to each question were summed for all respondents to ascertain the total number of responses to each of the individual evaluation questions and then were compiled into charts. The charts are included in the body of the report. Responses to qualitative questions were also compiled and analyzed. The replies were analyzed by their relevance to each respective task to identify key findings, recommendations and any lesson learned. Any synergies across tasks were also identified together with any associated recommendations etc. The outputs are presented in the body of the report both by task and for the CCEB as a whole, where applicable. 40 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex D: Key Informant Interview Questions KEY INFORMANT INTERVIEW Code Number: Name: Title: Organization Name and Location: Preamble: The Accelerating Capacity for Monitoring and Evaluation program (ACME) is undertaking a mid-term evaluation of the Catalyzing Clean Energy in Bangladesh (CCEB) program being implemented by Deloitte LLP. The evaluation is being carried out at the request of the United States Agency for International Development Mission to Bangladesh The objective of the evaluation is to determine the extent to which the CCEB program is on track (including process and outcomes) to meet its overall goals across the two major CCEB components. The two components are: Component 1: Improve Enabling Environment for Low Emissions Development and Component 2: Increase Energy Efficiency and Conservation. As part of the evaluation, I, _____________________________________________ want to ask you some questions about the CCEB. The questions will be on various aspects of the CCEB. To ensure we capture your responses appropriately, my colleague, _________________________________________ will be taking notes, and with your permission we would also like to record our discussion so we can accurately capture your responses. Please note, your responses will be treated confidentially and we will not be using your name, official designation or title in our evaluation report. 41 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT KEY INFORMANT INTERVIEW Please categorize you interaction with CCEB Counterpart Contractor or vendor Beneficiary Donor Please describe your CCEB interaction (please circle all those that apply) Policy and regulatory framework Capacity Building Project implementation Does the capacity exist in the responsible organizations to implement clean energy in the country? Yes No Not sure Are you or your organization better able to carry out your responsibilities as a result of CCEB? Yes No Not sure Question Scale Not at all Not very much Some￾what Very Much Extremely How aware are you of the Catalyzing Clean Energy in Bangladesh program (CCEB)? How aware are you of the CCEB goals and objectives? What is your organization’s involvement with respect to CCEB program focus areas? What was your organization’s expectation for the CCEB program? What do you see as the key accomplishments of CCEB? Given your involvement with CCEB, what do you consider as the key challenges to this program? What are the strengths of the program? How effectively has the CCEB coordinated with other stakeholders in its implementation? What are the key lessons learned from your interactions with CCEB? In your opinion, what are the key constraints to promoting clean energy development in the country? Scale Not at all Not very much Some￾what Very Much Extremely How much does CCEB help in addressing the constraints? How much impact has CCEB had on the policy and regulatory framework in the country? How much impact has CCEB had on building human capacity in clean energy? How much impact has CCEB had on increasing clean energy utilization? How sustainable are the outcomes of CCEB? How much interaction has your organization had with the CCEB? Do you use anything you have learned from CCEB as part of carrying out your specific duties? 42 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT MID-TERM EVALUATION QUESTION 1 Question: 1. To what extent is the CCEB project is on track, in terms of progress and outcomes, to meet its overall goals (for the five tasks under the components 1 and 2)? Question Scale Not at all Not very much Some￾what Very Much Extremely 1.1 (a) How effective has the program’s technical assistance approach been in institutional strengthening? Please elaborate 1.1 (b) How effective has the program’s technical assistance approach been in building capacity? Please elaborate 1.1 (c) How effective has the program’s technical assistance approach been in catalyzing clean energy development? Please elaborate 43 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 1.1 (d) How effective has the program’s technical assistance approach been in increasing access to financing? Please elaborate 1.2. How effective has the CCEB program been in achieving outputs or results? Question Scale Not at all Not very much Some￾what Very Much Extremely 1.3. To what extent has participant training under CCEB been able to build capacity of partner institutions, including capacity of women members, to enhance market development? Please elaborate 1.4 Given the progress to date, to what extent is CCEB on track to achieve the expected results? Please elaborate 44 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT MID-TERM EVALUATION QUESTION 2 Question: 2. What are the opportunities to enhance programmatic approach and effectiveness plausible under the stipulation of the contract? Question Scale Not at all Not very much Some￾what Very Much Extremely 2.1 Are the constraints to clean energy development you mentioned earlier being addressed by CCEB? Please elaborate 2.1 (a) What constraints are not currently being addressed by CCEB? Please elaborate 2.2 What can CCEB do in the remaining time period to help remove any constraints and achieve desired results considering the progress to date? Please elaborate 45 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT MID-TERM EVALUATION QUESTION 3 Question: 3. What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? Question Scale Not at all Not very much Some￾what Very Much Extremely 3.1 How sustainable and scalable are the activities being undertaken by CCEB? Please elaborate Question Scale Not at all Not very much Some￾what Very Much Extrem ely 3.2 Has your involvement in CCEB helped your business activities? Please elaborate 3.3 Are the activities you are undertaking for the CCEB part of your long term business plan and what plan do you have to sustain the activities? 46 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT MID-TERM EVALUATION QUESTION 4 Question: 4. To date, how effectively has gender been integrated or incorporated in the interventions? What have been the challenges and opportunities, if any? Question Scale Not at all Not very much Some￾what Very Much Extremely 4.1 To what extent has CCEB promoted and integrated gender into its activities? Please elaborate 4.2 Are any CCEB activities specifically targeted toward gender integration? 4.3 What challenges has CCEB encountered in integrating gender into its activities? Please elaborate 4.4 In your opinion do you see CCEB taking existing opportunities to facilitate and promote gender integration in its activities? Please elaborate 47 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex E: Focus Group Discussion Guides FOCUS GROUP DISCUSSION GUIDE Code Number: Name: CCEB Evaluation Focus Group Discussions Participant Group: Date: Location: Welcome: Please let me first thank you for coming today to discuss clean energy and energy efficiency in Bangladesh. My name is Richard P. Smith, and I am joined by Engineer Sattya Bhattacharjee and Ms. Tasim Akanda Ethu. We are from the Accelerating Capacity for Monitoring and Evaluation program (ACME), who has been asked by United States Agency for International Development Mission to Bangladesh (USAID/Bangladesh) to carry out a mid-term evaluation of the Catalyzing Clean Energy in Bangladesh (CCEB) program being implemented by Deloitte LLP. The objective of the evaluation is to determine the extent to which the CCEB program is on track (including process and outcomes) to meet its overall goals. CCEB Overview: The Catalyzing Clean Energy in Bangladesh (CCEB) program is intended to support energy sector development for energy security, economic growth, and climate change mitigation to Bangladesh and build capacity to design and implement supportive policies and regulations, and increase utilization of clean energy approaches and technologies for energy sector development on a low carbon trajectory. The CCEB program works across two major Components – Component 1: Improve Enabling Environment for Low Emissions Development and Component 2: Increase Energy Efficiency and Conservation. Under these there are five tasks: improve the regulatory environment for Clean Energy Development, strengthen analytical capacity for energy planning and policy making, promote energy efficiency analysis and adoption, adopt demand-side management programs for utilities and conduct market analysis and development for improved cook stoves. Ground Rules: We have asked you to participate in this focus group discussion so we can listen to and better understand your thoughts and ideas with respect to clean energy in the country. Please note, there are no right and/or wrong answers – just differences of opinion. I encourage you to share your thoughts and ideas with us even if you do not necessarily agree with what others say. Please also understand we welcome both negative and positive comments – we are here to learn and negative feedback can often help better inform and ultimately lead to better program implementation. This is one of a number of focus group discussions we are having. You are all here as you have been involved in the activities of CCEB in some way, either as a counterpart, direct beneficiary, or an implementing partner. My role as moderator is to help guide the discussion and I encourage your active participation. Ms. Ethu will be taking notes and, with your permission, we wish to record our discussion so we can accurately capture your responses. We find we receive very helpful ideas and insights from our group participants. However, we are often unable to write fast enough to capture everything. Please note, we will be using first names only during the discussion and we will not use anyone’s complete name in our report. What you say will be treated as highly confidential. I kindly request you either turn off or mute your cell phones. This will facilitate the discussion for everyone. However, if you need to answer a call I request you excuse yourself and take the call outside of the room. 48 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT FOCUS GROUP DISCUSSION – TASK 3 Focus Group Discussion Questions What are the challenges you face in growing and improving your business? How do energy issues impact your business? Are current polices likely to encourage you to adopt clean energy technologies and practices? Is there sufficient awareness of the benefits of adopting clean energy technologies throughout the country? Does the country have sufficient human capacity with appropriate training to use clean energy technologies in a sustainable manner? Can you tell us about any experience with clean energy and energy efficiency outside of your interactions with CCEB? If you were running the country, and could make one change that would accelerate use of clean energy what would you do? If you were running the CCEB and could make one change that would accelerate use of clean energy what would you do? 49 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex F: ICS Survey Instruments ICS SURVEY INSTRUMENT Code Number: Name: Title: Organization Name and Location: Preamble: The Accelerating Capacity for Monitoring and Evaluation program (ACME) is undertaking a mid-term evaluation of the Catalyzing Clean Energy in Bangladesh (CCEB) program being implemented by Deloitte LLP. The evaluation is being carried out at the request of the United States Agency for International Development mission to Bangladesh. The objective of the evaluation is to determine the extent to which the CCEB program is on track (including process and outcomes) to meet its overall goals across the two major CCEB components. The two components are: Component 1: Improve Enabling Environment for Low Emissions Development and Component 2: Increase Energy Efficiency and Conservation. CCEB Overview: The Catalyzing Clean Energy in Bangladesh (CCEB) program is intended to support energy sector development for energy security, economic growth, and climate change mitigation to Bangladesh and build capacity to design and implement supportive policies and regulations, and increase utilization of clean energy approaches and technologies for energy sector development on a low carbon trajectory. One of the major areas of focus has been to support the market development for improved cook stoves. As part of the evaluation, I, _____________________________________________ want to ask you some questions about your interactions the CCEB with respect to the development of the market for improved cook stoves. To ensure that we capture your responses appropriately, my colleague, _________________________________________ will be taking notes, and, with your permission we would also like to record our discussion. Please note, your responses will be treated confidentially and we will not be using your name, official designation or title in our evaluation report. Please categorize you interaction with the CCEB ICS activities Manufacturer Supplier End user Donor Financing organization Please describe the focus of your work on ICS (please check all that apply) Policy Market Delivery Promotion & Outreach Financing 1. In your opinion does the capacity exist in the country to achieve the 2030 targets of the Country Action plan for ICS? If no, please elaborate The question is intended to identify any capacity gaps where CCEB could provide additional training but only as it pertains to the development of the market supply chain Yes No Don’t know 50 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT ICS Survey Instrument – Manufacturers Question Answer 2. What types of business are you classified as? For Profit Not for Profit 3. Is your business woman-owned? Yes No 4. What percentage of your company’s revenues is directly attributable to sales of ICS? Less than 10% 20-50% Greater than 50% 5. Please categorize your business involvement in the market supply chain Manufacture only Manufacture and supply to distributors Manufacture and sale to end users 6. Was your company producing ICS prior to support from the CCEB? Yes No 7. Please describe the benefits of the support you received from CCEB? (select all that apply) Helped improved manufacturing process for existing product Helped design and introduce new product Helped expand existing manufacturing capacity Trained and built capacity of staff in management, operations and sales and marketing Increased access to financing for ICS business line Improved marketing and sales processes Improved management of business Reduced manufacturing costs 8. Has the performance of your Cook Stove (CS) design undergone testing by CCEB and relevant local bodies Yes No 9. Has your CS design received any test/performance certification from relevant local bodies? Yes No 10. Please provide information on your company’s gender profile (either in percentage terms or absolute numbers) Staff Male Female 11. Is your business woman-owned? Yes No 12. Has your company added staff as a result of its ICS business? Yes No If yes, please provide the number of staff added Male Female 13. When did you start selling ICS? 14. Are you willing to share information about your sales? 15. What are your annual number of ICS units sold? 2013 2014 2015 Yes No Do not have information 2013 2014 2015 16. What are your sales projections figures in ICS units for the upcoming years? 2016 2017 2018 17. Are you interested in expanding your ICS business activities? Yes No Not Sure 18. Do you have a business plan for expanding your ICS business? Yes No Not Sure 51 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 19. Are there constraints that prevent you from expanding your business? Yes No Not Sure 19.1 If yes, what are could be the major constraints to business expansion? (Please circle all that apply) Lack of access to financing Limited availability of manufacturing equipment Lack of trained staff Lack of demand for ICS product Other – please describe 20. Do you intend to expand operations into other parts of the market supply chain? Yes No Not Sure 52 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT ICS Survey Instrument – Suppliers/NGO Community Question Answer 1. What types of business are you classified as? For Profit Not for Profit 2. Is your business woman-owned? Yes No 3. What percentage of your company’s revenues are is directly attributable to sales of ICS? Less than 10% 20-50% Greater than 50% 4. Please categorize your business involvement in the market supply chain Supplier to end-users only Supply to other distributors Advocacy organization and sales to end users 5. Is or was your organization involved in the CAP for ICS prior to support from the CCEB? Yes No 5.1 If yes, please describe 6. What has been the outcomes of your organizations interactions with CCEB? Please circle all that apply Trained and built capacity of staff Increased access to financing for ICS Enhanced marketing and sales processes Improved management of organization Reduced marketing costs 6.1 If other please elaborate Other 7. Please provide information on your company’s gender profile (either in percentage terms or absolute numbers) Male Female 8. Has your organization added staff for ICS activities as a result of support from CCEB ICS business? Yes No 8.1 If yes, please provide the number of staff added Male Female 9. Are you willing to share information about your sales? Yes No Do not have information 10. What are your annual number of ICS units sold? 2013 2014 2015 11. What are your sales projections figures in ICS units for the upcoming years? 2016 2017 2018 12. Are you interested in expanding your ICS activities? Yes No Not Sure 13. Do you have a plan for expanding your ICS activities? Yes No Not Sure 14. Are there constraints that prevent you from expanding your business? Yes No Not Sure 15. If yes, what are the major constraints to expansion? (Please circle all that apply) Lack of access to financing Limited availability of manufacturing equipment 53 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Lack of trained staff Lack of demand for ICS product Other – please describe 16. Do you intend to expand operations into other parts of the market supply chain? Yes No Not Sure 54 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT ICS Survey Instrument – Other Stakeholders Question Answer 1. Please categorize you interaction with the CCEB ICS activities Manufacturer/Sup plier Other End user Donor Financing organization 2. Please describe the area of focus of your work on ICS (please check all that apply) Policy Market Delivery Promotion & Outreach Financing 3. Are you aware of the GOB targets under the Country action plan for ICS? Yes No Don’t know 4. If yes, what do you estimate the cumulative number of ICS deployed by the year indicated? 2017 2020 2025 5. Are you aware of the estimated numbers of ICS deployment as a result of CCEB support? Yes No Don’t know 5.1 If yes, in your opinion does the capacity exist in the market supply chain exist to achieve the CCEB targets? Yes No Don’t know 6. If you are a member of the financial community please describe your role Micro-financing Commercial Bank Development bank 7. Do you provide access to financing on soft terms? Yes No 7.1 If yes, please describe 8. Please elaborate who are your target customers? Manufacturers Suppliers NGO organizations Consumers Rural Communities 9. Please describe the outcomes of any support you received from the CCEB? (circle all that apply) Improved understanding of the ICS market Expanded support activities to ICS market Enhanced staff capabilities Increased financing to ICS businesses 10. Are you interested in expanding your ICS-related activities? Yes No Not Sure 11. Do you have a plan for expanding your ICS activities? Yes No Not Sure 12. Are there constraints that prevent you from expanding? Yes No Not Sure 12.1 If yes, what are the major constraints to expansion? (Please circle all that apply) Lack of access to financing Limited availability of local ICS manufacturing capacity 55 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Lack of trained staff Lack of demand for ICS product Other – please describe 13. Do you intend to expand your operations into other parts of the market supply chain? Yes No Not Sure 13.1 If yes, please elaborate 56 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT ICS Survey Instrument – End Users Question Answer 1. Do you use your ICS cook stove for all cooking? Yes No 1.1 If no, what other cook stoves do you use? 2. Please describe what you cook with the ICS? 3. What do you cook with your other cook stoves? 4. If you have more than one cook stove which do you prefer? Please circle ICS Other No preference 5. Do you pay for your fuel for cooking? Yes No 6. How many people are you cooking for? 7. What fuel/energy sources do you use in your cook stove? Firewood Agricultural waste Other 8. What factors influenced your decision to get a new ICS? (circle all that apply) Price Design Gift Energy saving Cost saving Time saving Marketing by suppliers Recommendation of other users in community Manufacturer’s warranty Cost subsidy Other 9. Who made the final decision to buy the ICS cook stove? (please circle all that apply) Principal user/ Cook Head of household The main financial contributor to the family Third party Family/community decision 57 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 10. Do you know the name of the brand or manufacturer of your ICS? Yes No 10.1 If yes, please provide 11. Were other ICS brand available for you to consider? Yes No 11.1 If yes, please provide the names of the other ICS designs 12. Do you know if the price of your ICS was subsidized? Yes No Not Sure 13. If you were not the person who made the decision to buy the ICS, do you know what factors influenced the buyer’s decision Yes No Not Sure 14. Have you shared your experiences with the ICS with other? Yes No Not Sure 14.1 Please circle who you have shared your experiences with Other household/family members Other community members Suppliers of ICS Other persons 15. Have you benefitted from the ICS? Yes No Not Sure 15.1 If yes, please circle all the benefits from using the ICS. Better quality of cooking Savings in fuel wood use Cost savings Time saving in collecting fuel wood Health benefits Improved home environment due to reduced smoke Other – please describe 16. What problems have you had with the ICS? No Availability in market Poor? Quality of construction No saving in cooking time No improvement in quality of cooking No savings in fuel wood 58 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT No change or improvement in Health No improvement in home environment No cost savings Other – please describe 17. If cook stoves were available that used alternate fuels which fuel would you prefer? Bottled Gas Electricity from renewable sources Prefer to use traditional fuels Electricity from grid Kerosene Gas from distribution network 59 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex G: Quantitative Results Charts Chart 1 illustrates quantified responses to qualitative questions asked during key informant interviews (KIIs) with key stakeholders and beneficiaries. The numbers on the chart represent the number of responses given. On average 28 to 29 informants responded to each question. It should be noted that most responders were Task 3 beneficiaries. They typically were more optimistic than key stakeholders (Tasks 1, 2, and 4). Most responders believe CCEB is on track to achieve expected results, that it is having an impact, and is increasing clean energy utilization. The most negative responses with respect to progress were given in response to the policy and regulatory area, where almost one-third of responses were negative. 5 2 0 3 0 5 3 0 0 10 9 13 8 9 12 9 7 14 12 2 0 4 4 7 5 0 2 4 6 8 10 12 14 16 How much impact has CCEB had on the policy and regulatory framework in the country? How much impact has CCEB had on building human capacity in clean energy? How effective has the CCEB program been in achieving outputs or results? Given the progress to date, to what extent is CCEB on track to achieve the expected results? How much impact has CCEB had on increasing clean energy utilization? Chart 1: Progress, Impacts and Outputs Extremely Very Much Somewhat Not Very Much Not at All 60 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Chart 2 illustrates quantified responses to qualitative questions asked during KIIs with key stakeholders and beneficiaries. The numbers on the chart represent the number of responses given. Between 26 and 32 informants responded to each question. It should be noted that most responders were Task 3 beneficiaries. They typically were more optimistic than key stakeholders (Tasks 1, 2, and 4). Most responders believe CCEB has helped build capacity and has helped somewhat in increasing access to financing. The Task 1 and 4 participants were less enthusiastic about CCEB’s financing efforts. 0 0 0 4 3 1 3 0 5 0 11 10 19 4 5 16 14 13 9 13 0 5 0 4 4 0 2 4 6 8 10 12 14 16 18 20 How effective has the program’s technical assistance approach been in institutional strengthening? How effective has the program’s technical assistance approach been in building capacity? How effective has the program’s technical assistance approach been in catalyzing clean energy development? How effective has the program’s technical assistance approach been in increasing access to financing? To what extent has participant training under CCEB been able to build capacity of partner institutions, including capacity of… Chart 2: Effectiveness of Programmatic Approach Extremely Very Much Somewhat Not Very Much Not at All 61 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Chart 3 illustrates quantified responses to qualitative questions asked during KIIs with key stakeholders and beneficiaries. The numbers on the chart represents the number of responses given. Between 26 and 29 informants responded to the four questions. It should be noted most responders were Task 3 beneficiaries. While most responders believe CCEB activities are sustainable, there were concerns expressed, most notably from the industrial company representatives that participated in the FGDs, that the program was not addressing some key constraints to sustainability. These included developing awareness among industry at large and concerns that the work with BERC was not improving the enabling environment. Overall, they were more optimistic than key stakeholders (Tasks 1, 2, and 4) with respect to sustainability. Most responders believe CCEB has helped build capacity and helped somewhat in increasing access to financing. The Task 1 and Task 4 participants were less enthusiastic about CCEB’s financing efforts. 4 2 0 0 0 3 4 2 16 9 7 5 5 14 15 18 1 1 2 1 0 5 10 15 20 Are the constraints to clean energy development you mentioned earlier being addressed by CCEB? How much does CCEB help in addressing the constraints? How sustainable are the outcomes of CCEB? How sustainable and scalable are the activities being undertaken by CCEB? Chart 3: Constraints to Sustainability Extremely Very Much Somewhat Not Very Much Not at All 62 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Chart 4 illustrates quantified responses to qualitative questions asked during KIIs with key stakeholders and beneficiaries. The numbers on the chart represent the number of responses given. Between 21 and 24 informants responded to the two questions. It should be noted most responders were Task 3 beneficiaries. While most responders believe CCEB is integrating gender into its activities, more than 12 percent felt CCEB was not doing enough to promote gender integration. A small minority expressed concern that the program was doing too much to promote gender and that CCEB’s strategy had impacted participation in the program. Overall, most responders were positive about the strategy used. 3 3 0 0 5 3 16 12 0 2 0 2 4 6 8 10 12 14 16 18 1 2 Chart 4: Gender Integration Series5 Series4 Series3 Series2 Series1 63 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex H: Document and Information Sources Name of the documents Type of Documents CCEB USAID in brief materials Activity Gantt Chart CCEB FGD Guide Document CCEB field work schedule doc. CCEB in brief ppt. CCEB questionnaires. CCEB Annual Progress Report Annex 1 doc CCEB Annual Report Year 1 Final Feb 3 2014 CCEB Annual Report Year 2 Final Revised V1 CCEB PMEP Reports CCEB Year 1 PMEP Apr 15 2013. CCEB Year 2 PMEP April 07 2014. CCEB Year 3 PMEP July 29 2015 CCEB Year 3 PMEP November 17 2014 CCEB Year 4 Revised Final PMEP 20151027 CCEB Quarterly Progress Report CCEB 1QFY14 Performance Report Mar 3 CCEB 1QFY15 Performance Report 15 Jan 2014 CCEB 2QFY14 Performance Report Apr 21 CCEB 2QFY15 Performance Report 18 Apr 2015 CCEB 3QFY14 Performance Report June 29 CCEB 3QFY15 Performance Report 15 Jul 2015 CCEB 4QFY14 Performance Report October 15 (Revised October 22) CCEB 4QFY15 Performance Report Revised 05 Nov 2015 CCEB Q1FY13 Performance Report CCEB Q2FY13 Performance Report CCEB Q3FY13 Performance Report Jul 7 CCEB Q4FY13 Performance Report CCEB Work Plan Task 4 work plan Year 3 CCEB Year 3 Task 4 Work Plan 15 01 26 Task 4 Revised Road Map 26Jan2015 Final CCEB Year 3 Work Plan Appendix 1 FINAL CCEB Year 1 Work Plan Final CCEB Year 2 Work Plan Feb 3 2013 CCEB Year 3 Work Plan FINAL Contact list CCEB Counterparts & Stakeholders Contact List CCEB contact list EMMP CCEB EMMP Oct 29, 2013 Grants CCEB Grants Manual Deloitte Grants Manual CCEB Deloitte Grants Manual CCEB Tracker No. 22 64 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Schedule CCEB Counterparts & Stakeholders Contact List CCEB Field Work Schedule Task 1 Task 1 Year 1 BERC Maturity Model Appendix Maturity Model BERC 30Sept Task 1 Year 1 Report Task 1 Year 2 BERC Regulatory Training Proceedings CCEB Task 1 - BERC Institutional Capacity Assessment CCEB Task 1 Year 2 Appendix CCEB Task 1 Year 2 Report (2) Task 1 Year 3 CCEB - Report on BERC Organizational Restructure 2015 - Final v3 Task-2 Task 2 Year 1 Assessment of GOB Capabilities Baseline Assessment Report CCEB Power Sector Screening Tool Energy Policy Analysis Power Sector GHG Emissions Data Repository Power Plant Database Task 2.1 Baseline Assessment Report Final Task 2.2 Year 1 Report Final Task 2.2 Year 1 Report Task 2 Year 2 CCEB Power Sector Screening Tool CCEB-Task 2 Host Organization Selection Report September 17, 2014 Final Data Repository Training Proceedings Jul 23, 2014 GHG Repository Standard Operating Procedures September 17, 2014 power sector ghg emissions data repository Task 2 Year 3 CCEB - Task 2 - Combined Repository and PSPAM OJT Proceedings Final CCEB-Task 2-Capability Assessment Report CCEB-Task 2-Policy Recommendations Paper Final Power Sector GHG Emissions Data Repository - Updated May 2015 Power Sector Policy Analysis Model-V3.1-05.27.15 Proceedings from Briefing on Policy Analysis Tools - Revised Final 2015. 65 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Task-3 Task 3 Year 1 Year 1 Audit Report CCEB Task 3 Final Report Appendix 5.5 CCEB Task 3 Final Report Appendix 5 CEA Training Proceedings Deloitte Grants Manual CCEB 20130305 Tracker No. 22 Task 3 Year-1 Final Report Workshop EE in Tex Plants 30 Sep 2013 Task 3 Year 2 Year 2 Audit Report Year 2 Grant Award 5 textile CCEB Task 3 Frozen Food Workshop Proceedings 22 Sep 2014 CCEB TASK 3 Yr 2 FINAL REPORT Latest Final-Steel Re-rolling Workshop Proceedings Final-WS Proceedings 05 Financing Facilitation May 2014 Final-WS Proceedings Financing Facilitation 05 May 2014 Final-WS Proceedings Steel Re-rolling 21 Jul 2014 Grant Selection Criteria Task 3 Year 3 Energy Audit Reports Yr 3 CCEB - Task 3 - 20 Investment Grade Energy Audits CCEB - Task 3 - 30 Walk-Through Audits CCEB - Task 3 - EE Financial Institution Training Proceedings-Final CCEB-Task 3.1.C-Proceedings on Five Workshops on Best Practices in IEE CEA Training Proceedings Final Proceedings from CCEB Dec 2014 CEA Training Task-4 Task 4 Year 1 Load Research and Analysis DESCO Nov 21 Load Research and Analysis DPDC Load System Selection Report DESCO 04 11 2013 Load System Selection Report DPDC Nov 11 Task 4 DSM Program Report 9 30 2013 Task 4 Roadmap Nov 25 Task 4 Year 3 CCEB Year 3 Task 4 Work Plan 15 01 26 Task 4 Revised Road Map 26 Jan2015 Final Task 5 Year 1 Year-1 Final Report 66 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Task-5 Task 5 Year 2 Meeting Minutes Year 2 Deliverable Assessment Report for CDM Financing Business Model Workshop Report Clean Cook Stove TVC.VOB Establishing Linkages Between Entrepreneurs and MFIs - Workshop Report - 5 March ICS Workshop Report Report on Testing Center Support Revised Market Segmentation Summary Report Second ICS Market Facilitation Platform Meeting Proceedings Final Stakeholder Mapping Report - Final Task 5 Year 3 CCEB-Task 5 3 B - ICS Financing Efforts and Successes Revised Final CCEB-Task 5- CSR Opportunities Assessment Report Task 5- Five Signed LOCs CCEB Evaluation draft work plan v8 11-19-2015-USAID comments CCEB Contract SOW CCEB Mod 3 Pg Concerning Task 1 CCEB Section C Private Sector Power Generator Policy of Bangladesh Policies Renewable Energy Policy of Bangladesh 3- Year Road Map (2008-2010) final Seven Five Year Plan FY2016- FY2020 Accelerating Growth, Empowering Citizens Action Plan For Energy Efficiency & Conservation Country Action Plan for Clean Cookstoves November 2013 Bangladesh Country Action Plan (CAP) - Draft Bangladesh Energy & Electricity Research Council Act, 2015 Bangladesh Energy Regulatory Commission Act. 2003 Bangladesh Climate change Strategy and Action Plan 2009 Bangladesh Policy Road Map for Renewable Energy Draft Renewable Energy Policy of Bangladesh ElecAct-Version with comments on 12.5.15 National Adaptation Programme of Action (NAPA)- November 2005 National Energy Policy 1995 Power & Energy Fast Supply (Amendment) Act, 2015 Power & Energy Fast Supply Enhancement (Special Provision) Act, 2010 Strategic Program for Climate Resilient Bangladesh Proposed Energy Policy by M. Delwar Hossain - SEP 2004 Rural Electrification Board Ordinance, 1977 SREDA Act Law Vetting 67 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Prices Bulk tariff BPDB Retail tariff BPDB distribution zones Retail tariff DESKO Retail tariff DPDC Retail tariff REB Retail tariff WZPDCL Annual Report 2011-2013 (BERC) Public Financial Management Risk Assessment Framework (PFMRAF) Manual, USAID LCG Env. & CC-Overview Clean Energy Lending Toolkit The AILEG Project Improved Capacity for Energy Access (ICEA) final report 2012 BERC IRG Approved Organogram of Bangladesh Energy Regulatory Commission ELIB Presentation Bangladesh CFL program, World Bank NARUC report - BERC support 2009 Powering Progress Project - Final Report (Feb-2013) Power Sector Organogram Power Sector overview USAID BERC support Regulatory Audit of Tongi Power Generation Plant, Bangladesh Energy Audit Report (Oct- 2012) 68 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex I: Evaluation Methodology – Risk Assessment Framework (RAF) The CCEB program activities are being undertaken simultaneously across the clean energy value chain and so it is important to consider the interdependences and linkages between the different activities to appropriately assess the extent that the program is on track to meet the overall development objective. As such, the team collected both quantitative and qualitative information to inform the evaluation findings and then used a Risk Assessment Framework (RAF) to assess the overall progress. Decision makers typically want to avoid initiatives and interventions when the threats of failure are highly probable with catastrophic consequences or when initiatives are not cost effective. The goal of a risk assessment is to use data to analyze the risks and rewards of a decision or action, thereby reducing the need for intuition and instinct. Risk assessment does not guarantee future success. It is a measurement of the risks associated with a particular intervention decision and the implementation approach used to achieve a given development objective. Identifying a risk is only a first step. It is imperative to have concrete actions to mitigate any such identified risk. The mitigation options for risks associated with CCEB are included in section 5. CCEB Risk Assessment Framework The RAF used for assessing the risk levels for CCEB activities and outcomes considers the probability and impact (severity) of an occurrence. There are four impact levels used: negligible, marginal, serious and catastrophic. With probability, there are also four levels used: remote, occasional, probable, and frequent. Associated definitions are presented in the following tables. Table H.1: Probability Rating Definitions Definitions for Impacts and related criteria definitions are presented in the table that follows. Probability Quantitative Qualitative Remote Less than a 0.25 probability. An adverse event is rare or would only occur in exceptional circumstances. There is little or no experience of a similar failure. Occasional Probability lies between 0.26 and 0 .50. An adverse event might occur because the conditions for it exist, but controls exist and are effective. Probable Probability lies between 0.51 and 0.75. An adverse event will likely occur because the controls are inadequate or are applied inconsistently. Frequent Probability lies between 0.76 and 0.99. An adverse event is expected to occur. There is near certainty of occurrence because the controls do not exist or are ineffective. 69 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table H.2:Impacts, and Associated Criteria Definitions The Risk Assessment Framework matrix used for CCEB is shown below. Impact International Practice Development Objective Commitment Accountability Catastrophic There are obvious and material divergences from good international practice. Realization of an adverse event associated with this risk factor would permit attainment of less than 40 percent of expect program outcomes. Expected effects include failure of the program widespread and severe dissatisfaction by stakeholders. Political and management commitment to attainment of good international practice is the exception or entirely absent. Vertical and horizontal responsible institutions have major gaps, or one or the other is severely under￾developed. Opposition is organized or widespread and therefore expected. Serious Significant elements do not reflect good international practice. Attainment of 40 to 70 percent of the expected outcomes associated with the development objective can reasonably be expected. Expected effects could include a major delay, limited dissatisfaction by stakeholders. Political or management commitment to attaining good international practice is inconsistent or questionable. Weaknesses in the horizontal and vertical responsible institutions are evident or one or the other shows significant gaps. Opposition is evident by some elements within the society. Marginal Work broadly reflects good international practice with some gaps or inefficiency present. Seventy to 95 percent of the development objective can be reasonably assumed to be attained. Expected effects could include minor delays in attainment, minor dissatisfaction by stakeholders, or a non-material financial impact. Political or management commitment to closing the gaps and eliminating inefficiencies is present. Weaknesses in the horizontal and vertical responsible institutions may be present or such institutions may be in an early and untested stage of development. Opposition to these institutions not a “given” Negligible Work reflects good international practice. The development objective, in the 95 to 100 percent range of expected program outcomes, can reasonably be assumed to be attained if conditions do not change. Strong political and management commitment to sound best practice is evident. Both horizontal and vertical responsible institutions are mature, function routinely, and are not under threat. Table H.3: CCEB Risk Assessment Framework Matrix The following table provides a summary of risk mitigation considerations options for each of the four impact classifications used in developing the CCEB RAF. CCEB Risk Assessment Framework Matrix Impact Catastrophic High Critical Critical Critical Serious High High Critical Critical Marginal Medium Medium High High Negligible Low Low Medium Medium Remote Occasional Probable Frequent Probability 70 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table H.4: CCEB Risk Mitigation Considerations Impact Classification Risk Mitigation Consideration Critical “Critical” requires stringent mitigating measures only if these have a high probability of success. Otherwise, we will terminate our exposure by delivering the assistance through other means. In rare cases where an effective transfer of risk mechanism exists and is deemed effective, we will consider transfer of the risk, albeit with a risk assessment of the ability of the transferor to deliver on its obligation. High "High” requires serious mitigating measures to treat the risk to avoid possible catastrophic and other major failures. Medium “Medium “requires mitigating measures but these may be periodic. Low “Low” requires monitoring and audit, but treatment of specific risks may be required if they can lead to Medium risk conditions. 71 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 72 Annex J: Interview and Contact Listing Organization Address Name Position IBTCI ACME Hs# 01, Rd# 54/A, Gulshan - 2, Dhaka John Dalton Chief of Party ACME Hs# 01, Rd# 54/A, Gulshan - 2, Dhaka Naba Krishna Muni Capacity Building Specialist ACME Hs# 01, Rd# 54/A, Gulshan - 2, Dhaka Nazrul Islam M&E Specialist Performance Monitoring ACME Hs# 01, Rd# 54/A, Gulshan - 2, Dhaka Farhana Rahman Financial Analyst Payroll ACME Hs# 01, Rd# 54/A, Gulshan - 2, Dhaka Ninisha Maksud Knowledge Management coordinator ACME Hs# 01, Rd# 54/A, Gulshan - 2, Dhaka Tasmin Akanda Ethu CCEB Evaluation Logistic Coordinator Implementing Partner Deloitte House no. 14 (2nd floor), Road no. 32 Gulshan-1. Craig Van Develde Chief of Party Deloitte House no. 14 (2nd floor), Road no. 32 Gulshan-1. Engr. S.M. Jakaria Senior Energy policy Advisor HBS Deloitte House no. 14 (2nd floor), Road no. 32 Gulshan-1. Mohammad Abdul Jalil Senior Regulatory Advisor Deloitte House no. 14 (2nd floor), Road no. 32 Gulshan-1. A.K.M. Anowar Hossain Mollah Deputy Task Leader Deloitte House no. 14 (2nd floor), Road no. 32 Gulshan-1. Engr. S.M. Mahmud Hassan Senior Advisor, Industrial Energy Efficiency Deloitte House no. 14 (2nd floor), Road no. 32 Gulshan-1. Salman Kamal Industrial Energy Efficiency Specialist Deloitte House no. 14 (2nd floor), Road no. 32 Gulshan-1. Biplob Kanti Mondal Monitoring & Evaluation Specialist Deloitte House no. 14 (2nd floor), Road no. 32 Gulshan-1. Md. Kamruzzaman Senior Industrial Energy Efficiency Specialist Deloitte House no. 14 (2nd floor), Road no. 32 Gulshan-1. Ruaksana Haque Communication & Training Specialist USAID USAID American Embassy Madani Avenue, Baridhara, Dhaka Karl Wurster Deputy Director-Economic Growth Environment USAID American Embassy Madani Avenue, Baridhara, Dhaka Jeff de Graffenried Project Development office (Program office) USAID American Embassy Madani Avenue, Baridhara, Dhaka A.K.D. Sher Mohammad Khan (EG office) Senior Energy Advisor & CCO AARI/EI (EG office) CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 73 USAID American Embassy Madani Avenue, Baridhara, Dhaka Shayan Shafi Project Management Specialist (Energy) (EG office) USAID American Embassy Madani Avenue, Baridhara, Dhaka Farzana Yasmeen Program Management specialist (EG office) Task- 1 BERC TCB Bhaban (3rd floor), 1 Karwan Bazar, Dhaka A R Khan Chairman BERC TCB Bhaban (3rd floor), 1 Karwan Bazar, Dhaka Md. Aminur Rahman Deputy Director (Gas) BERC TCB Bhaban (3rd floor), 1 Karwan Bazar, Dhaka Nishit Kumer Assistant Director BERC TCB Bhaban (Level 4), 1 Karwan Bazar, Dhaka Rahman Murshed Member BERC TCB Bhaban (Level 4), 1 Karwan Bazar, Dhaka Dr. Salim Mahmud Member (Commissioner) BERC TCB Bhaban (Level 4), 1 Karwan Bazar, Dhaka Md. Haronur Rashid Deputy Director (power) Task- 2 DPC Group House 17, Road 17/A, Block-E, (4th Floor), Banani, Dhaka MD. Mashiur Rahman Managing Director DPC Group House 17, Road 17/A, Block-E, (4th Floor), Banani, Dhaka Ansar A Mullaik (Raj) Manager (Business Development) BPDB Room # 503, 5th Floor, WAPDA Building, Motijheel C/A, Dhaka Md. Monower Zahid Khan Assistant Engineer BPDB Room # 503, 5th Floor, WAPDA Building, Motijheel C/A, Dhaka Jarifa Khatun Executive Engineer ( System Planning) BPDB Room # 503, 5th Floor, WAPDA Building, Motijheel C/A, Dhaka Roton Kumer Paul Director (System Planning) Power Cell Ministry of Power, Energy & Mineral Resources Bidyut Bhaban (9th Floor) 1, Abdul Gani Road Dhaka Md. Abdur Rouf Miah Director (Sustainable Energy) Power Cell Ministry of Power, Energy & Mineral Resources Bidyut Bhaban (9th Floor) 1, Abdul Gani Road Dhaka Abdullah-Al-Mohit Assistant Director (Contract Management) GIZ Road 90, House 10/A, Gulshan 2 Dhaka 1212, Bangladesh. Al Mudabbir Bin Anam Senior Advisor Sustainable Energy for development (SED) Task- 3 BD Technology Ltd. House no 02 (2nd floor) 15, New Baily Road, Dhaka Engr. Mohammad Abdullah Chief Advisor B.Sc. Eng. (Elect) BUET, FIEB (Former Chief Engineer BPDB & Former Director (Tech) CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 74 WZPDCL) (A Govt. owned company) BD Technology Ltd. House no 02 (2nd floor) 15, New Baily Road, Dhaka Engr. Humayun Kabir Director, Certified Energy Auditor (AEE, USA) BD Technology Ltd. House no 02 (2nd floor) 15, New Baily Road, Dhaka Engr. Niladri Shekhar Saha Lead Assessor & Project Coordinator BD Technology Ltd. House no 02 (2nd floor) 15, New Baily Road, Dhaka Engr. Nasim Saba Shommo Energy Efficiency Auditor Wellmake 12/B, Purana Paltan Line (2nd Floor), Dhaka Md. Abdul Alim Consultant Engineer Wellmake 12/B, Purana Paltan Line (2nd Floor), Dhaka Engr. A.F.M Muntasir Safayat Deputy Manager (Engineering Division) Wellmake 12/B, Purana Paltan Line (2nd Floor), Dhaka Engr. MD. Riazul Islam Asst. Manager (Project Division) Sodev Consultant International Hs# 198, Rd# 01, New DOHS Mohakhali, Dhaka Fazlul Q. Siddique Managing Director Sodev Consultant International Hs# 198, Rd# 01, New DOHS Mohakhali, Dhaka Zakirul Q. Siddique Director Sodev Consultant International Hs# 198, Rd# 01, New DOHS Mohakhali, Dhaka A.K.M. Mazharul Islam Senior Engineer Sodev Consultant International Hs# 198, Rd# 01, New DOHS Mohakhali, Dhaka Kowshic Ahmed Akash Junior Engineer FGD Groups IDLC Corporate Head office: Bay’s Galleria (1st floor), 57 Gulshan Avenue, Gulshan-1, Dhaka 1212 Md. Shahriar Rahman Executive Officer Green Banking desk, Corporate division. IDLC Corporate Head office: Bay’s Galleria (1st floor), 57 Gulshan Avenue, Gulshan-1, Dhaka 1212 Md. Mahbubur Rahman Manger Green Banking desk, Corporate division. Southeast Bank Limited Kawran Bazar Branch Jamuna Bhaban (1st floor) 2, Kawran Bazar C/A, Dhaka -1215. Md. Salimuzzaman Vice President & Manager Operation Southeast Bank Limited Kawran Bazar Branch Jamuna Bhaban (1st floor) 2, Kawran Bazar C/A, Dhaka -1215. Md. Mostafa Kamal Executive Vice President and Head of Branch City Bank 136, Gulshan Avenue Gulshan-2, Dhaka￾1212 Zoheb Ahmed Associate manager, Structured Finance Corporate Banking Division City Bank 136, Gulshan Avenue Gulshan-2, Dhaka￾1212 Md. Mominul Islam Senior Manager Project Assessment Unit CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 75 Mutual Trust Bank Limited Mutual Trust Bank Limited Corporate Head office MTB Square (Level-2) 210/A/1, Tejgaon Industrial Area, Tejgaon, Dhaka- 1208 Mohammad Mamunur Rahman Junior Officer & Credit Analyst SME Banking Division KNIT CONCERN GROUP 62, Water Works Road, Godnail Narayangonj Md Arif Hossain Bhyain Deputy Manager – ES & QA (Chemocal) KNIT CONCERN LTD. KC APPARELS LTD 62, Water Works Road, Godnail Narayangonj MD. Imran Sarker Asst. Manager (Gen.) Dyeing Unit. SADAT JUTE INDUSTRIES LTD, JANATA JUTE MILLS LTD. 7 Gulshan Avenue (7th floor), Gulshan-1, Dhaka Mahmudul Huq Managing Director of SADAT JUTE INDUSTRIES LTD Dy, Managing Director of JANATA JUTE MILLS LTD. Asian Group of Industries 28, Dilkusha C/A, (4th floor), Suite-404, Motijheel, Dhaka - 1000 Md. Shafiqur Rahman Executive Director Crony Group Corporate office: House# 365/4, Road# 06 (West) Baridhara DOHS Dhaka Md. Jahangir Alam Deputy General Manager The Cloth & Fashion LTD Off & Fact: Madani Super market (2nd-4th) Floor, Hemayetpur, Savar, Dhaka, Bangladesh. Md. Mahbub Rahman Compliance Manager AZALEA GROUP 47/2, Arambagh, Motijheel, Dhaka- 1000, Bangladesh. Shabbir m Ashadul Islam Managing Director KULIARCHAR GROUP Head office: Ideal Trade Centre (8th floor), 102 Shahid Tajuddin Ahmed Sarani, Tejgaon, Dhaka-1208, Bangladesh. Anjan Kumar Saha General Manager (Seafood division) Certified Energy Auditor H S International Corporate Office: Level- 3, House-567, Road- 9 (old link road) Mirpur DOHS, Dhaka- 1215 Farhana Akhter Chowdhury Manager, Operation SME Foundation Royel Tower, 4, Panthapath, Dhaka- 1215, Bangladesh Muhammad Khaleduzzaman Talukder Program Officer (Technology) RAHIMAFROOZ 260/B, 5th floor, Tejgaon Industrial area, Dhaka Md. Shahadat Hossain Executive, Q A & QMS Task- 4 DESCO House- 22/B, Farrukh Sarani, Nikunja- 2, Dhaka-1229, Bangladesh Monjurul Haque Superintendent Engineer DESCO House- 22/B, Farrukh Sarani, Nikunja- 2, Dhaka-1229, Bangladesh Engr. Sazzad Nazmul Alam Miah Sub- Divisional Engineer MIS, ICT CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 76 DESCO House- 22/B, Farrukh Sarani, Nikunja- 2, Dhaka-1229, Bangladesh Engr. Md. Ashfaq Ahmed Executive Engineer DPDC Biddyut Bhaban (2nd floor), Dhaka Abul Kalam Azad Executive Engineer Task- 5 Eco Stories 126/3, Monipuri para, Airport Rd Tejgaon, Dhaka- 1215, Bangladesh Maqbool ul Hossain Managing Director Future carbon 5th Floor, House- 42, Road- 1, Block- A, Niketon, Gulshan- 1, Dhaka- 1212 Mosharraf Hossain Officer, Projects and Operations. Future carbon 5th Floor, House- 42, Road- 1, Block- A, Niketon, Gulshan- 1, Dhaka- 1212 Raden Siddiqui Director of Operations, Strategy and Implementation CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex K: Final Report on ICS Survey Prepared for: Accelerating Capacity for Monitoring and Evaluation (ACME) & The United States Agency for International Development (USAID) House No. 1, Road 54/A, Gulshan-2, Dhaka, Bangladesh Prepared by: DPC Group of Consultants House # 17 (4th Floor, Apt-E-1), Road # 17/A, Block- E, Banani, Dhaka-1213, Bangladesh Tel.: +8802-9820733 E-mail: dpcgroup@gmail.com Website: www.dpcgroupbd.org Mid-term Performance Evaluation of Catalyzing Clean Energy in Bangladesh Final Report of Improved Cook Stoves End User Survey January 2016 77 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT ABBREVIATIONS ACME Accelerating Capacity for Monitoring and Evaluation CCEB Catalyzing Clean Energy in Bangladesh DOE Department of Environment EQS Environmental Quality Standards GOBGOB Government of Bangladesh IBTCI International Business & Technical Consultants, Inc. ICS Improved Cook Stoves MOEF Ministry of Environment and Forestry NGO Non-Governmental Organizations O&M Operation and Maintenance SDG Sustainable Development Goal ToR Terms of Reference USAID United States Agency for International Development 78 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT TABLE OF CONTENTS ACRONYMS............................................................................................................................................................... iv EXECUTIVE SUMMARY.......................................................................................................................................... vi Purpose, Objectives and Evaluation Questions............................................................................................. vi Methodology and Conceptual Approach ........................................................................................................ vi Reponses to the Evaluation Questions........................................................................................................... vii Evaluation Question 1: To what extent is the CCEB project on track, in terms of progress and outcomes, to meet its overall goals for the five tasks under Components A and B?......................... vii Evaluation Question 2: What are the opportunities to enhance programmatic approach and effectiveness plausible under the stipulation of the contract?................................................................... vii Evaluation Question 3: What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability?....... ix Evaluation Question 4: How effectively has gender been integrated or incorporated in the interventions? What have been the challenges and opportunities, if any?............................................... x Lessons Learned..................................................................................................................................................... x 1. INTRODUCTION.............................................................................................................................................. 1 2. PROJECT BACKGROUND: THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE.......... 2 3. PURPOSE OF THE EVALUATION................................................................................................................ 3 3.1. Intended Audience ................................................................................................................................... 3 3.2. Evaluation Questions............................................................................................................................... 3 4. EVALUATION METHODOLOGY AND LIMITATIONS......................................................................... 4 4.1 Planning and Document Review ...................................................................................................................... 4 4.2 Data Collection and Analysis............................................................................................................................ 4 4.3 Biases and Limitations........................................................................................................................................ 4 5. FINDINGS AND CONCLUSIONS................................................................................................................ 5 5.1 Findings - CCEB Overall Performance....................................................................................................... 5 CCEB Risk Assessment Framework (RAF) and Risk Mitigation Options................................................ 6 CCEB Risk Mitigation Assessment........................................................Error! Bookmark not defined. 5.2 Evaluation Question 1.................................................................................................................................. 13 5.2 Evaluation Question 2........................................................................................................................... 15 5.3 Evaluation Question 3........................................................................................................................... 17 5.4 Findings – Overall................................................................................................................................... 19 5.5 Findings – Task 1: Improve Regulatory Environment for Clean Energy Development......... 19 5.6 Findings – Task 2: Strengthen Analytical Capacity for Energy Sector Planning and Policymaking .......................................................................................................................................................... 20 5.7 Findings – Task 3: Promote Industrial Energy Efficiency and Conservation............................. 20 5.8 Findings – Task 4: Adopt Demand Side Management Programs for Electric Utilities........... 21 5.9 Findings Task 5: Conduct Market Analysis and Development for Improved Cook Stoves........ 21 Findings – CCEB support to market development ................................................................................. 21 79 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Findings – ICS Manufacturers....................................................................................................................... 21 Findings – ICS Suppliers and Distributors................................................................................................. 22 Findings – ICS End-Users .............................................................................................................................. 22 5.10 Conclusions – Overall........................................................................................................................... 23 6. RECOMMENDATIONS.................................................................................................................................. 25 6.1 Overall Recommendations................................................................................................................... 25 6.2 Task 1 Recommendations.................................................................................................................... 26 6.3 Task 2 Recommendations.................................................................................................................... 27 6.4 Task 3 Recommendations........................................................................................................................... 27 6.5 Task 4 Recommendations........................................................................................................................... 28 6.6 Task 5 Recommendations........................................................................................................................... 28 7. LESSONS LEARNED........................................................................................................................................ 30 7.1 Effectiveness: ............................................................................................................................................ 30 7.2 Sustainability: ........................................................................................................................................... 30 7.3 Program Improvement:................................................................................................................................ 30 Annex A: Gantt Chart of Activities...................................................................................................................... 31 Annex B: Evaluation Design Matrix...................................................................................................................... 36 Annex C: Data Collection Schedule and Data Analysis Methodologies...................................................... 38 Annex D: Key Informant Interview Questions.................................................................................................. 41 Annex E: Focus Group Discussion Guides......................................................................................................... 48 Annex F: ICS Survey Instruments......................................................................................................................... 50 ICS Survey Instrument – Manufacturers ............................................................................................................. 51 ICS Survey Instrument – Suppliers/NGO Community.................................................................................... 53 ICS Survey Instrument – Other Stakeholders................................................................................................... 55 ICS Survey Instrument – End Users..................................................................................................................... 57 Annex G: Quantitative Results Charts................................................................................................................ 60 Annex H: Document and Information Sources................................................................................................. 64 Annex I: Evaluation Methodology – Risk Assessment Framework (RAF) .................................................. 69 CCEB Risk Assessment Framework................................................................................................................ 69 Annex J: Interview and Contact Listing............................................................................................................... 72 Annex K: Final Report on ICS Survey ................................................................................................................. 77 ABBREVIATIONS..................................................................................................................................................... 78 TABLE OF CONTENTS......................................................................................................................................... 79 Preface......................................................................................................................................................................... 82 Executive Summary .................................................................................................................................................. 83 1. Introduction ........................................................................................................................................................ 86 2. Survey Methodology......................................................................................................................................... 86 2.1 Sampling.................................................................................................................................................... 86 2.2 Training for Enumerators by DPC and ACME................................................................................ 87 80 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 2.3 Survey Instruments................................................................................................................................ 88 2.4 Survey Staffing Plan ................................................................................................................................ 88 2.5 Program used for data entry operation and analysis..................................................................... 88 3. Constraints and Limitations of the survey .................................................................................................. 88 4. Findings of the survey....................................................................................................................................... 88 4.1 Specific findings based on data analysis............................................................................................. 88 5. Recommendations............................................................................................................................................. 90 6. Lessons Learned ................................................................................................................................................ 91 7. Conclusions ........................................................................................................................................................ 91 References Tables..................................................................................................................................................... 93 List of Annexures:.................................................................................................................................................. 106 Annex L: Mid-Term Evaluation SOW............................................................................................................... 107 A. Photos of different design of ICS and Users B. Terms of Reference (ToR) of Survey C. Original Survey Instruments (English and Bengali version) D. List of Interviewed Respondents E. Survey Questionnaires (134) 81 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Preface From the ancient period, energy is an inevitable necessity for human beings and the history of human development rests on the availability and use of energy, the transformation from the early use of fire and animal power that improved lives, to the present world with use of electricity and clean fuels for a multitude of purposes. The use of coal in the mid-1700s and the development of steam engines set off fast growth of cities, population, and further inventions, including internal-combustion engines and the discovery and use of oil, natural gas, and electricity. This accelerated growth period, known as the Industrial Revolution, matured by the end of the 19th century with significant use of fossil fuels and further electrification, and resulted in almost-exponential growth of population and energy use. All over the world, energy is the neglected issue of the development debate, and the lack of access to reliable and clean energy supplies is a major barrier to improving human well-being around the globe. There are an estimated 1.6 billion people living in the rural areas of developing countries who lack access to electricity and so dependence on fossil fuels while burning of fossil fuels produces large amounts of CO2, an important greenhouse gas. Concurrently the energy source is predicted to be scarce in the coming years as a result of population growth. Therefore, people prefer energy sources that are renewable, clean and cost effective. Energy generation and use are strongly linked to all elements of sustainable development; for instance; economic, social, and environmental including poverty and global warming. Many research studies create a compelling case for global action despite the fact that household air pollution has been linked to more than 4 million premature deaths annually. The traditional open fires and inefficient cook stoves using solid fuels is the source of a range of harmful impacts that hamper economic and social development and lead to significant loss of life in the developing world. Reality is that three billion people across the developing world cook their food each day over an open flame or on a crude stove using solid fuels like wood, coal, crop residues, and animal dung. In sub-Saharan Africa and Asia, the lack of access to clean cook stoves and fuels for cooking and heating is especially acute, with a third of the urban population and the vast majority of the rural poor using solid fuels to cook their daily meals over open fires or inefficient stoves made from clay, metal, or bricks. In many countries, the rate of solid fuel usage, especially in rural areas, is 80% to 90%, and without interventions to promote the adoption of clean cooking solution. Air pollutants are really a hazardous problem in Bangladesh. Clean cook stoves and fuels have the potential to reduce deaths from smoke-related illnesses, mitigate climate change, and lower air pollution. They can provide new sources of livelihoods for women while reducing the risk and hard work of fuel collection, and can lower household expenditures on cooking fuel. The Catalyzing Clean Energy in Bangladesh (CCEB) program is a 5-year program (2012-2017) to support and enhance energy security, economic growth and climate change mitigation in Bangladesh. One of the key components of this program is the “Market Analysis and Development for Improved Cook Stoves (ICS),” which focuses on promoting new technology throughout Bangladesh. The DPC Group of Consultants was contracted to conduct a survey as part of the Mid-Term Performance Evaluation of CCEB by the Accelerating Capacity for Monitoring and Evaluation (ACME) project. 82 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Executive Summary A Mid-Term Performance Evaluation of the Catalyzing Clean Energy in Bangladesh (CCEB) Program has been planned and conducted to assess any constraints and opportunities associated with the achievement of expected program results through the ICS End User Survey. This has been done to provide to specific recommendations for opportunities and constraints to enhance programmatic effectiveness and impact and strengthen the current approach and to determine the extent to which the CCEB activities to support market development are likely to be sustained. The initial plan was to select the respondents randomly from the over 22,000 cook stove users from various districts of the country. However, details information was not available with the addresses of cook stoves users in due time and seemed that would be difficult to have the complete list and duration of survey was shorter so an alternative approach has to be adopted. Finally, the survey was conducted for 134 respondents at end-users level in three selected districts named Bagerhat, Chapai Nawabgonj and Gaibandha those have been selected with different characteristics of importance as far as user of ICS conditions are concerned. The trained enumerators have done one to one interview with a prescribed questionnaire and documented the answers as per given format. The data has been entered in SPSS program and analyzed in lieu with the requirement of the client and accordingly this report has been prepared. This has provided an extensive and variety of opportunities to analyze the present program and accommodate all findings, recommendations and lesson learned for improvement. This program has some limitations, among them marketing and construction quality as well as inadequate design and high price of cook stoves mentioned repeatedly. Some of the designs require a special type of fuel which is not available or it takes more time to shape it, which may also explain why some are unenthusiastic to use this ICS for all cooking (Figure 1) and affect the number of times (Figure 2) they use it daily although they purchased it. On the other hand, refueling is also a problem in many cases while almost at the same time (Figure 3) as traditional cooker is needed for cooking and contributes excess smoke. This is found to be not completely environment friendly and in some cases, neither are there cost savings. All these dynamic reasons influence negatively. Figure 1: Use of ICS for all Cooking Figure 2: Daily Use of ICS 83 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Figure 3: Usual Time Needed for Cooking with ICS As mentioned earlier, many use ICS as an optional item while they use other traditional cookers as their main support (Figure 4) and major numbers have to buy fuel for ICS (Figure 5) which is not available in some cases due to the specific configuration of fuel type, materials, or size. Therefore, the low-income community who really needs this support cannot it afford at this stage. The urban poor and semi-urban poor who prefer to have piped gas for cooking but it is not available will become a large number of users if there are some measures taken considering their ability and willingness. It is recommended to either decrease the cost of ICS, subsidize, or introduce installment of selling with moveable and modified design (less and available fuel, and shorter time for cooking) to reach the poor and hard-core poor of the community. Although there are some limitations in construction levels and fuel selection levels reported by the users of ICS, they would want to use it with some modifications and having fuel easily available in their locality. This would have a major impact at the end user level if they get it with lower price and/or subsidy or installment facility. This is mostly used in the medium and lower-medium community and semi-urban areas. Therefore, there is much room for improvement, which all relevant agencies need to discuss and find out the shortcomings within the existing system. It may be appreciable to build up a network between producers, distributors and local NGOs who can assist for subsidy or loan for installments for the poor section of the community. At the same time, more production and highest selling will help to reduce ICS price and increase the accessibility of various income level people. The program also needs to extend all over the country through local contributors, which will help for reducing the effects of climate resilience in an extensive manner with the consideration of users suggestions (Figure-6). Figure 4: % of Daily Cooking done with ICS Figure 5: Buy Fuel for Cooking with ICS 84 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Figure 6: Suggestions from ICS Users (multiple responses considered) 85 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 1. Introduction The goal of Catalyzing Clean Energy in Bangladesh (CCEB) is to promote clean energy development. One of the prime tasks being undertaken by the CCEB program is to assist in building sustainable, improved cook stoves (ICS) market in Bangladesh in order to reduce energy consumption and greenhouse gas emissions. Market development can be achieved by developing sources of finance for consumers and/or clean energy businesses engaged in supplying the market with improved biomass cook stoves; strengthening businesses involved in the cook stoves supply chain; promoting the benefits of ICS to consumers and by better understanding of consumer behavior to generate market demand. Since program inception, the program has worked with and directly supported many of the key actors in the ICS supply chain. Over 22,000 improved cook stoves have been put into service due efforts directly attributable to the CCEB program. The results to date are impressive and it is anticipated that a substantial number of additional improved cook stoves will be deployed by the end of and as a result of the CCEB program interventions. As per requirement, a Mid-Term Performance Evaluation of CCEB was planned while the DPC Group of Consultants was contracted to conduct a survey of beneficiaries who have received support attributable to the CCEB program. As per the ToR, the purpose of the evaluation is to: 1) assess any constraints and opportunities associated with achievement of expected program results; 2) provide specific recommendations for opportunities to enhance programmatic effectiveness and impact and strengthen the current approach; and, 3) determine the extent to which the CCEB activities to support market development are likely to be sustained. The findings and recommendations of the evaluation will be used to improve implementation of the program and inform the design of other relevant DO4 projects. 2. Survey Methodology To obtain consistent and reliable data, the survey had covered 134 respondents who were chosen from the three selected districts namely Bagerhat, Chapai Nwabgonj and Gaibandha as per list provided by the client. 2.1 Sampling There are four (4) categories of stakeholders on whom it was planned to explore the effectiveness and sustainability of CCEB Cook Stove project. These four categories are: i) Manufacturers; ii) Suppliers/NGO Communities; iii) Other Stakeholders; and iv) End-Users; There are over 22,000 users of ICS in different districts of Bangladesh. At first, it was planned to select the respondents randomly from all 22,000 cook stove users. Due to lack of detailed information and not receiving the list of total cook stoves users within given time from suppliers (and getting the impression that it would be difficult to have the complete list), an alternative approach had to be adopted. The required data was collected from the end-users list that was provided by the local suppliers and partner NGOs. The survey was conducted in three selected districts and a total numbers of sampling were 134 respondents at end-users level; those are 48 from Bagerhat, 47 from Chapai Nawabgonj and 39 from Gaibandha districts selected with different characteristics of importance as far as user of ICS conditions are concerned. 86 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Additionally, all five (5) manufacturers and/or their representatives were also interviewed and held informal discussions to explore the benefits and limitations of ICS market promotion and acceptability. 2.2 Training for Enumerators by DPC and ACME A training course was organized for enumerators/data collectors and supervisors at DPC office as well as ACME office. The discussion points were as follows:  Focus on briefing everyone on the project and the questionnaire on the social aspects. o Briefing of project o Purpose of this survey  Get view points from the people of the various city/locations  Socio-economic situation of the households  Willingness/ability to pay for ICS o The issues supposed to be covered by this survey  Discussion on questionnaire o The questionnaire is the most important tool of the survey and it is meaningful to explore the questionnaire from the beginning to make sure all survey related staff understand, specifically on the technical matters which have been incorporated in several questions; o Each of the survey team members was given a copy of the questionnaire and allocated some time to read it and ask any questions if not understandable to them; o They were prepared and asked lots of questions, which the Survey Specialist and other senior staff clarified until they became satisfied, as they are the key persons to collect authentic data from the field and have good conversation with respondents while interviewing them.  Criteria of success for a survey o The enumerators were experienced, so they already knew the importance of these matters, rights and confidentiality and role of them as data collector. We did not need to go into much detail, but we shortly reminded them of these aspects. o Consistency in the collection of data  The phrasing of the questions supposed to be clear to each enumerator and not to interpret differently.  The words of the questions and attitude of the enumerators must be the same. o Important that they feel comfortable in the interview situation and answers as honest as possible. (e.g. not be intimidated by the question) o The respondents should be able to answer correctly (e.g. memory)  Rights of participants/respondents and confidentiality o They have a right not to participate or not to answer any question they do not want to. o They have a right to be anonymous, and it is important that they trust that their anonymity will not be compromised.  The role of the data collector o The data collectors understood that they are playing the most important to the results of this survey; o It was also important to make them understand the questions, terms, and topics in the same way; o It is important to let the rightful respondents answer the questions  According to the sample  Not to fill out the questionnaire by yourself/enumerators 87 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT 2.3 Survey Instruments The basic survey instrument is the questionnaire, which was originally developed in English to exchange views between the stakeholders and was finalized based on the comments from various sources including USAID, ACME and DPC team members. Afterwards the questionnaires were translated from English to Bengali and used for Field Test of Questionnaires by the enumerators that helped them to be acquainted as well as conceptualize the issues; questions were adjusted accordingly in the final version of questionnaire in both languages. 2.4 Survey Staffing Plan A total of nine enumerators were recruited, trained and engaged in the process of data collection in three selected districts and have been supervised by the senior professionals for ensuring the quality of the field level survey. After completion of the field survey, collected data was entered by the four (4) Data Entry Operators, and the statistician checked and corrected wherever needed. The Survey Specialist acted as Team Leader of the survey team and other senior officials coordinated overall process including liaison with the client. The survey Specialist has taken lead role to write the report while other senior officials also played vital role to make sure the diverse analysis of the collected data. 2.5 Program used for data entry operation and analysis The ICS end-user data was entered in a customized software using CSPro 6.0 incorporating necessary logical and range checks to minimize data entry error. After the completion of data entry, it was further checked and fixed the errors and was ready for analysis. Dummy tables were prepared and the data was analyzed accordingly in SPSS version 17. The database with all the data is preserved for future references and make it available to client if and when necessary. 3. Constraints and Limitations of the survey There were some constraints and limitations perceived by the DPC Groups of Consultants during the survey, a few of those are mentioned below:  The allocated time for this survey was too short; specifically, for finding the respondents at the isolated and inaccessible location of the coastal districts of the country;  A big part of the Bagerhat district where End Users were identified was hard to reach due to lack of proper communication; so enumerators had to take rickshaw, local boat, even walk a while to collect an interview which limited their work or pushed them to work even at night in other locations to complete the survey within the given time;  More time was required than assumed to reach the right respondents because the details of the ICS users related to location and specific address were not readily available in many cases;  It was not common, but in some cases, on-going municipal election became as a major obstacle which controlled the free movement in the locality and sometimes threatened the female enumerators due to unavoidable circumstances; 4. Findings of the survey The survey conducted with 134 respondents who had been interviewed at the End User level of Improved Cook Stoves. The survey was conducted in three (3) districts namely Bagerhat, Chapai Nawabgonj and Gaibandha. The respondents were scattered in the villages, semi urban areas and urban locations. In Bagerhat district total forty eight (48) interviews conducted while in Chapai Nwabgonj were forty seven (47) and in Gaibandha were thirty nine (39) only. In general, the major findings are positive of purchasing and use of ICS while the users also have reported some problems they faced. 4.1 Specific findings based on data analysis 88 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT One of the major findings is that young groups are the main users of ICS as the data shows that about 83% users are between 15-45 years old while 62% of the users are between 15-35 years old; and only 6% users are above 55 years old; and 68% respondents were female and 32% were male (Reference: Table-1). It is a notable finding that the literate people are the main users of ICS. As per Reference Table-1, the data shows that only three (3%) percent users are illiterate while 33% has basic education (14% users completed Class I-V and 19% completed Class VI-X); and 64% have obtained at least one certificate (38% are SSC/HSC; 26% are Graduate and above). All the respondents family are users of ICS, although the duration of using time of ICS is noted around six (6) months only by 96% of respondents, and out of them 59% have been using it for less than three (3) months. However, they use it for the majority of their cooking as 83% of them said they cook for 2-3 times in a day; and average one hour time is required to complete the cooking reported by 78% respondents as stated in the several reference tables (Reference: Table-2; Table-3; and Table-4). Eighty percent (80%) of ICS users have to buy fuel (Reference: Table-5) and maximum of them have to use Palette and a specific size of firewood which is not easily available in their locality (Reference: Table￾6). The families who consisted of the highest total members (six (6) in the family) that are using ICS, it is a dominating figure (91%) for their cooking while families consisting of less than four (4) members are second highest users as noted at 53% (Reference: Table-7); although only 39% use ICS for all cooking (Reference: Table-8). Most of them are also using other stoves to complete their daily cooking while the majority (64%) use traditional earthen stove (Reference: Table-9), but alternative choices or preference of using other stoves are mainly stoves required natural gases; 45% preferred stove by piped gas and 37% preferred by LPG (Reference: Table-10). The factors influenced (highest to lowest) in purchasing ICS as per Reference: Table-11, are as follows:  Energy saving,  Time saving,  Lower cost of energy,  Attractive design,  Manufacturer's warranty,  Lower price,  Cost subsidy,  Easily portable, and  Smokeless cooking Almost 98.5% of ICS users shared their ideas and experiences of using ICS with others (Reference: Table-12) while they shared with other households, neighbors, community members, ICS suppliers and others (Reference: Table-13); 94% of the respondents reported that they are benefitted of using ICS (Reference: Table-14). The major benefits of ICS using are mentioned (Reference: Table-15) as follows:  Saving fuel / firewood  Reduced smoke and improve home environment  Cost saving  Health benefit  Better quality of cooking  Time saving in collecting fire wood About 90% of respondents reported that they did not face any problems in purchasing ICS while about 10% of them complained that they faced problems for purchasing ICS (Reference: Table-16). The problems faced to buy ICS (Reference: Table-17) are mainly as follows: 89 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT  No installment facility  No subsidy  Scarcity in market  Quality of construction  Expensive It is remarkable that 79% of users (Reference: Table-18) said that they do not face any problems for using ICS while 20% of them faced the following problems (Reference: Table-19):  Problem in refueling  Takes more time for cooking  Excess smoke/Kali  No energy savings  Not environment friendly  Fan shuts suddenly  No cost savings  Not health friendly It is significant that 79% of users mentioned that they use 79% firewood; 64% agri-by-product and 19% Palette as fuel for other cook stoves (Reference: Table-20); while 71% users use Palette; 62% firewood and 2.2% agri-by-product as fuel in Improved Cook Stoves (Reference: Table-21). It seems firewood still one of the most important fuels although agri-by-product was reduced a huge amount for ICS. A total of 78% respondents replied that there were no “other ICS design available during purchase” while only 9% replied affirmative (Reference: Table-22). A total of 78% know the factors influences to buy ICS (Reference: Table-23); the three main factors influences to buy ICS are energy saving, time saving and lower cost of energy (Reference: Table-24). There are five more tables inserted for additional references. 5. Recommendations There are many areas that can be improved and some other aspects must be improved. Based on the findings and specific comments from the end users the following recommendations are made for future considerations: a) The design of the Cook Stove needs modifications as many of the users mentioned about the design, although with a total of one third of sample households using the ICS for all cooking that is quite a good number. It may be increased if the Cook Stove were to be made in a modified design; b) The new design must be easy in refueling, take less time for cooking, produce less smoke, and provide energy savings as well as be environment and health friendly; c) Fuel for ICS must be easily available in the locality or nearby market, which will help people to increase use of ICS. Some of the users complained about the fuel which must be gotten from a selective source only; i.e. Palette which is not easily available and a specific size of firewood that needs more time to make it in a respective size; d) Consider conducting a survey among different income categories. The ICS survey was concentrated in the medium and lower-medium income communities and semi-urban areas. As such, certain findings may warrant further investigation, notably those relating to the pricing of the ICS in areas of low-income urban users and the rural poor communities, particularly to ascertain if pricing is a significant barrier to deployment. e) The ICS must be available as per demand; at present in some areas there is a shortage of it in the local market where people demand it; 90 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT f) Quality of construction must be maintained as some of the users reported bad quality of products, which has an effect on new users; g) ICS information campaign needs to be improved using appropriate methods and materials that will help to reach various income groups and different localities as well. 6. Lessons Learned The following are the lessons learned: • The ICS survey was concentrated in the medium and lower-medium income communities and semi-urban areas. As such, certain findings may warrant further investigation, notably those relating to the pricing of the ICS. • Improved Cook Stove is well liked by the majority of users although there are some limitations at the design and construction level and with fuel selection. • Most end-users have less than 12 months of utilization experience. The performance of the ICS should be further monitored and assessed after the stoves have been in use for at least a year to determine the long-term acceptance for the ICS. • There are many factors influencing deployment of the ICS and its use. They include: o Marketing, availability in the local market and usefulness at the household level. o The price of ICS may be a factor particularly for the low-income community. • Issues pertaining to quality and design should be further investigated. • Given the overwhelming positive response from end users and given that the majority of the end users indicated they have shared experiences, the potential exists to leverage this in the overall promotion and marketing of the ICS. This could include having selected end users being an integral part of any ICS marketing events. 7. Conclusions This ICS End User survey is a part of Mid-Term Performance Evaluation of Catalyzing Clean Energy in Bangladesh (CCEB) program, which has been conducted to assess any constraints and opportunities associated with achievement of expected program results; to provide specific recommendations for opportunities to enhance programmatic effectiveness and impact and strengthen the current approach; and to determine the extent to which the CCEB activities to support market development are likely to be sustained. The findings mentioned above provide a wide range of opportunities to review the existing program and would be helpful for further modifications. This program is well liked although some limitations are there in marketing and construction level, which has a major impact in end user level. The price is a big factor for the low-income community who really need this support but cannot afford at this stage. It is also important to keep the existing users on track providing operation and maintenance (O&M) services so they can be continued as regular and permanent customer of the product. The users having good results are the ambassador of the product and can be helpful to increase the selling of products in the neighborhood. It is suggested to develop linkages with the government of Bangladesh (GOBGOB), more specifically with the Department of Environment (DOE) and Ministry of Environment and Forestry (MOEF), as well as maintain Environmental Quality Standards (EQS) to avoid environmental hazards and contribute to achieve Sustainable Development Goals (SDG). It may be appreciable to build up a network between producers, distributors and local non￾government organizations (NGOs) who can assist for subsidy or loan for installments for the poor section of the community. At the same time, more production and highest selling will help to reduce ICS price and increase the accessibility of various income level people. Design of ICS may also need to be modified for reducing fuel consumption, to keep environment less polluted or pollution free, to 91 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT be functional for using various size/types of fuel/firewood, user friendly and minimum cooking time. Many users’ first choice is a cost effective and portable improved cook stove that will reduce smoke and cooking time. A total of 24 Reference tables inserted in the next pages are discussed in the findings section of this report and a few more tables were added for further reference that summarize respondent’s answers as per survey questionnaires. 92 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT References Tables 29. Tables with summary of respondents answer as per survey questionnaires for additional references: Table 1: Number and Percentage of Respondents showed by Education, Age and Gender Responden t by Gender Respondent Age Respondent Education Total Respondents Illiterat e Grade I-V Grade VI-IX SSC /HSC Graduate or above Male (32%) 15-25 years - - 1 (33.3) 1 (33.3) 1 (33.3) 3 26-35 years - 2 (11.8) 3 (17.6) 6 (35.3) 6 (35.3) 17 36-45 years - 1 (6.7) 5 (33.3) 5 (33.3) 4 (26.7) 15 46-55 years - 2 (50.0) - - 2 (50.0) 4 56+ years - 1 (33.3) - 1 (33.3) 1 (33.3) 3 Total - 6 (14.3) 9 (21.4) 13 (31.0) 14 (33.3) 42 Female (68%) 15-25 years - 2 (11.8) 4 (23.5) 8 (47.1) 3 (17.6) 17 26-35 years - 3 (6.7) 7 (15.6) 19 (42.2) 16 (35.6) 45 36-45 years 2 (15.4) 3 (23.1) 3 (23.1) 3 (23.1) 2 (15.4) 13 46-55 years - 3 (27.3) 2 (18.2) 6 (54.5) - 11 56+ years - 2 (40.0) 1 (20.0) 2 (40.0) - 5 Total 2 (2.2) 13 (14.3) 17 (18.7) 38 (41.8) 21 (23.1) 91 Grand Total: 4 (3%) 19 (14%) 26 (19%) 51 (38%) 35 (26%) 133 Percentages and total are based on total respondents; here is missing data of one respondent out of 134. Table 2: Length of Time needed for each Cooking done with ICS Length of time needed for each cooking Percent of Cooking done with ICS Total <=10% 10-25% 25-50% 50-75% >75% * Half-an hour - - - 1 (3.2) 2 (3.8) 3 (2.2) One hour 4 (80.0) 17 (100.0) 23 (79.3) 18 (58.1) 42 (80.8) 104 (77.6) Two hours 1 (20.0) - 6 (20.7) 11 (35.5) 8 (15.4) 26 (19.4) Three hours - - - 1 (3.2) - 1 (0.7) Total 5 (100.0) 17 (100.0) 29 (100.0) 31 (100.0) 52 (100.0) 134 (100.0) Notes: Figures in the parenthesis indicates the percentage; >75% means almost all cooking is done by Improved Cook Stoves 93 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 3: Duration of using ICS (month) Duration of using ICS (month) Frequency Percentage of Respondents 1-3 months 79 59.0 4-6 months 50 37.3 7-9 months 4 3.0 >12 months 1 0.7 Total 134 100 Mean ± SD 3.36 ± 1.93 Table 4: Number of Times usually Cook/Day Table 5: Buy Fuel for Cooking Table 6: Opportunity of Collecting Fuel by category/type of fuel Number of times usually cook/day Frequency Percentage of Respondents Once 21 15.7 Twice 69 51.5 Thrice 44 32.8 Total 134 100.0 Buy fuel for your cooking Frequency Percentage of Respondents Yes 107 80 No 27 20 Total 134 100 Fuel type for ICSa Buy fuel for cooking Total Yes No Firewood 64 (59.8) 19 (70.4) 83 Agri-by-product 3 (2.8) - 3 Palette 86 (80.4) 9 (33.3) 95 Wood dust 1 (0.9) 1 (3.7) 2 Others 6 (5.6) 1 (3.7) 7 a. Group Fire wood in specific size and Palette specially made which is not easily available * Percentages and totals are based on respondents 94 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 7: Cooks for average number of persons daily Table 8: Percent of Cooking done with ICS Table 9: Type of Other Stoves Use by Respondents Cooks for # of Persons per day Frequency Percentage of Respondents <=4 71 53 5-6 51 38.1 7-8 9 6.7 9+ 3 2.2 Total 134 100 Mean ± SD 4.64 ± 1.59 Percent of cooking done with ICS Frequency Percentage of Respondents <=10% 5 3.7 10-25% 17 12.7 25-50% 29 21.6 50-75% 31 23.1 > 75% (Almost all cooking) 52 38.8 Total: 134 100 Type of other stoves used Frequency Percentage of Respondents Earthen stove 64 76.2 Zik chula 1 12 Rice cooker 29 34.5 LPG stove 16 19 Electric heater 5 6 Bondhu chula 5 6 # of cases 84 95 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 10: Preference of using other stoves Preference of using other stoves Frequency Percentage of Respondents Piped gas 59 44.7 LPG 49 37.1 Electricity from grid 10 7.6 Renewable electricity 6 4.5 Traditional fuel 10 7.6 Kerosene 1 0.8 Other 13 9.8 # of cases 132 Table 11: Factors influenced in Purchasing ICS by its benefits of use Factors influenced in purchasing ICSa Benefits of using ICSa Tot al Better quality cooking Saving fuel wood Cost savings Time saving in collecting fuel wood Health benefit Reduced smoke improve home environment Can do other works/S ave time Portable /Can cook in convenien t place Others Energy saving 81 (91.0) 109 (97.3) 95 (94.1) 61 (96.8) 89 (94.7) 101 (93.5) 18 (100.0) 16 (100.0) 4 (100.0) 115 Time saving 80 (89.9) 96 (85.7) 88 (87.1) 60 (95.2) 79 (84.0) 97 (89.8) 17 (94.4) 13 (81.3) 4 (100.0) 109 Lower cost of energy 69 (77.5) 91 (81.3) 89 (88.1) 56 (88.9) 81 (86.2) 90 (83.3) 15 (83.3) 14 (87.5) 2 (50.0) 98 Attractive design 68 (76.4) 81 (72.3) 69 (68.3) 40 (63.5) 70 (74.5) 72 (66.7) 8 (44.4) 6 (37.5) 2 (50.0) 88 Manufacturer's warranty 55 (61.8) 72 (64.3) 65 (64.4) 41 (65.1) 66 (70.2) 67 (62.0) 7 (38.9) 9 (56.3) 2 (50.0) 76 Lower price 20 (22.5) 22 (19.6) 19 (18.8) 18 (28.6) 20 (21.3) 21 (19.4) - 3 (18.8) - 22 Cost subsidy 21 (23.6) 21 (18.8) 20 (19.8) 19 (30.2) 21 (22.3) 21 (19.4) - - - 21 Easily portable 7 (7.9) 17 (15.2) 17 (16.8) 5 (7.9) 11 (11.7) 18 (16.7) 10 (55.6) 3 (18.8) - 18 Smokeless cooking 5 (5.6) 15 (13.4) 12 (11.9) 3 (4.8) 7 (7.4) 15 (13.9) 8 (44.4) 7 (43.8) 1 (25.0) 16 Save time/can do other work simultaneously - 5 (4.5) 5 (5.0) 2 (3.2) - 5 (4.6) 4 (22.2) 2 (12.5) 1 (25.0) 5 Gift 3 (3.4) 4 (3.6) 4 (4.0) 4 (6.3) 4 (4.3) 3 (2.8) - - - 4 Recommended by neighbors 3 (3.4) 3 (2.7) 2 (2.0) 2 (3.2) 2 (2.1) 2 (1.9) - - - 3 Supplies at home 2 (2.2) 2 (1.8) 2 (2.0) 1 (1.6) 2 (2.1) 2 (1.9) - - - 2 96 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Factors influenced in purchasing ICSa Benefits of using ICSa Tot al Better quality cooking Saving fuel wood Cost savings Time saving in collecting fuel wood Health benefit Reduced smoke improve home environment Can do other works/S ave time Portable /Can cook in convenien t place Others Installment facility 1 (1.1) 2 (1.8) 2 (2.0) 1 (1.6) 1 (1.1) 2 (1.9) 1 (5.6) - - 2 Can cook in convenient place 1 (1.1) - 1 (1.0) - - 1 (0.9) - - - 1 Others 5 (5.6) 6 (5.4) 6 (5.9) 6 (9.5) 5 (5.3) 6 (5.6) 1 (25.0) 1 (5.6) - 6 Note: Percentages and totals are based on respondents. a. Group Table 12: Shared experience of using with others Table 13: Persons Shared ICS Experiences with… Table 14: Benefitted from ICS use Shared experience of using ICS with others Frequency Percentage of Respondents Yes 132 98.5 No 2 1.5 Total 134 100 Persons shared the experiences with… Frequency Percentage of Respondents Other HH 125 95.4 Other Community Members 88 67.2 ICS suppliers 27 20.6 Other persons 5 3.8 # of cases 131 Benefitted from ICS use Frequency Percentage of Respondents Yes 125 94.0 No 8 6.0 Total 133 100 97 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 15: Benefits of using ICS Benefits of using ICS Frequency Percentage of Respondents Better quality cooking 89 71.2 Saving fuel wood 112 89.6 Cost savings 101 80.8 Time saving in collecting fuel wood 63 50.4 Health benefit 94 75.2 Reduced smoke to improve home environment 108 86.4 Can do other works/save time 18 14.4 Portable/Can cook in convenient place 16 12.8 Others 4 3.2 # of cases 125 Table 16: Problems faced in Purchasing ICS Problem faced in purchasing ICS Frequency Percentage of Respondents Yes 12 9 No 120 89.6 Can't remember 2 1.5 Total 134 100 98 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 17: Problems faced in Purchasing ICS Table 18: Problems faced in using ICS Problems faced in purchasing ICS Frequency Percentage of Respondents Scarcity in market 4 33.3 Quality of construction 2 16.7 No subsidy 4 33.3 No installment facility 7 58.3 Expensive 1 8.3 # of cases 12 Problem faced in using ICS Frequency Percentage of Respondents Yes 27 20.1 No 106 79.1 Don't know 1 0.7 Total 134 100 99 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 19: Problems faced in using ICS Table 20: Type of fuel used in other stoves Problems faced in using ICS Frequency Percentage of Respondents Takes more time 8 29.6 No energy savings 6 22.2 No cost savings 5 18.5 Not environment friendly 6 22.2 Not health friendly 3 11.1 Excess smoke/Smut 7 25.9 Problem in refueling 16 59.3 Fan shuts suddenly 6 22.2 Others 1 3.7 # of cases 27 Type of fuel used in other stoves Frequency Percentage of Respondents Firewood 66 78.6 Agri by-product 54 64.3 Palette 16 19.0 Wood dust 5 6.0 LPG 16 19.0 Electricity from grid 36 42.9 # of cases 84 100 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 21: Type of fuel used in ICS Table 22: Any other ICS design available during purchase Table 23: Know the influential factors to buy ICS Know the influential factors to buy ICS Frequency Percentage of Respondents Yes 42 77.8 No 8 14.8 Don't know 4 7.4 Total 54 100 Type of fuel used Frequency Percentage of Respondents Firewood 83 61.9 Agri by-product 3 2.2 Palette 95 70.9 Wood dust 2 1.5 Others 7 5.2 # of cases 134 Any other ICS design available during purchase Frequency Percentage of Respondents Yes 12 9 No 104 77.6 Don't know 18 13.4 Total 134 100 101 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 24: Factors influenced in purchasing ICS Factors influenced in purchasing ICS Frequency Percentage of Respondents Energy saving 124 92.5 Time saving 118 88.1 Lower cost of energy 105 78.4 Attractive design 90 67.2 Manufacturer's warranty 78 58.2 Lower price 24 17.9 Cost subsidy 22 16.4 Easily portable 21 15.7 Smokeless cooking 20 14.9 Save time/Other work can do simultaneously 8 6.0 Gift 4 3.0 Recommended by neighbors 3 2.2 Supplies at home 2 1.5 Installment facility 2 1.5 Can cook in convenient place 2 1.5 Others 6 4.5 # of cases 134 * Multiple responses considered 102 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 25: Items usually cooked in ICS * Multiple responses considered Table 26: Items usually cooked in other stoves Items usually cook in other stoves Frequency Percentage of Respondents Rice/Ruti 75 89.3 Curry/Dal 69 82.1 Snacks 23 27.4 Tea 29 34.5 Others 10 11.9 # of cases 84 * Multiple responses considered Table 27: Know if the price of ICS was subsidized Know if the price of ICS was subsidized Frequency Percentage of Respondents Yes 30 22.4 No 50 37.3 Don't know 54 40.3 Total 134 100 Items usually cooked in ICS Frequency Percentage of Respondents Rice/Ruti 112 83.6 Curry/Dal 125 93.3 Snacks 28 20.9 Tea 45 33.6 Others 6 4.5 # of cases 134 103 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 28: Person who made decision to buy ICS * Multiple responses considered Person who made decision to buy ICS Frequency Percentage of Respondents Respondent 110 82.1 Main earner 59 44.0 Principal user 40 29.9 HH head 33 24.6 Family/community decision 6 4.5 Others 2 1.5 # of cases 134 104 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Table 29: Suggestions from ICS Users Sl No. Suggestions from ICS users Frequency Percentage of Respondents 1 Self-needed to insert fuel/ firewood 25 21.7 2 Ensure palette supply 24 20.7 3 Reduce palette price 17 14.7 4 Reduce stove price 69 60.0 5 Time wastage to make firewood to small size 8 7.0 6 Need installment facility 24 20.9 7 Increase sales promotion & stove supply 16 13.9 8 Firewood instead of palette will be better 14 12.2 9 Use quality materials for stove 8 7.0 10 Double burner is better 5 4.3 11 Make changes in design28 32 27.8 # of cases29 116 28 Make Changes in Design includes ‘Better fan/charger’, ‘Enlarged entrance space’, ‘Enlarge combustion space’, ‘Make stove a bit higher’, ‘Include better quality handle’ and ‘Make body heat resistant’. 29 Multiple Responses Considered 105 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT List of Annexures: A. Photos of different design of ICS and Users B. Terms of Reference (ToR) of Survey C. Original Survey Instruments (English and Bengali version) D. List of Interviewed Respondents E. Survey Questionnaires (134) 106 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Annex L: Mid-Term Evaluation SOW Scope of Work (SOW) for the Catalyzing Clean Energy in Bangladesh (CCEB) Project Mid-Term Performance Evaluation I. BACKGROUND USAID/Bangladesh awarded Contract Number: AID-338-C-13-00001 to Deloitte Consulting LLP to implement the Catalyzing Clean Energy in Bangladesh (CCEB) project on October 10, 2012. CCEB is a five-year project to support energy sector development for energy security, economic growth and climate change mitigation. Back then, Bangladesh’s energy sector was facing a number of challenges. Foremost among them were severe power shortages, skyrocketing natural gas consumption, and rural energy poverty. More than 50% of the population still lacked access to electricity. The electricity supply and demand gap was about 1,500 MW. Natural gas consumption was also growing tremendously. The problems of growing demand and looming shortages in the power and gas sectors were also exacerbated by problems of mismanagement, low energy prices and lack of strong regulatory environment. Despite these challenges, several components of the Bangladesh energy sector were demonstratively strong. Those positive factors, combined with favorable political will of the then government for addressing energy crisis, provided a favorable environment for continued USAID support in this sector, potential alleviation of some of the energy sector crises, and continued movement of Bangladesh along a low carbon development trajectory. President Obama and the United States Government (USG) have committed to the various goals of the Copenhagen Accord and made Global Climate Change (GCC) a top priority for foreign policy as well as foreign assistance. As part of the Global Climate Change Initiative, USAID supports the USG’s Enhancing Capacity for Low Emissions Development Strategies (EC-LEDS) program, of which Bangladesh is a partner country USAID/Bangladesh has a specific commitment to GHG emissions reduction and sequestration. The 2011-2016 Country Development Cooperation Strategy (CDCS) is oriented around four development objectives (DOs). DO4: ”Responsiveness to Climate Change Improved” aims to increase the nation’s responsiveness, adaptation to climate change, and mitigation of climate change through three inter￾related intermediate results(IR):1) improved management of naturalresources; 2) enhanced adaptation capacity and resilience to shocks; and 3) strengthened capacity for low emissions development focusing on mitigation. CCEB contributes to the achievement of CDCS IR4.3: Strengthened Capacity to Reduce Emissions and the sub-IRs 4.3.1: Improved Enabling Environment for Low Emissions Strategies; 4.3.2: Increased Adoption of Renewable Energy; and 4.3.3: Improved Energy Efficiency and Conservation. The results that CCEB contributes to are shown in Figure1 below. Figure 1. CCEB Results Framework The CCEB Development Hypothesis is that, USAID support for enhanced planning capacity for energy sector and low-emissions development, increased investment in energy efficiency and renewable PROJECT TO BE EVALUATED Project Name Catalyzing Clean Energy in Bangladesh Contract Number Contract Number: AID-338-C-13-00001 Original Project Date October 10, 2012 - October 9, 2017 Original Funding $14,990,150 Implementing Partner Deloitte Consulting LLP 107 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT energy, and an improved energy sector regulatory framework will result in strengthened Bangladesh capacity to reduce emissions. The Results Framework depicted above is a graphical depiction of the Theory of Change. Please see Annex 1 for CCEB M&E Plan for more details. The CCEB program is intended to meet USAID’s clean energy directive, through deployment of clean energy. It will also support the EC-LEDS program in Bangladesh by strengthening GOB capacity for energy sector analysis, planning, and policy-making to enable low emissions development. It was envisioned in the Section C-Statement of Work (SOW)30 of the contract that the CCEB project would address a targeted number of issues that meet the criteria indicated below: • Support energy sector development for energy security, economic growth, and climate change mitigation to Bangladesh. • Build capacity to design and implement supportive policies and regulations, and increase utilization of clean energy approaches and technologies for energy sector development on a low carbon trajectory. The key two results expected under this program are: (i) Enabling environment in place for low emissions development as a result of BERC and building capacity of GOB institutions in energy sector analysis for low-emission planning and policy making, and (ii)Increased energy efficiency and conservation through increased adoption of energy efficiency technologies and practices by industry; implementation of demand-side management programs by utilities; and increased use of improved, more efficient cook stoves. II. PROJECT COMPONENTS The CCEB project consists of two components with five tasks: Component 1: Improve Enabling Environment for Low Emissions Development Task 1: Improve Regulatory Environment for Clean Energy Development: The program focuses on assisting Bangladesh Energy Regulatory Commission (BERC) in meeting targets as established under the Maturity Model as well completing an end-of-year assessment of BERC’s performance. The project will ensure that BERC has a capacity to institute a regulatory framework 30 Project Description Document (Section C- Description/Specs./Statement of Work; AID-388-C-13-00001, p.7- 69) 108 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT that supports private sector investment in clean energy. Activities under this task will be implemented in close collaboration with Task 4 that will lead to promoting utility demand-side management (DSM) programs of the electric utilities. Task 2: Strengthen Analytical Capacity for Energy Sector Planning and Policymaking: Activities under this task directly support USAID’s commitment to the U.S. Government’s Initiative on Enhancing Capacity for Low Emission Development Strategies (EC-LEDS) in Bangladesh. The project will support enhancing GOB and Bangladeshi stakeholder capacity for energy sector and low emission development analysis and planning so that future energy and climate change plans and policies can be more oriented towards reduced GHG emissions (such plans and policies may include future versions of the BCCSAP, national energy policy, and energy component of the five-year plans). Component 2: Increase Energy Efficiency and Conservation Task 3: Promote Industrial Energy Efficiency Analysis and Adoption: This task will focus primarily on privately owned small and medium size industries particularly the ones that are export￾oriented. Interventions in this area will focus on catalyzing industry/ enterprise investments in EE through a combination of project development, financing facilitation, and business advisory services/ technical assistance. This activity will assist companies in developing a pipeline of commercially viable projects, bringing projects to financial closure, and providing relevant technical assistance and training to support project sustainability and replication. Activities will also support long-term sustainability and replication by strengthening private and public sector capacity for energy efficiency project development, financing, and implementation, through training of energy service providers. Task 4: Adopt Demand Side Management Programs for Electric Utilities: Utility led demand side management (DSM) programs have emerged as effective models for wide-scale adoption of energy efficiency and conservation practices. Activities underthistask will work with urban distribution utilities to implement demand-side management programs. This work will be done in a sequenced manner with the activities recommended in Task1, supporting BERC in first establishing a regulation requiring utility-DSM projects, and then working with the utilities on cost-effective DSM initiatives and appropriate smart-grid applications. Task 5: Conduct Market Analysis and Development for Improved Cook stoves: Under this task, a set of activities is envisaged to build a sustainable, improved cook stoves market in Bangladesh in order to reduce energy consumption and green house pollutants, which will be achieved through expanding the market for improved biomass cook stoves by developing sources of finance for consumers and/or clean energy businesses engaged in supplying the market with improved biomass cook stoves; strengthening businesses involved in the cook stoves supply chain; and better understanding consumers in order to generate market demand. Please see Annex-2 for CCEB Contract Section-C for deliverables and other details and Annex-3 for a list of districts where the project works. III. EVALUATION OBJECTIVES The purpose of the CCEB mid-term performance evaluation is to assess the extent to which the CCEB project is on track to meet its overall goals and inform management of any challenges or opportunities that warrant adjustments to the project to ensure the achievement of those results. The findings and recommendations of the evaluation will be used to improve implementation of the project and inform the design of other relevant DO4 projects. With the exclusion of procurement sensitive sections, USAID intends to disseminate the report to pertinent stakeholders (i.e. Bangladesh Energy Regulatory Commission (BERC), relevant government ministries and agencies, consumer groups etc.) and to general public through the Development Experience Clearinghouse (DEC). 109 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT The specific objectives of this evaluation are to: Determine the extentto which theCCEB projectis on track (including process and outcomes)tomeetits overall goals acrossthe twomajor CCEB components. The evaluation will – 1. Identify constraints and opportunities associated with achievement of expected projectresults. 2. Provide specific recommendationsfor opportunitiesto enhance programmatic effectiveness and impact and strengthen the approach within the confines of the CCEB contract 3. Determine the extentto which USAID investmentsin the CCEB activity are likely to be sustained and make viable recommendationsforfuture (post CCEB) projectfocus areas. IV. EVALUATION QUESTIONS: 1. To what extentis the CCEB project on track, in terms of progress and outcomes,to meetits overall goalsfor the five tasks underthe components A and B? 2. What are the opportunitiesto enhance programmatic approach and effectiveness plausible underthe stipulation ofthe contract? 3. What have been the major constraints and opportunities with respect to sustainability of the interventions? What measuresshould be taken to enhance sustainability? 4. Till date, how effectively has gender been integrated orincorporated in the interventions?What have been the challenges and opportunities, if any? V. EVALUATIONMETHODOLOGY The M&E contractor for the USAID Bangladesh Economic Growth (EG) office, Accelerating Capacity for Monitoring and Evaluation (ACME), will carry out this evaluation. Based on the SOW from USAID, ACME will recruit and manage the evaluation team. The evaluation work plan and evaluation protocol drafted by the evaluation team will be reviewed and approved by USAID. The detailed methodology of the evaluation will be designed by the evaluation team in the work plan; this will include presentation of a data collection matrix that will explicitly link evaluation questions to particular data collection approaches and data sources. It is suggested that the evaluation team use a mixed- method approach utilizing both qualitative (including non-experimental design) and quantitative analysis (key informant interviews, stakeholder interviews/mini- surveys, and focus group discussions).The evaluation team should develop the best evaluation design methodology in light of the evaluation questions, timeframe, budget, data collection requirements, quality of existing data sources, and potential biases. The evaluators should utilize several different, yet complementary and inter-related forms of gathering information / data. These are: Document Review: Evaluation team members will review documents throughout the evaluation process including program reports, relevant studies and evaluations and BERC documents to ensure that comprehensive and grounded best practices will be identified. Key Informants Interview: The team will conduct one-on-one interviews with a variety of stakeholders including BERC, BPDB and various stakeholders, and other projects supporting energy sector development. Self-assessment: The IPs will respond to a self-assessment either through a questionnaire or standard interview checklist put together by the evaluation team and approved by USAID before use. Expert Opinion Survey: Utilizing expert opinion is a technique used increasingly in the energy sector, particularly clean energy. The Evaluation team, with approval of USAID, can apply this method as well. 110 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Participatory Rural Appraisal (PRA): This form of survey method will be applied to seek views and opinion of the local community on the issues related to improved cook stoves program. This qualitative survey requires sufficient triangulation.31 Focus Group Discussions (FGD): FGD (small group of 6 to 10 people) will be used to lead open discussion through a skilled moderator to gather semi-structured qualitative data. The preselected participants will discuss issues and concerns based on a list of key theme drawn up by the moderator. No more than 10 questions will be addressed by a group. These sessions will encourage free flowing discussion about the activity. Social and Economic Analysis: This type of analysis might require sample survey (e.g. ICS acceptability and sustainability). Due to time and resource constraints, it is neither possible nor desirable to carry out a complete census for social and economic analysis. So, a sample size needs to be determined, based on the project area of focus, for the survey, which is statistically sound in terms of representativeness of the sample, and which is most widely used for inferential analysis. Standard statistical formula which is commonly used to determine a representative sample size stands as follows: This formula can be applied to estimates sample size at 95% level of confidence with 5% margin of error, which is universally accepted and practiced. The evaluators can use any other standard formula at their own convenience. In case where total population (universe) data is not available, or available but numbering of each unit of population is not possible, then skip interval method will be applied to select the primary sample units, say the households. There are three methods of choosing a respondent within the households – the birthday method, quotas and Kish Grid.32 To obtain the true reflection of the views of all walk of people (man, woman, youth, old) within the households/community, the Kish Gridtool is widely used. Structured questionnaire will be used to collect data in such cases. Regardless of data collection and analysis methods, USAID requires qualitative and quantitative data disaggregated by gender.33 Methodological limitations and challenges for this evaluation are expected to include: • Ensuring that samples of interview sources are sufficient to support evaluation findings; 31 Davis (Davis, A.C.S., ‘Participatory Rural Appraisal’, Rural Travel and Transport Program, 2001, 5.6.a, TRL Limited) reveals that PRA triangulation is often carried out in groups of at least 3 (three) to increase the credibility of each survey technique. 32 Kish, Lesile (1949), ‘A Procedure for Objective Respondent Selection within the Households’, Journal of American Statistical Association, 44(247) 380-387]. 33 USAIDADS 203.3.4.3 111 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT • Taking systematic actions to counter any biases in (a) reporting by data collection sources and (b) interpretations of collected data by the evaluation team; and ensuring “actual” results can be measured, which will only be possible if data can be gathered and analyzed beyond respondent perceptions. All the methodological strengths and weaknesses should be explicitly described in the evaluation report. VI. EXISTING SOURCES OF INFORMATION The evaluation team should consult a broad range of background documents apart from project documents provided by USAID/Bangladesh. USAID and CCEB project will provide the assessment team with a package of briefing materials which will include the documents indicated below: • Project Description Document (Section C- Description/Specs./Statement of Work; AID￾388-C-13- 00001, p.7-69) • CCEBDatabase • CCEBM&E Plan • Project Quarterly andAnnualreports, work plans, and managementreview • DQAreports • USAID/BangladeshDO4 PerformanceManagement Plan • USAID/Bangladesh CountryDevelopment Cooperation Strategy 2011 – 2016 (Public version) Theevaluationteammust alsoresearchanyotherrelevant documentsfromGOB, Donors andotherinternational organizations pertinentto the sectorthat CCEB serves. VII. EVALUATIONTEAM COMPOSITION The evaluation team should consist of two consultants: one international and one national. The international consultant will work as Team Leader who will be assisted by a national expert. Following skills and qualifications are expected from the evaluation team: 1. Team Leader/ International Energy Expert (33 person days) Duties and Responsibilities The Team Leader/International Energy Expert will provide overall leadership for the team, and s/he will finalize the evaluation design, coordinate activities, arrange periodic meetings, consolidate individual input from the other team member, and coordinate the process of assembling the final findings and recommendations into a high quality document. S/he will lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Bangladesh team and other major partners. Skills and Qualifications: • Master’s or higher degree in energy, and environmental economics or other related fields; • Experience in designing, implementing and evaluating projects aiming at capacity development of public institutions, especially in the energy sector • At least 10 years of experience in implementing and evaluating energy programmes and/or projects for USAID or other international development agencies; • At least 10 years of experience in conducting quantitative and qualitative evaluations • Knowledge of USAID regulations and systems including GCC indicators, performance monitoring guidance, evaluation policy, gender policy, annual reporting etc. is essential • Experience working with government institutions, on policy reform, public administration improvements, infrastructure financing and taxation; • Knowledge of capacity building approaches and models that have been used to engage institutions in change and change management; Knowledge in Energy audits is a plus; • Relevant experience in Bangladesh or South Asia preferred • Strong analytical skills; 112 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT • Excellent communications and writing skills in English; • Excellent coordination and team working skills; • Knowledge of Bangla is an asset 2. National Energy Expert (28 person days) Duties and Responsibilities: The National Energy Expert will be responsible for collection of background materials upon request by the evaluation team leader. S/he will actively participate in the desk review of materials and assist the team leader in developing methodologies, work plans and report outlines. The National Energy Expert will assist the Team Leader in setting and conducting interviews with relevant stakeholders and actively take part in these. S/he will participate in team meetings, site visits, and draft the sections of the report relevant to his/her expertise. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Bangladesh or otherstakeholders. Skills and Qualifications: • Master’s degree in energy, natural resource management, environmental economics or other related fields; • 7-10 years of experience in energy project development, evaluation, or implementation. • Strong understanding and knowledge of GHG emissions context with regard to energy sector of Bangladesh,including clean energies and conventional energies; • Experience with USAID GCC/NRMprojects is a strong asset; • Strong analytical skills; • Strong oral communications and writing skills in English; • Excellent team working skills; Conflict of Interest All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated. USAID will provide the conflict of interest forms. VIII. DELIVERABLES All deliverables are internal to USAID and the Evaluation Team unless otherwise instructed by USAID. Evaluation deliverables are indicated below: Evaluation Team Planning Meeting: During the meeting, the team should review and discuss the SOW in its entirety, clarify team member roles and responsibilities, prepare the work plan, develop data collection methods, review and clarify any logistical and administrative procedures for the assignment and instruments, and prepare for the in-brief with USAID/Bangladesh. Work Plan: Prior to initiation of the evaluation activities, the evaluation team will provide a detailed initial work plan to the ACME COR. The ACME COR will provide any necessary feedback or edits to the work plan, after which the evaluation team will have three days to submit a final version of the document. The initial work plan will include (a) a task timeline, (b) a description of the methodology to answer each evaluation question, (c) team responsibilities, (d) document review process, (e) key informant and stakeholder meetings, (f) site visits, and (g) draft and final report writing. The work plan will be submitted to the ACME COR at USAID/Bangladesh for approval no later than the 5th day after commencement of evaluation. In-briefing Meeting: The evaluation team will meet with USAID/Bangladesh within two working days of the Team Leader’s arrival in country. Evaluation Design Matrix: A table that lists each evaluation question and the corresponding information sought, information sources, data analysis methods, and limitations. The matrix should be 113 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT finalized and shared with USAID/Bangladesh before evaluation field work starts. It should also be included as an annex in the evaluation report. Data Collection Instruments: Development and submission of data collection instruments to USAID/Bangladesh during the design phase and after the evaluation is completed. Regular Updates: The Evaluation Team Leader will brief the ACME COR and any other designated evaluation POC on progress with the evaluation on at least a weekly basis, in person or by electronic communication. Any delays or communications must be quickly communicated to USAID/Bangladesh to allow quick resolution and to minimize any disruptions to the evaluation. Emerging opportunities to strengthen the evaluation should also be discussed with USAID/Bangladesh as they arise. Preliminary Draft Evaluation Report: The evaluation team will submit a Preliminary Draft Evaluation Report to the ACME COR five working days before the Mission debriefing. Within three working days after receipt, USAID staff will provide preliminary comments prior to the Mission debriefing. Debriefing with USAID: The evaluation team will present the major evaluation findings to USAID/Bangladesh through a PowerPoint presentation before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any preliminary recommendations. The team will consider USAID comments and incorporate them in the Draft Evaluation Report. Draft Evaluation Report: A draft report on the findings and recommendations should be submitted to USAID/Bangladesh 10 days after departure of international team leader. The written report should clearly describe findings, conclusions, and recommendations. The draft report must be of high quality with no grammatical errors or typos. A report is high quality when it represents a thoughtful, well￾researched and well organized effort to objectively evaluate what worked in the project, what did not and why. The draft report must have well-constructed sentences that are presented in a way that clearly presents findings, conclusions and recommendations. The report should answer all the evaluation questions and the structure of the report should make it clear how the questions were answered. The draft report must meet the criteria set forth under the Final Report section below. USAID will provide comments on the draft report within 10 working days of submission. The team will then have 5 work days to respond to the Mission’s comments and submit Final Evaluation Report to the Mission. Final Evaluation Report: The evaluation team will submit a Final Evaluation Report that incorporates Mission comments and suggestions no later than 10 working days after USAID/Bangladesh provides written comments on the Draft Evaluation report. The format of the final report is provided below. The report will be submitted electronically in English. It should be a thoughtful, well-researched and organized effort to objectively evaluate what worked in the project, what did not, and why. The draft report must have well-constructed sentences that are presented in a way that clearly presents findings, conclusions and recommendations. The report should answer all the evaluation questions and the structure of the report should make it clear how the questions were answered. The total pages of the final report, excluding references and annexes, should be no more than 30 pages. A second version of the evaluation report, excluding any potentially procurement-sensitive information, will be submitted to the Development Experience Clearinghouse (DEC) and disseminated among implementing partners and other stakeholders within ten days following approval from USAID. All quantitative data, if gathered, should be (1) provided in an electronic file in easily readable format; (2) organized and fully documented for use by those not fully familiar with the project or the evaluation; (3) owned by USAID and made available to the public barring rare exceptions. A thumb drive with all the data could be provided to the ACME COR. 114 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT The final report will be edited and formatted by the evaluation team and provided to USAID/Bangladesh 5 working days after the Mission has reviewed the content and approved the final revised version of the report. IX.REPORTING REQUIREMENTS: The total pages, excluding references and annexes, should not be more than 30 pages. The following content (and suggested length) should be included in the report: Table of Contents List of Acronyms Executive Summary – concisely state the project purpose and background, key evaluation questions, methods, most salient findings and recommendations (2-3 pp.); 1. Introduction – country context, including a summary of any relevant history, demography, socio- economic status, etc. (1 pp.); 2. The Development Problem and USAID’s Response – brief overview of the development problem and USAID’s strategic response, including design and implementation of the CCEB activity and any previous USAID activities implemented in response to the problem, (2-3 pp.); 3. Purpose of the Evaluation – purpose, audience, and synopsis of task (1 pp.); 4. Evaluation Methodology – describe evaluation methods, including strengths, constraints, and gaps (1 pp.); 5. Findings and Conclusions – describe and analyze findings for each evaluation question using graphs, figures, and tables, as applicable, and also include data quality and reporting system that should present verification of spot checks, issues, and outcomes. Conclusions should be credible and should be supported by the findings (12-15 pp.); 6. Recommendations – prioritized for each evaluation question; should be separate from conclusions and be supported by clearly defined set of findings and conclusions. Include recommendations for future project implementation or relevant program designs and synergies with other USAID projects and other donor interventions as appropriate (3-4 pp). 7. Lessons Learned – provide a brief of key technical and/or administrative lessons on what has worked, not worked, and why for future project or relevant program designs (2-3 pp.); 8. Annexes – to include statement of work, documents reviewed, bibliographical documentation, evaluation methods, data generated from the evaluation, tools used, interview lists, meetings, FGDs, surveys, and tables. The Evaluation Design Matrix must be presented as an annex to the report. Annexes should be succinct, pertinent, and readable. Should also include if necessary, a statement of differences regarding significant unresolved difference of opinion by funders, implementers, or members of the Evaluation Team on any of the findings or recommendations. The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins one-inch top/bottom and left/right. X. TIMELINE AND LEVEL OF EFFORTS: Work will be carried out over a period of six weeks, during August-September 2015. Below is an estimate of the evaluation level of effort (LOE): A. Preparatory Work Team Leader National Energy Expert Comprehensive document collection and review. 3 days 3 days Travel to Bangladesh 2 days 0 days Team planning meeting and meeting with USAID/Bangladesh. 1 day 1 day 115 CATALYZING CLEAN ENERGY IN BANGLADESH (CCEB) PERFORMANCE EVALUATION FINAL REPORT Development of evaluation work plan (concurrent with document review and initial meetings). 2 days 2 days Develop preliminary interview instruments and begin scheduling key interviews. 3 days 3 days B. Data Gathering In-country information and data collection. Includes interviews with key informants (stakeholders and USAID staff) and site visits. 18 days 18 days C. Data Analysis/Drafting Report Data analysis in preparation for presentations 3 days 3 days In-country discussion with USAID and presentation of preliminary analysis and draft of final report. 1 day 1 day Presentation of preliminary results and recommendations to the USAID/Bangladesh Mission and relevant stakeholders 1 day 1 day Depart Bangladesh 2 days 0 Analysis of data and draft of final evaluation report. Draft must be submitted within 10 working days after the departure of international team 5 days 5 days Evaluation team has ten days to update and finalize final evaluation report. 5 days Total Estimated Level of Effort 46 37 XI. SCHEDULING AND LOGISTICS: Funding and Logistical Support USAID/Bangladesh’s ACME project will be responsible for all off-shore and in-country administrative and logistical support, including identification and fielding appropriate local staff. They will take care of arranging and scheduling meetings, international and local travel, hotel bookings, working/office spaces, computers, printing, and photocopying. A local administrative assistant/coordinator may be hired to arrange field visits, local travel, hotel, and appointments with stakeholders and provide translation services. Scheduling Work is to be carried out over a period of approximately 6 weeks, beginning in October, 2015, with field work completed in the same month and final report and close out concluding o/a / November, 2015. See Annex-3 for Bangladesh Mission Holiday Schedule. A six-day work week (Saturday-Thursday) is authorized for the evaluation team while in Bangladesh. The evaluation team will submit a work plan as part of the evaluation methodology proposal with timeline and develop a GANTT chart displaying the time periods during which activities occur. Pre-departure arrangements should include: travel approval; airline tickets; visa; lodging; work facility and vehicle transport arrangements; dates for meetings with USAID/Bangladesh EG staff and key contacts, in-country travel agenda; and accommodations. 116