PERFORMANCE EVALUATION March 12, 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by Matthew Addison, Farhat Rahman, Marc D. Shapiro, and Elizabeth Stahl of Development & Training Services, Inc. (dTS). INITIATIVE PRIVATE FINANCING ADVISORY NETWORK Acknowledgements: The authors and dTS would like to thank the many people and institutions that supported the design and execution of the performance evaluation of CTI PFAN. K. Rayne Loken was instrumental in the design of the evaluation, communication with partners, and the drafting of the data collection instruments. USAID staff in the E3/Global Climate Change Office provided invaluable guidance and support. Peter Storey, Taiki Kuroda, and Elmer Holt, Jr. at CTI actively demonstrated commitment to learning from this pilot activity and supported dTS in obtaining necessary documentation and maximizing learning opportunities. This evaluation was carried out with support provided by the United States Agency for International Development (USAID), Global Climate Change Office (GCCO), Bureau for Economic Growth, Education, and Environment (E3), through the Global Climate Change Monitoring and Evaluation Project (GCC M&E), Contract Number AID-RAN-I-00-09-00015, Task Order Number AID-OAA-TO-12-00001. Development & Training Services, Inc. (dTS) is an international development company that leads initiatives in social and economic development with a view to promoting equality, accountability, and sustainability. For information about dTS and its projects worldwide contact: Development & Training Services, Inc. (dTS), 4600 North Fairfax Drive, Suite 402, Arlington, VA 22203, USA. Phone: +1 703-465-9388; Fax: +1 703-465-9344; Email: info@onlinedts.com; Internet: www.onlinedts.com. Recommended citation: Addison, Matthew, Farhat Rahman, Marc D. Shapiro, and Elizabeth Stahl. 2015. Performance Evaluation Final Report: Climate Technology Initiative Private Financing Advisory Network. Arlington, Virginia, USA: Development & Training Services, Inc. (dTS). GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network ii PERFORMANCE EVALUATION FINAL REPORT: CLIMATE TECHNOLOGY INITIATIVE PRIVATE FINANCING ADVISORY NETWORK March 12, 2015 DISCLAIMER The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. AUTHORS Matthew Addison, Farhat Rahman, Marc D. Shapiro, and Elizabeth Stahl of Development & Training Services, Inc. (dTS). GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network iii Contents ...................................................................................................................................................................... iii Tables ............................................................................................................................................................................. v Figures ............................................................................................................................................................................ v Acronyms ..................................................................................................................................................................... vi Executive Summary................................................................................................................................................ viii 1 Evaluation Purpose and Evaluation Questions ......................................................................................... 1 1.1 Evaluation Purpose ................................................................................................................................................................... 1 1.2 Evaluation Questions .............................................................................................................................................................. 1 2 Program Background ........................................................................................................................................... 2 2.1 The PFAN Approach ............................................................................................................................................................. 4 3 Evaluation Methods and Limitations .............................................................................................................. 6 3.1 Research Design ........................................................................................................................................................................ 6 3.2 Data Collection Methods ..................................................................................................................................................... 6 3.2.1 Data Sources .................................................................................................................................................................... 6 3.2.2 Data Collection ............................................................................................................................................................... 6 3.2.3 Data Analysis .................................................................................................................................................................... 7 3.2.4 Stakeholders Surveyed and Interviewed ......................................................................................................... 7 3.3 Data Quality ................................................................................................................................................................................. 7 3.4 Evaluation Limitations ............................................................................................................................................................. 8 4 Findings, Conclusions, and Recommendations .......................................................................................... 8 4.1 General Program ....................................................................................................................................................................... 8 4.1.1 USAID PFAN Targets ................................................................................................................................................. 8 4.1.2 PFAN General Assumptions and Objectives ............................................................................................... 9 4.1.3 Underlying HypothesIs ............................................................................................................................................ 10 4.2 Performance .............................................................................................................................................................................. 12 4.2.1 Introduction ................................................................................................................................................................... 13 4.2.2 Cost Effectiveness of PFAN in Clean Energy Financing ...................................................................... 14 4.2.3 Cost Effectiveness of Establishment and Maintenance of Lender-Developer Relationships 15 4.2.4 Leveraging Private Sector Resources .............................................................................................................. 16 4.2.5 Cost Effectiveness of Technology Deployment ....................................................................................... 16 4.2.6 Strengths and Weaknesses of the PFAN Organizational and Partnership Model .............. 18 4.2.7 Gender Considerations ........................................................................................................................................... 18 4.3 Overall Effectiveness ............................................................................................................................................................ 18 4.3.1 Operational and Coach Stakeholders ............................................................................................................ 18 4.3.2 Project Developers .................................................................................................................................................... 19 GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network iv 4.3.3 Financial Partners ........................................................................................................................................................ 20 4.4 Strengths and Weaknesses .............................................................................................................................................. 20 4.5 Effectiveness .............................................................................................................................................................................. 21 4.5.1 Closure Characteristics ........................................................................................................................................... 21 4.5.2 The Role of PFAN in Financial Closures ...................................................................................................... 22 4.5.3 Barriers to Clean Energy Investment .............................................................................................................. 24 4.5.4 PFAN and Gender ..................................................................................................................................................... 29 4.6 Sustainability and Replication .......................................................................................................................................... 31 4.6.1 Performance of PFAN Participants After Financial Closure ............................................................. 31 4.6.2 Updating or Transitioning PFAN to a Self supporting Entity ........................................................... 32 4.7 Recommendations ................................................................................................................................................................ 35 4.7.1 Monitoring and Evaluation .................................................................................................................................... 35 4.7.2 Performance .................................................................................................................................................................. 35 4.7.3 Effectiveness ................................................................................................................................................................... 36 4.7.4 Sustainability and Replication ............................................................................................................................... 36 4.7.5 Gender .............................................................................................................................................................................. 36 GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network v Table 1: USAID PFAN Expected Outcomes ............................................................................................................................................. x Table 2: USAID PFAN Expected Outcomes ............................................................................................................................................. 9 Table 3: Performance Comparison ............................................................................................................................................................... 14 Table 4: Cost Effectiveness of Developer-Investor Relationships ............................................................................................... 15 Table 5: Cost Effectiveness by Technology .............................................................................................................................................. 16 Table 6: PFAN Structural Effectiveness ....................................................................................................................................................... 18 Table 7: Project Developer Satisfaction with technical assistance and capacity building .............................................. 19 ........................................................................................................................... 20 ............................................................................................. 23 Table 10: Importance of the Three Prongs .............................................................................................................................................. 24 Table 11: Is PFAN Successfully Addressing Barriers to Clean Energy? .................................................................................... 26 Table 12: Barriers Not Adequately Addressed ...................................................................................................................................... 27 Table 13: Additional Areas for PFAN Assistance ................................................................................................................................. 28 Table 14 Selected Characteristics by Gender ......................................................................................................................................... 30 Table 15 Ranking Barriers to CE Investment ........................................................................................................................................... 30 Figure 1: PFAN Model .............................................................................................................................................................................................. 4 Figure 2: Services that Contribute to Securing Financing ................................................................................................................. 23 Figure 3: Major Barriers for Developers ..................................................................................................................................................... 25 GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network vi ADB Asian Development Bank AFD French Development Agency BEEF Bulgaria Energy Efficiency Fund CE Clean energy CO2e Carbon dioxide equivalent COI Conflict of interest CTI Climate Technology Initiative DFID United Kingdom Department for International Development dTS Development and Training Services, Inc. E3 Bureau for Economic Growth, Education and Environment EBPD Everyone but project developers EE Energy efficiency ECP I Energy Conservation Project I EGTT Expert Group on Technology Transfer FI Financial institution GCC Global climate change GCC M&E Global Climate Change Monitoring and Evaluation Project GEEP Georgia Energy Efficiency Program GHG Greenhouse gas GIZ German Agency for International Cooperation GWh Gigawatt hours ICETT International Center for Environmental Technology Transfer IFC International Finance Corporation IFI International financial institution IP Implementing partner IRB Institutional review board KII Key informant interviews LED Light emitting diode M&E Monitoring and evaluation MT Megaton MW Megawatt OP Operational partner OSO Operational stakeholder organization PFAN Private Financing Advisory Network PV Solar photovoltaic RE Renewable energy SDC Swiss Agency for Development and Cooperation SEF Sustainable Energy Facility SOW Statement of Work UNFCCC United Nations Framework Convention on Climate Change USG United States Government GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network vii USAID United States Agency for International Development USG United States Government GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network viii Introduction The purpose of the evaluation of the Private Financing Advisory Network (PFAN) activity is to assess the performance, effectiveness, and sustainability of PFAN and its participants. This evaluation, part of the portfolio of evaluations conducted for the Un Economic Growth, Education, and Environment (USAID/E3) under the Global Climate Change Monitoring and Evaluation Project (GCC M&E), is designed to provide information for future programmatic and policy￾related decision-making, offer contextual learning for USAID and other involved partners and stakeholders, and demonstrate accountability for resources. Recommendations on specific program elements in the evaluation may also be used to change or up . Project Background The PFAN activity is a program designed to assist clean energy (CE) project developers in accessing finance by providing technical assistance and capacity building and introducing them to investors. PFAN is designed to bridge the gap between financiers and project developers, and it provides mentoring to project developers to help them create more robust business plans and communicate effectively with potential sources of financing. PFAN exists for two reasons. The first reason is that market failure creates a number of barriers to increased CE penetration, and PFAN exists to address a combination of those barriers, primarily in the business prong. Second, based on the results of this survey, it is apparent that PFAN services also indirectly addresses some of the financial barriers PFAN was initiated by the Climate Technology Initiative (CTI)1 in cooperation with the United Nations Framework Convention on Climate Change (UNFCCC) Expert Group on Technology Transfer (EGTT), and is supported by a number of private sector companies involved in financing CE industries, including renewable energy (RE) and energy efficiency (EE). USAID is a partner of the PFAN alliance and provides support to the program through a cooperative agreement. The agreement is currently supported by a $5 million budget and lasts from October 2007 to September 2015. PFAN has established regional networks in Latin America, Africa, Asia, and Eastern Europe, as well as dedicated country-specific networks. Documents reviewed indicate that the intention is for PFAN to consider becoming a self-supporting entity. Evaluation Questions Evaluation questions are grouped according to issues of performance, effectiveness, sustainability, and replication. Performance 1. What has been the cost-effectiveness of PFAN in relation to: a. Clean energy technology financing; b. Establishment and maintenance of lender/developer relationships; c. Leveraging private sector resources, such as in-kind services and mentoring; and d. Clean energy technology deployment? 1 -PFAN. In this report, CTI is used in reference to the implementer, and PFAN is used in reference to the program. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network ix 2. What are the strengths and weaknesses of the PFAN organizational and partnership model? Effectiveness 3. Why have certain PFAN projects been more or less successful in the speed of achieving financial closure? 4. In what ways and to what extent do project developers credit PFAN for their successfully securing financing? 5. Are there barriers not being addressed or not being addressed effectively by PFAN? i.e., what additional assistance can PFAN offer under its mandate? Sustainability and Replication 6. What is the performance of PFAN participants post financial closure, including reaching and maintaining operational status, replicating or expanding business, and producing co-benefits for themselves or their community? 7. form or as the program transitions to a self-supporting entity? Evaluation Methods and Limitations The evaluation used a convergent parallel mixed methods design, which involved collecting qualitative and quantitative data concurrently and analyzing the two data sets separately. Data sources included administrative and institutional documents, existing performance information, an online survey instrument, and key informant interviews (KII). The online survey included project developers, mentors, operational stakeholders, resource partners, and financial institution representatives. Survey respondents volunteered to be interviewed. An additional request was made of project developers whose projects had reached financial closure, but the response rate for this key subgroup remained low. Some aspects of the analysis were limited by low response rates for closed projects and by the limited time to review additional key documents obtained at the end of the analysis period. Findings, Conclusions, and Recommendations PFAN is a unique program. There have been a few projects that target business and also provide finance. Other programs, like the International Fina commercial banks and provide technical assistance and finance. PFAN is focused on providing services to the project developer community to facilitate and increase CE investment. PFAN is expected to meet or exceed all performance targets and compares favorably with similar projects and programs designed to address barriers to CE investment2 . To date, 49 projects are estimated to have reached financial closure3 , meaning they have been matched with and funded by a financial institution, raising a total of $561.5 million. Another 255 projects, which would represent $6.7 billion of investment, remain in nt with USAID that of getting more projects financed over its objective of broadening access. Compared to investments in similar projects and programs though, funding to PFAN from USAID and US Department of State has 2 The data is a priori of actual closing and actual operations. 3 USAID financing. These are estimated to have reached financial closure because PFAN does not actually report based on closing but on the intent to close. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network x leveraged more private sector investment, avoided more tons of CO2 equivalent per dollar of USAID funding, and provided more megawatts (MWs) of CE of USAID funding. Table 1: USAID PFAN Expected Outcomes SUMMARY OF EXPECTED OUTCOMES Target Low Range Target High Range Achieved to Date # Total Projects inducted into Pipeline 54 92 304 # Total Projects reaching investor maturity 30 58 119 # Total Projects reaching Financial Close 11 27 49 % Project Developers reaching closure4 20% 29% 16.10% Total Financing Leveraged $72,500,000 $325,000,000 561,539,200 Aggregate Leverage Effect (1 :XX) 41 186 112.31 Source: EEM-A-00-08-00005, Modification 6. Subsequent modifications did not change the targets The one exception of currently meeting the target is the percent of project developers reaching financial closure. This is due to the fact that the evaluation is taking place while PFAN is still operating and the deadline for achieving these targets is not until September 2015. While PFAN met or exceeded most performance targets, there are insufficient data collected5 and survey responses to test whether PFAN contributed significantly to financial closure. In fact, the lack of funding for data collection, monitoring, and evaluation of PFAN prevents PFAN from testing its methods and refining its procedures. For example, no data is maintained systematically on the 85 to 90 percent of applicants not accepted into the PFAN pipeline. Important questions remain unanswered because of the lack of data, the answers for which may help PFAN and its funding partners be more effective and to use its scarce resources in a more targeted manner. While PFAN represents a cost- lack of consistent data on projects in the pipeline mean that even cost effectiveness calculations could potentially under-represent actual performance. In the opposite direction, insufficient data exists to definitively conclude the degree to which projects have, have started operations and will remain in operation. Similarly, although there is an apparent basic relationship between the time required to reach financial closure and perceived project risk, insufficient pipeline data are available to draw any meaningful relationships that could inform donors or PFAN, and too few project developers who reached closure responded to the survey. Available data suggest PFAN may have a neutral to positive impact on end-use beneficiaries from a gender perspective, but the data are sufficiently limited that conclusions must be heavily caveated. In terms of organizational and structural issues, stakeholders predominantly found the PFAN model moderately effective along several dimensions. Project developers who responded to the survey were moderately to completely satisfied with most elements of technical assistance and capacity building 4 This is the percentage of projects inducted into the pipeline that reach financial closure. Roughly 20 percent of the projects inducted so far have reached financial closure. 5 The collection of data on non-selected projects or on closed projects was not part of the funding of PFAN and this evaluation was meant to provide supporting evidence to the data which PFAN collects which are both before operation and before actual closing. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network xi provided. Project developers and financial partners pointed to strengths in advisor services, such as coaching, and investor relations. Respondents identified as weaknesses the effects of donor prioritization on funding decision making and the robustness of effort to facilitate introductions to finance and investors and early￾stage finance. In addition, PFAN advisory services might be expanded to include more thorough screening of potential developers prior to selection and assistance with the process of financial closing following the investor forum. Overall, PFAN was viewed as being effective, but developers recognize that barriers still remain. PFAN deals primarily with the clean energy business barriers and, within that area, several unaddressed barriers remain. These include such areas of assistance as early stage finance, expanding access to financiers, and assistance in negotiations. PFAN is unique in that it addresses almost exclusively the business barriers, which is where it has its competitive advantage. PFAN does not directly address policy or finance barriers, and doing so would cause it to expend resources on areas where it is not specialized or experienced. In many of the countries where PFAN operates, it also would be duplicating work undertaken through other donor- or IFI￾funded projects. Survey respondents indicated that PFAN was successfully addressing some barriers but that both policy and financial barriers remain important limitations on increased CE investment. Although PFAN is not designed to address directly all these barriers, they affect implications for future programming. Although the basis for respondents indicate that financial barriers remain the main impediment to increased CE investment. The implications are that PFAN will have relatively greater impact (there will be more right projects) where these barriers have been or are being addressed by other p track record but has implications for USAID in choosing where and when to deploy PFAN. PFAN is expected generally to have relatively greater impact where policy and financial barriers have or are being addressed than where they are not. Respondents have indicated areas where PFAN can add value beyond what it is currently providing, as discussed in the body of the report, although tradeoffs could be evident in broadening focus, especially if the plan is for PFAN to become more self-supporting. The results of this survey and other studies point to serious market imperfections in the policy, financial, and business prongs. Based on historical examples and the presence of other donor-funded programs that could serve as competition reducing demand from the better CE projects, moving to a fully self-supporting role likely will mean that PFAN will reach far fewer projects and much smaller, more marginal projects. Several important recommendations of this evaluation are: USAID should consider funding expanded PFAN data collection and monitoring and evaluation. This would allow PFAN to develop an appropriate performance monitoring plan, collect data including during closure proper, conduct routine exit surveys, and evaluate key areas of difficulty including procedures. developing context remain relevant, USAID may wish to consider revising or adding indicators in the future that focus on broadening access. PFAN should consider first addressing the remaining barriers within the clean energy business area such as addressing negotiations and assistance for financial closure to reduce time to closure and improve reliability of data. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network xii While there is already a high level of coordination among PFAN operational partners, PFAN might consider expanding coordination efforts with their country-specific programs or projects that are addressing barriers in policy and finance, which could reduce costs for other FIs and PFAN, mitigate risks, and increase the number of FIs eligible for work with PFAN. PFAN should not attempt to directly address policy or financial barriers, since business barriers remain a formidable obstacle and there are many other donor programs addressing those other barriers. In order to better use PFAN as a vehicle to promote gender equality and inclusion, the easiest solution is for USAID to fund additional coaching services for woman developers. When considering promoting greater inclusion of female project developers, USAID should weigh the tradeoffs on commercial viability and the impact that is expected to have on CE investments. USAID may consider altering its collaborative agreement with PFAN with instructions on including gender, having gender-related targets (developed together with PFAN), and recommending considerations and targets. dTS has given several ways that this might be accomplished without accepting marginally viable projects and thus affecting It is believed that PFAN is in a better position to recommend to USAID which of these is best suited for results. Consideration should be given by donors and stakeholders to PFAN moving to a partially self-supporting model as an alternative to moving to a fully self-supporting model to better maintain effectiveness and achieve intended outcomes. Doing so would require continued areas and expanding the fee model for some services, while CTI determines appropriate pricing to avoid becoming uncompetitive. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 1 1 1.1 EVALUATION PURPOSE The evaluation of the PFAN activity aims to assess the performance, effectiveness, and sustainability of PFAN and its participants. This evaluation provides information for future programmatic and policy-related decision-making, offers contextual learning for USAID and other involved partners and stakeholders, and demonstrates accountability for resources. Recommendations on specific program elements in the program implementation or as the program transitions to a partially self-supporting or fully self-supporting entity. The evaluation is intended to ascertain whether PFAN has been cost-effective and to determine if cs that make them more likely to achieve financial closure and help CE projects become operational. These findings may assist CTI and USAID in more accurately targeting their assistance. The performance evaluation should elucidate whether developers perceive PFAN to be sufficiently addressing the barriers and challenges they face in gaining access to sources of CE finance and/or how PFAN can address these issues further. The performance evaluation also creates an opportunity to gain greater insight into the longer-term impacts of the mentoring and financing support that PFAN provides. The PFAN activity collects information on project developers related to the anticipated financing, greenhouse gas (GHG) emission reduction benefits, location, and technology type of their projects. One of the purposes of the evaluation was to collect additional information outside of the scope of the cooperative agreement, such as the status of PFAN dent of PFAN, and the co￾benefits that developers and/or their communities receive. The results of the evaluation can be used to adjust the design of the PFAN program within the scope of its mandate, determine the cost￾insights that can be used to showcase this program in USAID and PFAN communications products. 1.2 EVALUATION QUESTIONS To aid in the design process, the evaluation questions have been organized to assess performance, effectiveness, sustainability, and replication. Performance What has been the cost-effectiveness of PFAN in relation to: a Clean energy technology financing; b Establishment and maintenance of lender/developer relationships; c Leveraging private sector resources, such as in-kind services and mentoring; and d Clean energy technology deployment? What are the strengths and weaknesses of the PFAN organizational and partnership model? GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 2 Effectiveness Why have certain PFAN projects been more or less successful in achieving financial closure? All projects admitted to the pipeline remain in the pipeline until reaching closure or until being withdrawn from the pipeline for reasons of project termination and or changed circumstances. This question thus is In what ways and to what extent do project developers credit PFAN for having successfully secured financing? Are there barriers not being addressed or not being addressed effectively by PFAN? i.e., what additional assistance can PFAN offer under its mandate? Sustainability and Replication What is the performance of luding reaching and maintaining operational status, replicating or expanding business, and producing co-benefits for themselves or their community? or implementation, either in its current form or as the program transitions to a self-supporting entity? 2 6 PFAN was initiated by the CTI in cooperation with the UNFCCC EGTT and is supported by a number of private sector companies that finance CE industries. CTI is responsible for coordinating the PFAN program while the International Center for Environmental Technology Transfer (ICETT) is responsible for administering the program. USAID is a partner of the PFAN alliance and provides support to the program through a cooperative agreement. The agreement is currently supported by a $5 million budget and lasts from October 2007 to September 2015. PFAN activities started in 2006 as a pilot project and expanded in 2008. During the project expansion phase, PFAN established regional networks in Latin America, Africa, Asia, and Eastern Europe, as well as dedicated country-specific networks. Currently, the program is funded through various international partners that may also offer support in the form of discounted service rates and fees. Documents reviewed indicate that the intention is for PFAN to become a partially self-sustaining entity, and preparations are underway to facilitate this transition. The projects accepted into the program are showcased on the PFAN website in a project pipeline summary report and tracked through financial closure. As of December 2014, 304 projects had been inducted in the development pipeline, representing $6.7 billion of investment. Forty-nine projects that can be credited to USAID assistance have reached financial closure, raising $561 million. 6 This section is equivalent to and satisfies the GCC M&E contractual requirements to include a section on activity description. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 3 Broaden access to financing for climate-friendly and technology transfer projects in the RE/EE sectors in developing countries and economies in transition; and Secure private sector financing for more Clean Energy and climate-friendly projects to accelerate technology transfer under the UNFCCC. PFAN targeted the lack of access to financing by bridging the gap between financiers and project developers. By providing coaching, PFAN helps project developers create more robust business plans and communicate effectively with potential sources of financing. The first aspect of the program is providing advice and guidance on: Overall project structure; Finance structuring, and sourcing and procuring financing; Technical and engineering issues; and Preparation and presentation of investment proposals. The second aspect of the program consists of connecting potential investors and project developers through investor matchmaking, usually in conjunction with regional workshops where project developers can present their proposals directly to investor members of the PFAN network. The program targets midsized projects in the $1 million to $50 million range although one pipeline project has been valued up to $800 million. In order to qualify for PFAN support, a project must undergo a rigorous selection process based on initial description, proposal, and other relevant information. Projects accepted into the program go through three more stages of review that cover: Project economics and viability; Technical and engineering aspects; and Problem solving and marketing. Each review stage is summarized in a formal written memorandum that provides an analysis of the strengths and weaknesses of the project proposal and suggests next steps for improvement. As stated in the CTI-PFAN Cooperative Agreement for the period 10/2007-9/2015, goals in line with USAID indicators are: Percent of inducted projects reaching closure 20 to 29 percent; Total public and private dollars leveraged by USG for energy infrastructure projects between $72.5 million and $325 million; and Number of commercially and concessionally-financed projects as a result of USG assistance between 11 and 27. support via the: GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 4 Partnership, Methane to Markets: acceleration of the implementation of priority projects by the creation of expanded access to financing; Promotion of sustainable development in developing countries in the areas of CE/RE and EE, achieved by the innovative PFAN approach which is explicitly designed to ensure capacity building (especially in financing) thereby further promoting economic and social stability; In the course of their capacity building efforts the PFAN Consultants will be actively using and promoting the use of the UNFCCC Guidebook on Preparing Technology Transfer Projects for Financing; Promotion and acceleration of the technology transfer process in key areas to help developed and developing countries meet their UNFCCC obligations and to mitiga 2.1 THE PFAN APPROACH catalyze clean energy investment, as shown in Figure I. The three prongs are business, finance, and policy. 7 In essence PFAN helps the developer bridge this gap by providing technical assistance and introducing investors. 7 Figure 1: PFAN Model GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 5 Financial barriers arise because of imperfect capital markets and result in suboptimal investment. Financial barriers include: Loan tenures that do not match the economic life of assets The risk perceived by financial entities is greater than the actual risk Rules of capital repatriation Lack of access to credit for consumers High Transactions costs Financial entities unfamiliar with clean energy Small deal size Balance sheet financing when most SME developers have no balance sheet Collateral requirements in excess of the project cost. Policy barriers arise because either the existence of certain policies or the lack of enabling frameworks for clean energy result in suboptimal investment in clean energy, and usually greater investment in fossil fuel based energy. Policy barriers include: Subsidies for fossil fuels Subsidized grid extension Tax policies that favor extractive industries Government ownership of conventional energy sources Trade barriers (unequal treatment in custom duties, etc.) PFAN focuses primarily on CE business and projects, one of the three areas where efforts are needed to overcome the barriers to CE finance. It does this by providing assistance to CE business and projects and then introducing prescreened projects to interested financial entities. PFAN provides commercial and technical services to assist project developers to: Develop and refine project concepts and business plans; Review project structuring; Review technical and engineering aspects; Make investor presentations; Advise on business growth strategy; and, Meet investors, including at investor fora where PFAN projects that excel in business plan competitions are presented to potential financiers. In limited cases, PFAN does provide financial assistance for feasibility and technical studies. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 6 3 8 3.1 RESEARCH DESIGN9 The evaluation uses a convergent parallel mixed method design that allows for the collection of qualitative and quantitative data concurrently as well as analysis of the two data sets separately. The quantitative instruments are needed for obtaining and analyzing generalizable statistical data. Qualitative instruments are well suited for explaining processes and impacts. Together, the two parts provide both analytical and explanatory power, while validating the findings through triangulation of data from multiple sources. Given the challenges with data collection as well as the importance of learning lessons, as opposed to merely measuring differences, this mixed method approach is considered both appropriate and essential. 3.2 DATA COLLECTION METHODS The evaluation design matrix found in Appendix IV provides an overview of the evaluation questions, measures and indicators, methods, and data sources required to answer those questions. The list of indicators provides the background context on the types of information that the evaluation collects. 3.2.1 Four data sources were used to collect the cross-sectional data. The two primary sources were qualitative KIIs centered on subjective data and a confidential, primarily quantitative, electronic survey open to most PFAN stakeholders. The secondary sources included administrative and institutional data in documents provided to dTS by either CTI or USAID that address the organization and administration of PFAN. The reports, and publicly available data. 3.2.2 Using SurveyMonkey, an online software tool, the evaluation team developed a survey for project developers,10 coaches, operational stakeholders, resource partners, and financial institution (FI) representatives.11 piped to direct respondents to specific subsequent questions based on their answers, and customized to gather information specific to each of the subgroups. At the end of the survey instrument, respondents could choose to participate in a KII. In-depth KIIs were conducted when possible to provide further insight into collected quantitative data in terms of attitudes and behaviors. The KIIs were used to provide additional detail in areas that could not be 8 This section is equivalent to and satisfies the contractual requirements to include a section on research design and evaluation methods. 9 What follows is a condensed version of this section; for the full text, see Annex IV. 10 Project developers include individuals, companies, communities, and NGOs. 11 Coaches are individuals who provide business coaching to developers on a cost-share basis. Resource partners are organizations, such as non-governmental organizations or government-owned CE entities, which have a vested interest in promoting CE and see PFAN as assisting them in achieving their objectives. Operational stakeholders are donors, implementers, country coordinators, and other PFAN project personnel. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 7 adequately dealt with in the survey. The KII guide consisted of general questions for all informants and distinct and targeted interview guides for each of the sub-groups. 3.2.3 An analysis map was developed that linked each evaluation question and/or sub-question to the corresponding survey question(s) for quantitative data and to the corresponding KII guide question(s) for related qualitative data, in order to efficiently organize data and track correlations across the population as a whole. software to analyze quantitative data. Qualitative data provided as part of the online survey was used primarily for descriptive, anecdotal information and further analysis, depending on the length of the narrative. Qualitative data collected as part of KIIs were analyzed in depth for emergent patterns of perceptions, representations, portrayals of utility and effectiveness, and aspects that suggest potential sustainability. 3.2.4 For the survey, the evaluation team sent out 687 emails that included the survey invitation and instructions, with 674 invitations delivered successfully, generating a total of 156 responses. Subpopulations surveyed include project developers, FIs, coaches, operational stakeholders, and resource partners. Appendix IV includes detailed information on and analysis of the demographics of those surveyed by subpopulation, response rates, and motivation for including in the survey. Seventy-one persons were interviewed. There are three broad categories of project developers developers with closed projects, developers still in the pipeline, and developers that were deemed not qualified or are not selected for funding reasons very early on and excluded from PFAN assistance. Of the developers that responded to the request for interviews, two had reached financial closure, and a third was listed in the pipeline as having reached financial closure but funds had not yet been disbursed. Only five representatives of FIs responded to the survey, and all were involved in the CE finance business prior to participating in PFAN. Eleven coaches out of 46, representing 24 percent of the PFAN coaching population, participated in the survey. Operational stakeholders included representatives from USAID, CTI, ICEETT, United States Department of State, the Renewable Energy and Energy Efficiency Partnership, the Energy and Climate Partnership of the Americas, International Development Research Centre, and other PFAN managerial and funding stakeholders. Representatives from these organizations were interviewed in order to obtain insights into the multi-lateral interagency partnership, management and operations, and program design and execution. Resource partners tend to be government agencies in developing countries that have been set up to support CE investments in their countries. They help to ensure that efforts are aligned with governments and project objectives. They provide local context and support in terms of investment climate and CE technology needs and gaps. 3.3 DATA QUALITY methodologies in order to meet expectations for data quality.12 promote data quality include pre-testing of survey instruments, performing multiple mock KIIs, training and 12 Expectations for data quality are specified in the USAID Automated Directives System 203, Assessing and Learning. These are namely validity, integrity, reliability, precision, and timeliness. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 8 close supervision of enumerators as needed, data entry controls, qualitative data recording, summarizing, transcribing, and use of mixed methods. 3.4 EVALUATION LIMITATIONS Independent Verification. The evaluation team was not able to independently verify the population characteristics within each stratum, as some individuals may have been included in more than one category and so many not be truly representative of the population. Precision of the estimates for online surveys may also be an issue. The specific findings of the evaluation may not apply to the diverse PFAN population. However, since a mixed-methods approach is used, a degree of certainty can be attached to more general findings and recommendations presented to CTI and USAID because they will be supported through more than one data source. Many of the metrics in this report will not accurately reflect the long-term potential since PFAN is still an on-going project. Two examples of this discrepancy include the following. Cost-effectiveness. The cost effectiveness of the activity, in terms of investment leveraged, MWs, and GHGs avoided, is provisional in that it measures investment leveraged and MWs at reported financial close13, not at project operation. GHGs avoided are based ex ante on assumptions for capacity factors and using default emission factors, and do not represent verified ex post reductions. Additional project benefits not currently represented in the pipeline data could result from scale up and operation, resulting in an under Sustainability and Replicability. It is difficult to measure sustainability and replicability in a reliable manner in the short run. Many of these projects have been undertaken by new businesses and it is hard to determine now whether they will weather the course of time and be sustainable. Measuring replicability is complicated by the fact that it takes time to build capacity, change minds, penetrate markets, and have an impact on business and financial institutions. Conflicts of Interest. Discussion of conflicts of interest can be found in Appendix VII. 4 4.1 GENERAL PROGRAM 4.1.1 Conclusion 1. PFAN has exceeded the targets for the USAID Cooperative Agreement and modifications. 13 PFAN does not report at financial closure but rather when their assistance to project developers end; when there is an agreement to finance but not when actual financing is consummated. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 9 USAID began with three quantifiable targets with respect to its assistance to PFAN, and with time these have increased in amount. The summary outcomes or expectations upon which USAID assistance is predicated are shown below in Table 2. Table 2: USAID PFAN Expected Outcomes SUMMARY OF OUTCOMES Target Low Range Target High Range Achieved to Date # Total Projects inducted into Pipeline 54 92 304 # Total Projects reaching investor maturity 30 58 119 # Total Projects reaching Financial Close 11 27 49 % Project Developers reaching closure14 20% 29% 16.10% Total Financing Leveraged $72,500,000 $325,000,000 $561,539,200 Aggregate Leverage Effect (1 :XX) 41 186 112.31 Source: EEM-A-00-08-00005, Modification 6. Subsequent modifications did not change the targets. It should be noted that PFAN reports financial data that can only be attributed directly to their efforts. Thus, if a small developer gathers funds from friends or family or provides in-kind contributions then these are not reported. Similarly, if the project brings in funds from a source outside of the financial institution that is working with PFAN, then this will not be counted. It is unclear how large this additional investment might be. For the purposes of this evaluation, it is noted that the financial results will be undercounted. 4.1.2 Conclusion 2. stated in the collaboration agreement with USAID, over another putting the objective of pr PFAN's stated primary objectives are to: sectors in developing countries and economies in transition; and get more RE and climate friendly projects financed in the private sector and thereby to accelerate 15 whose developers would not otherwise have access to, or resources to pay for commercial consulting services. These sorts of projects were being largely 16 14 This is the percentage of projects inducted into the pipeline that reach financial closure. Roughly 20 percent of the projects inducted so far have reached financial closure. 15 USAID PFAN Collaborative Agreement, pages 14 and 15. 16 USAID PFAN Collaborative Agreement, pages 14. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 10 Together the objectives and the statement above might suggest balanced performance criteria focusing on both broadening access as well as getting more projects financed. However, the indicators from the cooperative agreement with USAID are focused on total financing and total projects; thus, they more strongly support the second objective. Broadening access would mean putting an emphasis on cultivating more developers. But the criteria focuses on projects, not developers or entities. Note that an individual developer/entity could have more th services are raising the capabilities of developers, a strategy of serial investments (allowing developers to have more than one project in the PFAN pipeline) could reduce costs per project, especially if capacity building services are reduced after the first project. The survey results provide some support that serial investment is taking place. In some cases, PFAN is assisting project developers that: (1) already had other CE projects, and (2) have already had successful experience accessing finance.17 Additionally, slightly more than 16 percent of developers reported that they had two or more projects in the PFAN pipeline. Including developers who already have successfully accessed financing for CE projects is not in violation of the agreement between USAID and PFAN. It does, however, raise questions regarding how best to meet rther, this tension highlights that the objective to increase the number of projects financed can compete with the objective to broaden access. Broadening access to finance might also include increasing the number of financial entities that are not just stakeholders of PFAN but also participating in financing PFAN projects. Additionally, this might suggest increasing the involvement of commercial banks in PFAN. The survey listed the low participation of finance entities at investor fora as an issue. Others respondents indicated that the general lack of investor contact was a major barrier not being addressed by PFAN. 4.1.3 Conclusion 3. One of the hypotheses underlying the PFAN approach remains untested and results of the survey present contradictory views. Understanding this assistance to PFAN; where it gives, how much, in conjunction with what other conditions and the results it can expect. no shortage of private sector money for the right projects. Yet financial barriers are, according to survey respondents, the main impediment to increased CE small and that both policy and financial barriers are important limitations on increased CE investment. 17 In some cases, the developer has acquired partial financing for the project prior to joining PFAN and is looking for . In other cases, the developer has had an earlier successful CE project that was financed. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 11 The survey asked the following question, framed as a hypothesis, to operational stakeholders because they would have access to and understand the genesis of PFAN, whereas other respondents would not. An early was not the availability of finance . ancing is available, and the remaining 13 percent answered that they did not know. Yet these same respondents, when asked to identify the major barriers to CE investment, ranked financial and policy barriers as the most important. How can it be that there survey respondents indicate that financial barriers are a major impediment to increased CE investment? This evaluation had neither the resources nor access to data to conduct a thorough analysis of what underlies this seeming contradiction. However, we believe that it can be partly explained by the following factors. First, it hinges critically on ho What defines a right project and is that a small number of projects or a large number of projects? The term right is ambiguous. The fact that donors are interested in subsidizing services to bring projects to financial closure indicates that there is a difference between the as defined by donors and the number defined by investors. The expectations of different PFAN participants differ on the number of right projects. Similarly, it will differ the second hypothesis: "While there is a ready supply of good potential projects, there is a shortage of good project financing proposals meeting the standards and criteria of the international financing community." This would lead to the conclusion that there is a ready supply of projects that can be made right through PFAN assistance. But again, how big is ready? market conditions) and what is the right number for donors. There is a divergence between the social optimum for donors and the market optimum in any PFAN countries. And this divergence also helps to explain why the supply of finance might not be sufficient for the right number of projects: different Additionally, PFAN acknowledges working primarily through equity investors. The capacity of those investors is limited. While they may have ample funds compared to the opportunities that reach their attention, their funds available are quite small compared to the market demand for CE investment. Second, the number of right projects is a function of many factors including: market size (the number of right projects will be higher in India compared to Malawi, with the familiar ceteris paribus conditions); the presence of fossil fuel subsidies and rural electrification by grid extension (the number of projects will be higher in Cambodia compared to another country where only these differences prevail); the number of alternatives; the current level of CE penetration; and the degree of local funding sources. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 12 The number of right projects hinges critically upon the presence of other barriers including policy and financial barriers. The following are important financial barriers that are present in PFAN countries: The number of right projects may mean the number of projects that investor would fund in the presence of other barriers. For example, investors have been willing to finance rice gasifiers in Vietnam, but far fewer than the number financed by investors in Cambodia over the same time period. This was due to a variety of factors including the lack of subsidies for conventional energy in Cambodia, the larger market of Vietnam and the difference in banking laws. Similarly, if the number of right projects is those that the investors will fund in the presence of other barriers, then it is quite likely that developers, resource partners and donors will believe the number of right projects is higher. In many developing countries, banks have loanable funds but the conditions are not right for investment. a For example, collateral requirements can be over 200%. b In most countries loan tenure is quite short and does not match the life of the investment. c CE is not an option (bank does not loan in that area). d Transaction costs are too high for large scale investors. e investments by companies and end-users; f Limited access to long-term finance by corporations and SMEs; g Unfamiliarity of financial intermediaries (FIs) with green growth and climate change financing business, 18 Survey respondents indicated that too few investors and too few different kinds of investors were present at fora. Early stage finance19 is lacking on insufficient. 4.2 PERFORMANCE This section deals primarily with cost-effectiveness. Specifically, it addresses the following question and sub￾questions. Evaluation Question 1. What has been the cost-effectiveness of PFAN in relation to: a CE technology financing; 18 http://www.ifc.org/wps/wcm/connect/32102a804fd330648d60ef0098cb14b9/SEF-Factsheet-Final.pdf?MOD=AJPERES 19 Financing in the very early stage of development including finance for concept development ore refinement, engineering, environmental and other feasibility studies, and permitting. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 13 b Establishment and maintenance of lender/developer relationships; c Leveraging private sector resources, such as in kind services and coaching; and d CE technology deployment? Evaluation Question 2. What are the strengths and weaknesses of the PFAN organizational and partnership model? 4.2.1 Examples of cost-effectiveness calculations include dollars of technical assistance (TA) spent per MW of CE or the dollars of TA spent per ton of CO2e avoided. By itself, a single measure tells little about whether $4 per ton of CO2e avoided is cost-effective. To know this, calculations must be compared to a baseline or to other projects. Baseline data were not available in this instance. Thus, the evaluation team developed measures from publicly available data on other donor-funded projects and their work with some CE projects. Performance metrics for projects or programs identified for comparative purposes are presented in Table 3. Few donor-funded programs are designed similarly to PFAN. Most of these projects carry some form of projects focus on the finance prong in the PFAN model but also provide limited developer services as they are aimed at strengthening the business prong but also include direct loans and sometimes grants. Yet, comparisons are valid because ultimately the examination focuses on money spent and the outcome. These different projects reflect different models, markets, and approaches. The program with the closest approach and public data was E+CO, which was supported by USAID, IFC, the Rockefeller Foundation, and others. E+CO focused at the project- or developer-level by providing many of the same services that PFAN offers and also including limited finance. It offered both services and capital to companies, staying with entrepreneurs throughout the investment, operations, and maturity process. However, it targeted much smaller entities and transactions. I 20) in China, the Bulgaria Energy Efficiency Fund (BEEF21 (GEEP). E+CO managed the IFC Sustainable Energy Facility.22 20 Implementation Completion and Results of ECP I, 2007, p.12 21 BEEF Implementation and Project Completion report 2010 22 SEF Midterm Evaluation report 2009. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 14 Table 3: Performance Comparison23 Project Watts of capacity per dollar of technical assistance $ of funds mobilized per dollar of technical assistance24 Avoided GHGs per dollar of technical assistance (tCO2e/$)25 E+Co SEF 14.71 6.09 0.35 ECP I NA 20.95 1.00 BEEF NA 39.67 1.12 GEEP NA 6.77 2.5 PFAN 77.5 112.31 4.20 The measures in the table above are based on projects in which donor funding and donor-funded activity has been completed. The exception is SEF, for which the measures represent a mid-term evaluation. USG assistance to PFAN is ongoing. Many of the projects currently in the pipeline will reach closure and will improve the measures in Table 3 relative to the other projects. The MWs and GHGs may be overestimated because they are projected, not actuals, and may not occur. They may be underrepresented because they do not track future projects that occurred as a result of the TA. 4.2.2 Conclusion 4. -efficient vehicle. USAID spent less on technical assistance per projected MW of capacity than other funders, and USAID funds supported more tons of CO2e avoided per dollar of technical assistance than all other programs. As of December 2014, USAID had obligated almost $5 million for PFAN, and $3.93 million had been expended. As a result of the assistance provided to developers through this funding, 49 projects have closed that be credited to USAID26. These projects are credited annually by PFAN with 388 MW of electricity￾generating capacity, saving almost 100 Gigawatt hours (GWh) of generation per year via EE projects, and avoiding 1.8 million tons of CO2e annually. These benefits do not include all of the 49 projects27. In fact, greater CE benefits have been generated but are not included in these measures of supply (MW, GWh), such as those resulting from the use of biofuels and the generation of steam. A total of 15 closed projects, 23 These numbers are prior to actual operation for PFAN and for many of the other projects and, therefore, assume that projects will achieve the results that they predict they will. 24 Technical Assistance spending was chosen rather than donor contribution to make the programs comparable in e TA was used on these other projects. 25 These are life of project benefits and so PFAN was converted assuming a very conservative ten years for benefits to endure. 26 One project closed before USAID funding began and is excluded from these calculations. 27 Fifteen projects do not report MWs and 10 do not report GHGs avoided. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 15 almost a third of those reaching closure, create inaccuracy in the estimates produced here by having provided no measure of MW of generating capacity added for the calculations of cost effectiveness. Similarly, ten of the closed projects provided no measure of CO2e avoided.28 Thus, these routine measures of CE performance likely understate the true impact of PFAN in terms of measures based on cost per MW and CO2e avoided. Further, the total MW of generating capacity claimed by the 49 closed projects supported by USAID assistance29 totals 387.9 MW. ost of TA per MW has been $12,889. This equates to approximate 1% of the average total cost of installation per MW of CE30. Generally, the lowest cost for CE generation technologies averages $1.2 million per MW but can be substantially higher depending on the technology and resource availability.31 MT of CO2e avoided per $1 = 4.2 4.2.3 As data were not provided on an activity basis, dTS cannot determine the cost effectiveness of the isolated element of PFAN that involves establishment and maintenance of relationships between lenders and developers. However, some general comments can be made on this issue, which are not based on data. Some resources must be dedicated to contacting prospective participants and holding fora and other events where project developers and lenders meet and eventually conclude business. However, the overwhelming reason that lenders and developers will maintain relationships with PFAN is that (a) stakeholders continue to see potential and (b) the expected benefits of participating outweigh the costs. Maintenance of the relationships is then a function of all activities and thus all costs. If projects are not well designed and presented, developers and financiers will not reach closure. If closures are not reached in sufficient time and quantity to justify costs, either at this point or in the future, relationships would not be maintained. 304 projects at varying stages with 49 reaching financial closure as of this evaluation. Table 4 provides calculations related to cost effectiveness from the perspective of FIs, which shows one measure of the effectiveness of establishing and maintaining lender-developer relationships. Table 4: Cost Effectiveness of Developer-Investor Relationships Cost per Project Spending Per Financial Institution Prospective Investment per $ Spent $16,382.01 $73,478.15 $1,337.12 28 The data identified is simply not available because of the nature of the projects concerned eg rural electrification or biofuels projects where there is no electricity generation and projects for which there is no established methodology for calculating CO2e mitigation potential. 29 While a total of 50 projects have closed, one reached closing before USAID funding was provided. They have been excluded from the calculations. 30 Based on dTS estimates of the cost of CE from various reports including the Renewables Global Status Report and estimates by IRENA. 31 $1.2 million per MW is based on a review of studies for all clean energy technologies in the size range of PFAN. $1.2 million is for run-of-river hydro from the Canadian RetScreen. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 16 4.2.4 Conclusion 5. USAID funds leverage $112.31 per dollar of USAID funding, significantly more money per dollar than other similar programs. PFAN is designed to leverage both financial and coaching or resources for coaching. Financial entities provide funds for equity or loans, and coaches provide in-kind contributions of time to assist developers. Based on the project pipeline data, total financial sector resources leveraged amounted to $561,539,200 In-kind contributions are made by coaches and other private sector resources. Data was abstracted from the PFAN quarterly reports and amounts to $2.97 million to date. This is in tandem with USAID funds expended to date and it is anticipated to continue to match USAID on a one to one basis. Although an attempt was made to verify this through the survey information on the time and in-kind value provided by coaches, unfortunately, only one coach answered the contribution questions and that answer was deemed inapplicable. In terms of financial sector resources leveraged, USAID funding of almost $5 million to PFAN have leveraged $112.31 per dollar contributed. This appears to be quite large particularly in comparison to the results presented in Table 3: Performance Comparison. This difference between PFAN and comparison programs would be expected to narrow with time because those chosen for comparison (with the exception of the E+CO SEF) were designed to strengthen the capacity of financial entities or businesses that will continue to invest in clean energy projects well beyond the end of donor funding. This continued investment has in fact -I work with energy service companies in China. Nonetheless, PFAN would be expected to leverage more funds than the other comparison programs. 4.2.5 Conclusion 6. To date, the one wind power project represented in the pipeline has been deployed at far less cost to USAID per MW than any other technology that PFAN is assisting and has the highest ton-per￾dollar yield of any form of USAID assistance. This is an anomaly based on the fact that a large amount of the funds for this project came from a non-PFAN source. Cost-effectiveness of CE financing addresses the overall PFAN program, while this sub-section focuses on the individual technologies deployed through PFAN. A limitation of this analysis is that the data are not publically available to allow comparison with the programs shown in Table 3: Performance Comparison. Similarly, the records provided to dTS are not sufficient to break out costs among the various technologies, nor are all costs separable without some arbitrary assignment. For example, fora are not technology specific and benefit all technologies and projects involved equally. Thus, this analysis assumes the most equitable approach to allocating costs across technologies is to allocate costs evenly on a per-project basis, with each project or PFAN client being accorded the same use of resources. Table 5 presents the measures of cost￾effectiveness by technology that result from this allocation approach along with other metrics. The two measures reviewed are USAID cost per MW of CE and tons of CO2e avoided per dollar of USAID spending. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 17 Table 5: Cost Effectiveness by Technology Technology # Closed Projects MW Capacity tCO2e/yr Total Investment Cost of TA per MW32 tCO2e/$ of TA33 Hydro 8 70.6 170,780 178,910,000 $11,562.70 2.09 Biomass 9 79.1 517,220 135,450,000 $11,610.21 5.63 W2E 3 14.5 128,300 62,500,000 $21,111.89 4.19 Wind 1 140 840,000 2,800,000 $728.86 82.32 Biofuels 4 1.5 56,830 5,150,000 $272,108.84 1.39 Solar 5 50 92,516 70,950,000 $10,204.08 1.81 EE 8 21 82,369 40,360,000 $20,459.31 1.01 Biogas 9 12.95 224,353 49,954,000 $70,916.40 2.44 Clean Transport 2 0 37,980 15,465,200 1.86 Total 49 388 2150348 561,539,200 $12,895.21 4.30 Source: PFAN Pipeline data Putting aside the wind project, solar projects have proven to be by far the most cost-effective technology deployed through PFAN. This conclusion does not indicate that PFAN should target wind over other technologies. Technology choice is driven by many factors, and some areas simply are not suitable for wind. Moreover, this does not represent the entire cost of the technology but only that amount raised by PFAN. These results are counterintuitive for energy efficiency. EE projects worldwide generally represent the low hanging fruit and routinely result in lower cost-per-unit of energy than do supply projects. The reason that PFAN results do not reflect this can be explained partly by the fact that 50 percent of the EE projects have no estimate for energy saved, and 38 percent indicate no CO2e avoided.34 The reason that hydro deployment shows a low CO2e avoided per dollar of USAID funding may be that many of these were off￾grid projects, and the CO2e avoided is often not registered because the carbon avoided is black carbon.35 32 USAID cost spread evenly over all projects. 33 Lifetime CO2e avoided / USAID cost per technology. This is based upon the assumption that project benefits last ten years. Clearly this is low for some technologies such as hydro and wind. However, even using this conservative measure, benefits are significant. 34 Given the country and industry where these EE technologies were deployed, it is clear that CO2e was avoided, even if PFAN accounting did not capture it as a result. 35 Black carbon is soot, and, although black carbon is often not registered, recent scientific work finds double the impact of black carbon on global warming relative to carbon dioxide. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 18 4.2.6 Conclusion 7. Stakeholders rated the PFAN operational structure as effective but still weak in areas. In particular, the presence of donors is seen as a great strength while project areas (e.g., specific technology or country) was seen as a weakness, as such bias may affect funding decisions for reasons other than cost￾Each stakeholder group understands the PFAN organization model differently and thus different questions were asked of each group. Operational stakeholders were anticipated to be the only stakeholders that would fully understand the PFAN organizational and partnership model based on their background, exposure to the general problem, and experience. The interviews confirmed this. Coaches had a limited technical assistance and capacity building to bring the deal to closure was their basis for response. 4.2.7 to make strong assertions about the performance of PFAN with respect to gender considerations. Survey responses have been aggregated by gender where meaningful and practical and reported below. 4.3 OVERALL EFFECTIVENESS 4.3.1 Two questions were asked to get at the overall effectiveness of PFAN. The results, as shown in Table 6: PFAN Structural Effectiveness, are somewhat mixed. Seventy-two percent of coaches surveyed think oderately to completely effective. A key element of this structure is a public-private partnership. Sixty-eight percent of coaches believe that PFAN would be effective if run as a commercial international assistance project. Table 6: PFAN Structural Effectiveness Structural Effectiveness Rating by Respondents (in percent) Level of Effectiveness How effective is the three-part36 operational structure of PFAN? How effective would PFAN be if it were conducted like a commercial international assistance project? # of Respondents n = 19 n = 20 Completely ineffective 0 0 Moderately ineffective 16.67 15.79 36 The three parts are Alliance Partners like USAID and ICETT for administration and finance, and PPL for implementation GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 19 Moderately effective 44.44 36.84 Completely effective 27.78 31.58 Don't know 11.11 15.79 Source: PFAN Survey 4.3.2 capacity building. For presentational purposes, responses were assumed to be ordinal37 as 1. Table 7, ordered by average rating, provides the individual responses by category as well as an average rating that results with a score of 1, meaning that 100 percent of the respondents rated their level of satisfaction with the coaching services as completely satisfied. A score of 4 would indicate that all respondents were completely dissatisfied. On average, developers were satisfied with all technical assistance and capacity building, but they were most satisfied with assistance on the business plan and least satisfied with the role of technical assistance and capacity building in introduction to investors. This may reflect the fact that most of the developers answering the survey had not reached financial closure. Table 7: Project Developer Satisfaction with technical assistance and capacity building Answer Options Completely Dissatisfied Moderately Dissatisfied Moderately Satisfied Completely Satisfied Response Count Business plan 8.5% 5.1% 30.5% 55.9% 59 Coaching on overall project structure 7.7% 5.8% 42.3% 44.2% 52 Project datasheet 8.0% 12.0% 36.0% 44.0% 50 Coaching on commercial/ finance aspects 9.3% 16.7% 33.3% 40.7% 54 Investor presentation 14.3% 14.3% 28.6% 38.8% 49 Investor forum 6.8% 31.8% 27.3% 34.1% 44 Advice/guidance on technical aspects 16.3% 14.0% 46.5% 23.3% 43 Introduction to investors 23.1% 28.2% 28.2% 20.5% 39 37 As there is no formal hypothesis testing across time or versus an expectation, the assumption of ordinality does not affect the findings. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 20 4.3.3 Given that PFAN is based on a premise of the availability of financing and the need for solid project proposals to use this funding, PFAN financial partners were asked to rank by value the services PFAN provides to project developers. Table 8 presents the ranking of PFAN services by FIs as they see the services helping developers prepare for financing. FIs value the coaching on commercial and financial aspects of projects the most and advice on the technical/engineering aspects the least. Table 8: Financial Partners Value of PFAN Services As a financial partner of PFAN, which of these services to prepare project developers do you value the most? Rank these with 1 being the most important variable and 8 the least important. Answer Options Average Rating by FIs Response Count Coaching/mentoring on commercial/finance aspects 2.40 5 Introduction to investors 3.40 5 Business plan 3.60 5 Investor forum 4.40 5 Coaching/mentoring on overall project structure 4.50 4 38 Investor presentation 4.60 5 Project datasheet 5.00 5 Advice/guidance on technical/engineering aspects 7.50 4 Number of respondents 5 4.4 STRENGTHS AND WEAKNESSES Many of the questions asked in the survey and the interviews are designed to be closely related. Responses were categorized into major areas where possible. The top responses among survey respondents39 about ent their project to potential investors. Investor Relations: The fact that PFAN has activities related to investor relations generally is seen as a strength in that there is an opportunity for developers to showcase their projects and to meet investors. However, performance in making the connection between investors and project developers is seen as somewhat of a weakness. 38 Respondents left some items blank. 39 This excludes resource partners since they would not have direct knowledge of this in relation to PFAN. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 21 -ended responses and interviews, were: PFAN does not sufficiently address access to financiers, including the number and type of financiers such as venture capital and mezzanine finance, and provides limited access to early-stage finance. PFAN does not address policy areas. PFAN advisory services need attention in areas such as more screening of potential developers and assistance with the process of financial closing following the investor forum. Another interesting response from several of the operational partner respondents was that the goals of funding partners may lead PFAN to focus on projects to satisfy funding partner targets rather than projects that may expand the overall PFAN footprint. 4.5 EFFECTIVENESS Under the rubric of effectiveness, the evaluation sought to answer three principal questions: Evaluation Question 3. Why have certain PFAN projects been more or less successful in the speed of achieving financial closure? Evaluation Question 4. In what ways and to what extent do project developers credit PFAN for their successfully securing financing? Evaluation Question 5. Are there barriers not being addressed or not being addressed effectively by PFAN? i.e., what additional assistance can PFAN offer under its mandate? In addition, this section covers general issues required to be considered under any USAID-contracted evaluation, such as gender inclusion. 4.5.1 Conclusion 8. There is an apparent relationship between the time required to reach financial closure and project risk. However, there is insufficient data to draw any meaningful relationships that could inform donors or PFAN. One of the original evaluation questions sought to address why some PFAN-supported projects were successful in reaching closure while others were not. However, CTI informed dTS that once a project is accepted in the pipeline, it will stay there until it reaches closure or until it is removed from the pipeline for reasons of changed circumstances. Instead, the evaluation team sought to address the question of why some projects reach closure more quickly than others. The average project has reached financial closure almost nine months after being inducted in the PFAN pipeline. While the most effective method for determining those factors responsible for speedy closures would have been detailed analysis of individual project documents, appropriate data were not readily available and the scope of the evaluation was not appropriate to undertake this data collection from primary sources. Thus, a basic statistical analysis of pipeline data from PFAN was undertaken to understand the role of potential risk of the project with time to closure. The analysis constituted a simple linear regression of the time to closure of projects in the pipeline as the dependent variable, or outcome, against a single independent, or explanatory, variable that could be gleaned from the data available. That variable was GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 22 developed as a binary, or indicator, variable that represents a qualitative measure of repayment risk. The indicator measures 0 if the risk appears low and 1 if high. The equation that results is presented below. Let Yi= 0 + 1Xi + i, where Yi = time to closure of the ith project measured in days, Xi = a qualitative assessment of risk of the ith project, 0 = x-axis intercept, 1 = coefficient on X, and I = error term The coefficient of the intercept terms is 354 (p<0.001), which represents the predicted days to closure. The coefficient on the explanatory variable, X, is - 152.7 days (p=0.0013), which indicates that projects that have reduced risk are predicted to close 153 days, almost a half year, earlier than those that do not. The R￾square value of 0.2 indicates that the explanatory variable of subjective risk alone explains about 20 percent of the variation in closing.40 While the intercept and explanatory variable are strongly significant, the low correlation coefficient predictably indicates that other forces are at play that cannot be measured at this time given the limited data available. The only apparent relationship based on data made available was between time to closure and a binary variable representing the degree to which the investor can understand repayment potential. For example, projects connecting to the national grid with a power purchase agreement appear to have closed more quickly than those that are off-grid or do not have a buyer whose credit characteristics are easy to ascertain. Similarly, EE projects, where the developer was the entity accruing the savings and has a proven track record, are easier to assess in terms of risk than are projects where either the developer or the buyer does not have an established track record. Another factor that may be in play in determining the speed of closure is the extent to which the developer had been successful in obtaining finance before PFAN or, perhaps, had tried and failed, as opposed to developers that had no previous experience. The team wanted to test that relationship but only two of the 50 closed developers completed the survey and, thus, there was insufficient data for analysis. 4.5.2 Conclusion 9. project developers who reached closure answered this question, with one indicating they could and one that they could not have reached financial closure without PFAN assistance. However, developers rated the coaching service they received in coaching services are important to financial closure. As discussed above, one of the main concepts behind the PFAN model is that developers need coaching services in order to bring their proposal up to standard for financing. Developers were asked to rate the services they received. 40 The overall F = 11.93 with a p=0.0015. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 23 Seventy-four percent of developers sought financing prior to coming to PFAN, and 63 percent of those were not successful in obtaining finance. When asked the reasons they could not secure funding, the four most cited reasons were: Lender/investor not familiar with CE; Collateral requirements too high; Lender/investor considered risk too high; and/or Could not complete application to investor/lender satisfaction. Almost two- ces. Each of these reasons, shown in Figure 2, was listed as a barrier to increased CE investment and as one the barriers that PFAN helped them to overcome. Coaches and PFAN provide a variety of services ranging from assistance in project structuring to the investor forum. Overwhelmingly, developers find the assistance effective, as shown in Table 9: Project Satisfaction with Coaching Services. Table 9: Satisfaction with Coaching Services Satisfaction with Coaching Percent Completely ineffective 9.1% Moderately ineffective 9.1% Figure 2: Services that Contribute to Securing Financing 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% Coaching/mentoring on overall project structure Coaching/mentoring on commercial/finance aspects Advice/guidance on technical/engineering Business plan Investor presentation Project datasheet Introduction to Investors Investor Forum What do you consider to be the three most important services offered by PFAN that contributed to your securing clean energy financing for your project? Choose up to three. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 24 Moderately effective 40.0% Completely effective 41.8% Source: PFAN Survey. n=66. Finally, developers were asked similarly to rank the services that contribute to reaching financial closure in order of importance. The most important service in their view was coaching on commercial aspects followed by coaching on the overall project structure and then coaching on the business plan, as shown in Figure 2. However, a different pattern emerged when developers were directly asked if they believe they would have reached financial closure without PFAN. As of December 2014, 50 projects have reached financial closure through PFAN; 49 assistance. Project developers are split evenly between those that believe they would have reached financial closure without PFAN and those that could not have. 4.5.3 Conclusion 10. Overall, PFAN is viewed by all stakeholder groups as successfully addressing the barriers to CE investment. Conclusion 11. Important barriers that PFAN is not addressing successfully generally fall into two areas: policy and financial barriers. This is not to infer that PFAN was supposed to address these barriers or should directly address these barriers. Rather, it states that even with PFAN assistance in the business prong and investment. Conclusion 12. Women want more and expanded coaching services. Introduction As part of the survey, participants were asked to select those barriers (from among the three PFAN 3 and in Table 10) that they considered to be the most important to increased CE investment. Table 10 presents the relative importance of these three prongs as rated by the survey respondents based on their rating of the barriers. Prongs are discussed here as these constitute the underlying basis of the PFAN model. Table 10: Importance of the Three Prongs PFAN Prong Raw Score41 Normalized Score42 41 For example, government subsidizes conventional energy, and lack of favorable government policies would be totaled into the raw score of government. 42 Normalization is a routine procedure that was used to essentially give an equal weighing to each stakeholder group. At times, this is the appropriate way of comparing stakeholder responses when there is a justification to equally weight the input of all stakeholders. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 25 Government 99 3.7 Financial 99 7.0 Business 70 5.5 Note: n=156 This result is heavily influenced by the number of project developers responding to the survey, as there were far more developers responding (111) than all other groups combined (45). When the results are normalized, then a different rank ordering is revealed, with financial barriers constituting the most important, followed by business and lastly government. Within the financial barriers, collateral requirements and lack of bank interest in CE are the two main subcomponents. Other barriers include the lack of early-stage finance, the lack of more advanced financial mechanisms such as mezzanine finance43 and risk adjustment mechanisms, and the large gap between actual and perceived risk. Project developers cited five major barriers within the three prongs. They are outlined in Figure 3. Source: PFAN survey PFAN Effectiveness in Addressing Barriers to Clean Energy Investment The starting point for the analysis of the question of whether PFAN is effectively addressing the barriers to 43 Mezzanine finance is a hybrid of debt and equity financing that is typically used to finance the expansion of existing companies. Mezzanine financing is essentially debt capital that gives the lender the rights to convert to an ownership or equity interest in the company if the loan is not paid back in time and in full. It is generally subordinated to debt provided by senior lenders such as banks and venture capital companies. Source: Investopedia. Figure 3: Major Barriers for Developers 0 10 20 30 40 50 Government subsidizes conventional energy Lack of favorable government policies Collateral requirements too high Banks not interested in clean energy Project Developers not experienced in business aspects GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 26 That is, the survey asks the respondents the barriers in their own estimation. Excluding developers who do not yet have projects admitted to the PFAN pipeline, the results were moderately positive that PFAN is successfully addressing the barriers to CE investment (Table 11) with 67 percent of respondents indicating Only 34 percent of the group that was not accepted into PFAN thought that PFAN is successfully addressing the barriers. Table 11: Is PFAN Successfully Addressing Barriers to Clean Energy? Stakeholder Yes No Total All Respondents (n=88) 63.4% 36.6% Developer (n=65) 66.2% 33.8% Other (n=23) 56.5% 43.5% Developers Not Accepted (not included in Total) 34.0% 66.0% FIs appear to be the most optimistic that PFAN is successfully addressing the barriers. This may be reflective of the views of the FIs or it may be due to a low response rate, with only 7.4 percent of PFAN financial partners having participated in the survey. The 91 project developers in the PFAN pipeline who responded to this question listed a variety of barriers, which the evaluation team then grouped into three categories: Investor experience; Assistance on the business plan; and Strengthening capacity of project developers. These three are groups to which the evaluation team assigned individual, open-ended responses. Investor experience mainly refers to the investor network that PFAN brings to the process, the opportunity to meet investors through PFAN, and the chance to see first-hand how financial entities work and make decisions. Assistance with the business plan is hands-on and covers the entire process of developing a business plan. Finally, strengthening the capacity of project developers covers all types of assistance, from coaching to standardized templates. What barriers are not being effectively addressed by PFAN? The assessment by project developers of the barriers not being addressed by PFAN (Table 12) depends heavily on where each developer is in the PFAN process. Developers that reached financial closure identify very different barriers than those who have not reached closure and those who were not qualified for PFAN. For those project developers who have not reached financial closing, policy barriers are ranked as the most important barrier PFAN is not adequately addressing. Advisory services (including business plan assistance, coaching, and technology advice) and finance (such as collateral requirements, lack of interest in CE on part of banks, and general lack of investment funds for CE) were identified as the second most important GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 27 barriers that are not being adequately addressed. This includes the mix of investors, the number of investors at events, and investor perception of CE viability. It is interesting to note that women saw advisory services as a major area where PFAN was not adequately addressing the barriers. A detailed analysis of the answers indicates that is more the quality of coaching services that women are unhappy with. Responses included: Involvement of project conception, business plan mentoring More detailed mentor involvement at the business plan level Technology analysis and scrutiny Developers that have achieved financial closing cite investor relations as the most important barrier that PFAN is not adequately addressing. This covers the mix and number of investors, and matching investors and projects in the early stage. Table 12 shows responses from developers who have and have not reached financial closure. Table 12: Barriers Not Adequately Addressed44 Major Barriers According to Project Developers Not Yet Closed Woman Owned & Not yet closed Achieving Financial Closure n = 70 n = 9 n = 2 Policy 21.1% 11.1% 0% Finance 15.8% 22.2% 33.0% Advisory Services 15.8% 44.4% 0% Investor Relations 14.0% 0% 66.0% PFAN Selection Criteria 10.5% 11.1% 0% While respondents indicated that addressing collateral requirements was an area where PFAN was not adequately addressing barriers, a further analysis of the surveys responses in multiple questions tends to support that PFAN is addressing collateral barriers indirectly through its choice of financial partners and through the advisory services it offers. Women and mixed-gender ownership developers that had previously sought financing unsuccessfully listed the major reason that they were turned down was that they could not meet collateral requirements, but yet they are now in the PFAN pipeline. PFAN may reduce collateral work that reduces the risk gap.45 At this time, there is insufficient data to fully explore the reasons behind 44 Percent of total developers responding to specific issue. Note that Advisory services, Investor relations and Early stage are part of Finance barriers but they were listed separately by respondents. 45 This is the gap between the actual risk on clean energy investments and that perceived by financial institutions. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 28 this finding, but given their inclusion in the pipeline, it can reasonably be assumed that PFAN is making some positive contribution in this area of financial barriers. What additional assistance can PFAN offer under its mandate? PFAN is focused primarily on the business prong in its model, represented by the project developer. PFAN brings investors to meet developers and assists developers in preparing and presenting their case to investors. This is important and does, as has been shown, address some of the barriers to increased CE penetration. Generally, survey respondents from all groups agree that additional assistance is needed in better engaging investors and governments. This tracks with the information in Table 18 and with the barriers that other stakeholders identified as not adequately addressed by PFAN. Table 13: Additional Areas for PFAN Assistance46 Areas for PFAN Assistance Coaches Financial Institutions Developers Female Developers Operational Partners n = 10 n = 5 n = 72 n=9 n = 19 Market Advice for FIs 80.0% 60.0% Policy/Regulatory Advice 60.0% 60.0% 38.2% 54.6% 14.9% Technical Studies 20.0% 51.7% 63.4% Early Stage Funding 60.0% 12.4% 33.3% Advice/Training to FIs 60.0% 46.7% 54.6% 42.9% Greater Public Awareness 57.4% Financial Mechanisms 5.0% 40.0% Expanded Coaching Services 50.0% Note: Columns can sum above 100 percent, as respondents could select multiple areas. The principal areas where developers see the need for additional assistance are technical studies and advice, and training to FIs. This includes either direct provision by PFAN for prefeasibility studies or environmental impact assessments, for example, or grant funding for this kind of work. Recognizing that technical studies are an important barrier, PFAN began a small effort in this area. In effect, developers are calling for an expansion of the technical studies that PFAN is already doing. This may reflect the fact that 91 percent of the 46 developers citing technical studies as a major barrier are from Africa. As a region, Africa has been later and slower than other PFAN regions in developing CE, including resource mapping, technology assessment, and other information that facilitates project planning and development. Additionally, while PFAN began a program of technical assistance in Central America and the Caribbean, it only recently became active in Africa. 46 Respondents were allowed to list all the areas where they believed additional assistance was needed. Each respondent could list more than one item. Therefore the totals can sum to more than 100 percent. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 29 Female developers to all categories and included an additional category which can be term expanded coaching services. Some of the services requested are: Project costing on a more realistic terms; Legal assistance; HR and manpower planning for project implementation from a long term sustainability and profitablity; making commercial bidding documents and contractual expertise and having more technical coaches. Project developers listed specific areas in which they believe that financial institutions need assistance: Developing standardized commercial agreements; Developing risk management practices; Training on CE technologies and due diligence; and Investor education. FIs themselves list standardized templates for commercial transactions and CE underwriting guidelines, a risk management tool, as major areas where assistance is required. 4.5.4 Conclusion 13. Neither the PFAN program nor the cooperative agreement are designed to broaden access specifically in a way that facilitates setting social targets, because the model does not include subsidizing financing nor is it designed to correct for social targets. PFAN selects CE projects primarily on the basis of financial viability and size. PFAN is designed on the basis of providing business skills to financially viable projects where the basis of selecting a project is size and likelihood of reaching financial closure. It was not designed to address gender issues nor inclusion concerns for other disadvantaged groups. The cooperative agreement is not designed with financing subsidized in a way that might account for social targets nor does USAID provide funding specifically for that purpose (which also would affect cost-effectiveness). Further, sex of ownership does not translate directly into financial viability. The collaborative agreement between USAID and PFAN does not mention gender nor have any subsequent modifications. This does not mean that PFAN will not have a measurable gender impact. Thus, gender differences to the extent they are measureable cannot be attributed to PFAN but rather the underlying conditions of the countries in which PFAN works. Conclusion 14. It was not possible to separately report the quantitative measures of performance and effectiveness on a gender basis because (a) PFAN does not collect and report data on that basis and (b) too few closed projects answered the survey to provide meaningful estimates. The quantitative measures of performance and effectiveness need to come from survey responses or PFAN data sources on closed projects. There were too few survey responses to develop any quantitative measure, and the evaluation had to rely on the PFAN pipeline data. PFAN does not collect or report data on a gender basis. This did not mean that other quantitative or qualitative data was not available. Where data permits, we describe the gender characteristics of PFAN and answers to important qualitative areas disaggregated by gender. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 30 Overall, female participation47 in the PFAN survey is 11 percent, as depicted in Table. The highest representation of females is in the area of coaches (18 percent), and the lowest is among FIs, where no female representatives took part in the survey. Table 20: Gender Representation in the PFAN Survey Operational Partner Coaches Resource Partner Developer Financial Partner Total n = 20 n = 19 n = 10 n = 72 n = 5 Female 5.0% 18.2% 11.1% 11.7% 0.0% 10.9% Male 95.0% 81.8% 88.9% 88.3% 100.0% 89.1% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Some notable differences appear in the characteristics of these developers based on gender as shown in Table 19 below. Fewer woman owned developers have obtained financing prior to PFAN, but as a group more are pursuing multiple CE projects than male only owned or mixed ownership developers. Table 14 Selected Characteristics by Gender Characteristics Male Female Mixed Sought & Obtained Financing Prior to PFAN 54% 44% 46% Percent Pursuing Multiple CE Projects 70% 92% 83% More telling is how they see barriers to CE investment. Table 15 indicates an interesting difference. Women acknowledge that the lack of experience in developers is a major obstacle and see collateral requirements as the major barrier to CE. Neither male nor mixed ownership see lack of experience in developers as a major issue. Both view the lack of interest of banks in CE as the major barrier. Table 15 Ranking Barriers to CE Investment48 Barriers Male Female Mixed Lack of favorable government policies 2 2 Collateral requirements too high 3 1 2 Banks not interested in clean energy 1 2 1 Project Developers not experienced in business aspects such as completing a business plan 2 47 Clearly gender deals with more than just differences in sex but these differences are not directly encompassed in PFAN data. 48 A rank of 1 means that more respondents chose this barrier than any others. In some cases, there were an equal number of respondents per barrier and so they have been equally ranked. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 31 Number Responding 51 11 30 Finally, when we asked if PFAN was successfully addressing the barriers to CE, 64% of women developers answered yes compared to their male counterparts (59%) and the mixed ownership at 54%. This does not rs to CE are more helpful by sex of ownership. Conclusion 15. Available data suggest PFAN may have a neutral to positive impact on end-use beneficiaries from a gender perspective. While PFAN was not designed to directly address gender issues, by focusing on the triple bottom line, PFAN will address some aspects of gender-related issues. For example, projects that target rural energy access or projects that focus on cooking technology and fuels will generally disproportionately impact women and children. d 49 with benefits that are disproportionately female-oriented. This represents a little more than 10 percent of the total investment value of the pipeline. Aside from the end-use beneficiaries of PFAN projects, PFAN has gender implications in project ownership. According to the survey, 11 percent of project developers are female and another 32 percent of projects are jointly owned by male and female partners. 4.6 SUSTAINABILITY AND REPLICATION Evaluation Question 6. What is the performance of PFAN participants post-financial closure, including reaching and maintaining operational status, replicating or expanding business, and producing co-benefits for themselves or their communities? Evaluation Question 7. either in its current form or as the program transitions to a self-supporting entity? This section focuses on several types of sustainability and replication. The two types of sustainability are from the investment or project side and from the program side, or PFAN itself. Replication examines replication or expansion by the project beneficiaries, the financial institutions, and coaches. 4.6.1 Conclusion 16. Insufficient data exists to definitively conclude the degree to which the closed projects have started operations and will remain in operation. PFAN does not keep records on the number of projects that begin operations, nor is it required to do so. In general, projects that reach financial closure will start operations. In the case of many of these types of CE projects, the start of operations is not nearly as important as the continued operations over the midterm. This is when the issues of service delivery, maintenance, and consumer valuation of service come full stage and when the project developer and the concept are severely tested. This information had to be determined by survey. 49 These include projects for cookstoves, biogas dedicated for rural poor, off-grid generation for the poor, and similar access projects. The evaluation erred on the side of undercounting gender benefits. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 32 When projects are commercially financed, as they are through PFAN, the financiers have an incentive to work with the project to ensure that it continues to operate and service its financial obligations. Only four of the 49 -survey outreach to gather the required information. Both of these projects having reached financial closing also have started operations. 4.6.2 as a self-supporting or self-financed entity. In this context, updated means that the basic public-private nature of PFAN is maintained but its scope is changed. First, updating within the existing focus on the project developer is addressed and then this section looks at where PFAN can expand. The last section looks at PFAN as a self-supporting entity financed entirely or mainly from services. Conclusion 17. If the objectives of PFAN remain valid (both to broaden access to and increase the financing of CE projects), then PFAN as an alternative to transitioning it to a self-supporting entity. The results of this survey and other studies point to the fact there are serious market imperfections in the policy, financial, and business prongs. Addressing directly only one of these, as PFAN does will mean that a fully self-supporting PFAN will reach far fewer projects. Conclusion 18. There are several important ways that PFAN can be updated: by expanding the current business role to include assistance in financial closing, including negotiations; expanding PFAN to work in one or more of the remaining prongs, (policy and finance); or aligning PFAN to support other donor- and IFI-funded projects that focus on the policy or finance prongs.50 An Updated PFAN51 need assistance with formulating a business plan, structuring the project or meeting investors, they lack This would give PFAN greater influence over the entire process and would lead to more accurate reporting. In an example of problems with accurate reporting, at least one project reported by CTI to have reached financial closure indicated, when contacted about the survey, that they had not yet closed. Second, PFAN is mainly addressing one prong of the CE challenge. The penetration or replication from PFAN investment depends upon the capacity of the developers and financial institutions that appear to be quite limited. Much of what PFAN does or attempts to do is to compensate for market failures, account for externalities, and address the divergence between socially optimal CE investment and market outcomes. Third, it is clear that barriers remain in the other prongs. One option is for PFAN to continue the work it does so well in the business prong while simultaneously working in markets alongside other programs that address financial and policy barriers, i.e. programs that are aligning their work with other donor projects that address these other barriers.. For example, the IFC has a sustainable energy finance program in the 50 PFAN has begun this to a limited extent in its cooperation with Renewable Energy and Energy Efficiency Partnership in the Phased Financing Facility. This facility is an early-stage funding mechanism and gets at the barriers mentioned by many of the survey respondents but does not directly address investment financing. 51 Any discussion of an updated PFAN that considers an expansion of its role will of necessity need additional funding. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 33 Philippines with Bank of the Philippines Islands and Banco De Oro. PFAN might work with those Banks as financial partners for some of its Philippine projects. This combined effort could reduce time to closure and bring additional projects into the pipeline at the same time. Finally, there is the question of payment. It is clear that PFAN services are valued and, judging from the interviews, that partners are willing to pay some fee to participate. While this may be insufficient to cover all and project developer at the time of closing is one alternative that is recommended based on the experience of E+CO (see below). Transitioning PFAN to Stand Alone To address if and how PFAN can be transitioned to a self-supporting entity, it is important to understand the PFAN project is providing them? PFAN exists for two reasons. The first reason is that market failure creates a number of barriers to increased CE penetration, and PFAN exists to address a combination of those barriers, primarily in the business prong. Based on the results of this survey, it is apparent that PFAN services also indirectly address some of the financial barriers. The second reason is that donors and IFIs believe that the socially desirable level of CE investment is greater than what the market would produce. PFAN is one of the programs designed to achieve a socially desirable outcome. PFAN Services On the surface, PFAN is about providing assistance to CE project developers. It is readily apparent that PFAN builds capacity in CE businesses to develop projects and complete the investment application process in a manner acceptable to investors and lenders. It does this through coaching services that it offers at a subsidized rate. But the program has other benefits as well. The capacity of coaches is developed and their reputation strengthened as successful providers of business advice. One hundred percent of coach respondents indicated that their participation with PFAN had improved their ability to help bring projects to financial closure, and 60 percent reported that their business had increased as a result of their participation with PFAN. Further, PFAN is reducing the search costs for developers, coaches, and financial entities. Financial institutions listed some form of reduced cost as their major reason for participating in PFAN. Finally, PFAN is providing a service to donors. Donors believe that the ideal level of social investment in CE is greater than what the market will deliver, so they invest in programs like PFAN. Are Markets Ready? The demand for PFAN services is a function of the cost of its services, the presence of substitutes for its services, the demand for clean energy compared to traditional energy, and many other factors. There are a number of substitutes at both the financial entity and project level for PFAN. First and foremost are competing PFAN platforms like USAID direct PFAN contracts. These offer similar services to this t work through commercial banks and offer free or subsidized technical assistance. Third, there are numerous clean GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 34 energy donor projects from a large variety of donors including the German Agency for International Department for International Development (DFID), the Swiss Agency for Development and Cooperation (SDC), the French Development Agency (AFD), and international finance institution projects such as the World Bank and Asian Development Bank. While these are not perfect substitutes to PFAN, they address CE market barriers, provide finance and technical assistance, and would compete with a fully self-supporting PFAN. Conclusion 19. The presence of donor- and IFI-funded CE promotion programs is likely to reduce significantly the demand for PFAN services from the better CE projects, as stronger and price-sensitive project developers instead attempt to get more heavily-subsidized assistance elsewhere. At the same time, the need to become 100 percent self-financing will force PFAN to look at the best of the projects, those which could likely obtain financing independently, albeit at a slower pace, while leaving the majority of projects behind. It is likely that fewer and smaller projects will be the result. Learning from Past Experience E+CO was close to the PFAN model except that it was able in some cases to provide financing itself, and in many cases the cost of debt or equity to the developer was at market rates. The hallmark of the program was its technical assistance designed not only to bring projects to closure but to stay with projects to make sure that they were operational and profitable. Similarly, E+CO focused on the triple bottom line. The company won numerous finance awards, was cited as a model for innovation, and was a favorite of the donor and IFI community. Evaluations of E+CO implemented projects were solid. Yet, in 2012, E+CO went into reorganization. eloping assistance to de- 52 This is not to say that PFAN will necessarily go the same way if funding is reduced entirely or significantly. PFAN is, however, likely to encounter the same problems faced by E+CO. Those problems were the following.53 E+CO was viewed by some developers as an extension of donors, so developers believed they did not need to pay for assistance even when they agreed to do so; As E+CO became more successful, the next tier of projects was smaller and per-unit transaction costs increased significantly; and As E+CO moved from technical assistance provided through donor grants or paid for by donors to a -funded assistance where available. 52 Quote form Christine Eibs Singer former CEO of E+CO Huffington Post, 10/03/2012, last referenced 01/21/2015 at http://www.huffingtonpost.com/david-bank/eco-avoids-liquidation-ba_b_1932503.html. 53 Source: M.W. Addison notes from the evaluation of E+Co, 2009. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 35 4.7 RECOMMENDATIONS Most of the recommendations revolve around the collection of data and the need to occasionally evaluate the PFAN model and approach. The implicit assumptions in the way that PFAN has operated are that the to be empirically verified. 4.7.1 Recommendation 1. Consider funding PFAN to develop an appropriate performance monitoring plan, collect data, evaluate key areas of PFAN, and make mid-course corrections if necessary. As illustrated throughout this report, there were limits on data available to thoroughly address all evaluation questions. Thus, there are a number of areas where additional data54 should be maintained so that performance can be monitored to better or further evaluate, on issues such as: Testing the key hypot Determining if all projects accepted into the pipeline should remain there; Better maintaining separate costs for activities to allow for measuring the cost effectiveness of establishment and maintenance of lender/developer relationships; Understanding the application rates or rejection rates; and Determining where greater weighting in service areas will reduce time to closure, increase the number of closings, or attract additional financiers to the program. To facilitate better evaluation of these issues, PFAN needs to collect additional general operational and financial data on all projects that apply in addition to those projects it supports, on closures and not just pre￾closures, and on closed projects until they reach stable operations. One way of collecting the information is a contractual agreement to continue to provide data after closure in return for PFAN services or even in return for consideration for PFAN services. The latter approach, if feasible, could help build a database for a counterfactual at least at the bottom end of the quality distribution. 4.7.2 Recommendation 2. still relevant and then use the performance indicators to specifically address the outcome of this exercise. Securing more private sector finance might lead PFAN to choose developers that already have some finance, already have projects in the PFAN pipeline or already have successful experience with closing projects over other developers. The full implications of this have not been investigated. USAID and other , use the performance indicators, and, if necessary, tailor new, appropriately-scoped and manageable performance indicators. 54 This data, hopefully ex post, includes ownership by gender, co-benefits, and other GCC metrics such as MWs installed capacity and GHGs avoided. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 36 4.7.3 Recommendation 3. PFAN should attempt to collect data on closures from the parties that actually sign the closure documents. Recommendation 4. PFAN should consider adding to its process routine exit interviews or surveys with closed projects to ascertain their performance and issues surrounding startup and operations. from PFAN and too few closed projects participated in the survey or the interview to objectively evaluate how to improve effectiveness and speed in reaching closure. PFAN should consider adding to its process routine exit interviews or surveys with closed projects. This feedback can allow PFAN to ascertain their performance and issues surrounding startup and operations and address inefficiencies in project implementation. PFAN should also collect data on closures not from the coaches but rather from the parties that sign the closure documents the financial institutions and project developers as the coach leaves the process before closure is reached. Recommendation 5. USAID should considering additional funding that would allow PFAN to address remaining barriers in the business area such as negotiations and assistance for financial closure. 4.7.4 Recommendation 6. Consideration should be given to a scenario in which PFAN moves to a partially self￾supporting model. Moving to a fully self-supporting model would diminish the reach of PFAN. There are two aspects of sustainability that PFAN needs to address. The first aspect is the sustainability of the clean energy investments. The response rate on closed projects was too low to draw any meaningful conclusions on sustainability given the lack of additional data on projects after closure. The second aspect is the sustainability of the PFAN program itself. To move to a purely self-supporting model for PFAN is likely to cause impact and a loss of much of the effectiveness from the structure that has been built. By viewing PFAN components separately rather than as a whole, a model of continued support of key elements of PFAN is likely to better leverage donor investments, maintain institutional structures already installed, and continue the impact on the underlying issue of CE finance barriers. Partial assistance would also allow CTI to test how far PFAN can be self-sufficient before becoming uncompetitive relative to other donor-funded programs. It might do so, for example, by offering variable cost-shares to study elasticity of demand for its services. The very existence of PFAN acknowledges the fact of social objectives of donor countries that are not borne out in markets; the basis of the PFAN program is on this market failure. The former can be considered at least partial justification for maintaining some continued assistance unless and until the markets in developing countries reflect the social values of donor countries, and something that PFAN could not recoup in fees until such time. The latter, the assumption of market failure in investing in clean energy projects in the developing and transitional context, is an area where PFAN can begin to collect fees. 4.7.5 Recommendation 7: There are several ways USAID could consider to promote gender in PFAN, but each has restrictions, costs and ramifications for PFAN as a program that USAID should weigh. GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 37 First, it is clear that female developers want and need additional coaching services. USAID could provide additional funding to strengthen the services. This might mean more coaching time and might include coaching areas where PFAN does not normally work such as legal advice. That reputation is founded on selecting, improving, and uniting viable CE projects with serious financiers. Anything that results in PFAN attracting and promoting marginally viable projects can result in damage to that image. If there are times when the supply of viable projects exceed PFAN resources and some projects are not accepted into the program, preference could be given on the basis of sex of applicant(s). This would be one way of promoting empowerment of female developers in the delivery of PFAN services without damaging its reputation. One way to do so would be selection on an ownership basis with a gender weighting. Another method is not in the selection based on ownership but rather on the downstream benefits of the project, based on gender impacts. Viable projects having greater benefits specifically for women, children and/or disadvantaged groups (e.g. improved morbidity and mortality rates in women and children due to reduced pollution from clean cooking fuels) could be given greater weighting. Similarly, in the selection of coaches, preference could be given on a gender basis with additional training as needed. In those cases where the number of viable projects is not restricted due to funding constraints, inclusion as selection criteria of additional factors not linked to project viability (such as gender), by definition would result in the selection of less-financially viable projects. There are a few ways that gender consideration could be considered without dam seek to target some social/green funds where gender is a consideration in their portfolio selection. Generally, these social/green funds expect lower rates of return and thus what would be marginal for purely for-profit financiers may not be marginal for these entities. PFAN would entail search costs to locate and then convince these funders of the value of looking for projects graduating from its program. CTI and USAID would need to consider whether this requires additional funding for this effort. Two, USAID could set aside funds for PFAN both for the business TA and for financing these marginal investments at a subsidized rate. ds of investors would remain intact. However, the challenge here is to find a vehicle to administer the fund. It Three, CTI could examine whether additional focus on female-led or co-led projects could be provided at existing investor fora or through organizing additional investor fora highlighting only those projects. Finally, USAID could attempt to get PFAN to attract more marginally-viable projects on a gender basis. That may cause PFAN to consider whether or not to accept USAID funding and could call into jeopardy the work that USAID and PFAN are doing together. Recommendation 8: empowerment and disadvantaged people could be meaningfully integrated in follow-up activities for PFAN or similar programs in future cooperative agreements. This consideration should account carefully for the tradeoffs between promoting consideration of additional social inclusion factors and commercial viability and the likely effect on CE investments given the context of clean energy project finance. and uniting financially viable CE projects with financiers. Any approach that USAID takes to promoting resources and attention from PFAN both in terms of mentoring and perhaps for financing less viable GCC M&E Performance Evaluation Final Report of CTI Private Financing Advisory Network 38 projects. If USAID is interested in setting social targets, USAID should review how its cooperative agreement is designed and consider subsidized financing that might correct for social targets. However, to the extent that the supply of right projects is larger than that which PFAN could fund, PFAN could weight more heavily during its selection process gender of ownership in its decision criteria. Recommendation 9: USAID could alter its collaborative agreement with PFAN. This update could include instructions on considering gender, having gender related targets (developed together with CTI), provide gender considerations and targets. dTS has given several ways that this might be accomplished without accepting marginally viable projects and affecting projects. It is believed that PFAN is in a better position to recommend to USAID which of these is best suited for results. It was prepared by Development & Training Services, Inc. (dTS). March 12, 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by Matthew Addison, Farhat Rahman, Marc D. Shapiro, and Elizabeth Stahl of Development & Training Services, Inc. (dTS). PERFORMANCE EVALUATION DRAFT FINAL REPORT OF CLIMATE TECHNOLOGY INITIATIVE PRIVATE FINANCING ADVISORY NETWORK GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN i Acknowledgements: The authors and dTS would like to thank the many people and institutions that supported the design and execution of the performance evaluation of CTI PFAN. K. Rayne Loken was instrumental in the design of the evaluation, communication with partners, and the drafting of the data collection instruments. USAID staff in the E3/Global Climate Change Office provided invaluable guidance and support. Peter Storey, Taiki Kuroda, and Elmer Holt, Jr. at CTI actively demonstrated commitment to learning from this pilot activity and supported dTS in obtaining necessary documentation and maximizing learning opportunities. This evaluation was carried out with support provided by the United States Agency for International Development (USAID), Global Climate Change Office (GCCO), Bureau for Economic Growth, Education, and Environment (E3), through the Global Climate Change Monitoring and Evaluation Project (GCC M&E), Contract Number AID-RAN-I-00-09-00015, Task Order Number AID-OAA-TO-12-00001. Development & Training Services, Inc. (dTS) is an international development company that leads initiatives in social and economic development with a view to promoting equality, accountability, and sustainability. For information about dTS and its projects worldwide contact: Development & Training Services, Inc. (dTS), 4600 North Fairfax Drive, Suite 402, Arlington, VA 22203, USA. Phone: +1 703-465-9388; Fax: +1 703-465-9344; Email: info@onlinedts.com; Internet: www.onlinedts.com. Recommended citation: Addison, Matthew, Farhat Rahman, Marc D. Shapiro, and Elizabeth Stahl. 2015. Performance Evaluation Final Report: Climate Technology Initiative Private Financing Advisory Network. Arlington, Virginia, USA: Development & Training Services, Inc. (dTS). GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN ii DISCLAIMER The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. AUTHORS Matthew Addison, Farhat Rahman, Marc D. Shapiro, and Elizabeth Stahl of Development & Training Services, Inc. (dTS). GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN iii Acronyms ........................................................................................................................................................................................................................iv Appendix I: Evaluation Statement of Work ................................................................................................................................................ 1 Appendix II: Evaluation Team Composition Table ..............................................................................................................................13 Appendix III: Evaluation Design Matrix ........................................................................................................................................................15 Appendix IV: Detailed Evaluation Methods and Limitations ..........................................................................................................19 Appendix V: Qualitative Data Collection Instruments ......................................................................................................................28 Appendix VII: Disclosure of Any Conflict of Interest .........................................................................................................................43 Appendix VIII: Options for Improved Transparency and Documentation ............................................................................49 Appendix IX: Online Survey Instrument ....................................................................................................................................................51 GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN iv ADB Asian Development Bank AFD French Development Agency BEEF Bulgaria Energy Efficiency Fund CE Clean energy CO2e Carbon dioxide equivalent COI Conflict of interest CTI Climate Technology Initiative DFID United Kingdom Department for International Development dTS Development and Training Services, Inc. E3 Bureau for Economic Growth, Education and Environment EBPD Everyone but project developers EE Energy efficiency ECP I Energy Conservation Project I EGTT Expert Group on Technology Transfer FI Financial institution GCC Global climate change GCC M&E Global Climate Change Monitoring and Evaluation Project GEEP Georgia Energy Efficiency Program GHG Greenhouse gas GIZ German Agency for International Cooperation GWh Gigawatt hours ICETT International Center for Environmental Technology Transfer IFC International Finance Corporation IFI International financial institution IP Implementing partner IRB Institutional review board KII Key informant interviews LED Light emitting diode M&E Monitoring and evaluation MT Megaton MW Megawatt OP Operational partner OSO Operational stakeholder organization PFAN Private Financing Advisory Network PV Solar photovoltaic RE Renewable energy SDC Swiss Agency for Development and Cooperation SEF Sustainable Energy Facility SOW Statement of Work UNFCCC United Nations Framework Convention on Climate Change USG United States Government GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN v USAID United States Agency for International Development USG United States Government GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 1 STATEMENT OF WORK PERFORMANCE EVALUATION OF THE PRIVATE FINANCING ADVISORY NETWORK June 2, 2014 Submitted to: The United States Agency for International Development Prepared by: Development & Training Services, Inc. (dTS) USAID Contract Number AID-RAN-I-00-09-00015, Task Order Number AID-OAA-TO-12-00001 Global Climate Change Monitoring and Evaluation Task Order DISCLAIMER: nt do not necessarily reflect the views of the United States Agency for International Development or the United States Government GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 2 PERFORMANCE EVALUATION OF PRIVATE FINANCING ADVISORY NETWORK 1 dTS and its sub-contractor Tetra Tech have been contracted by USAID, through the Global Climate Change Monitoring and Evaluation Project (GCC M&E), to design and conduct research to inform future provide an essential contribution to learning, in its efforts to mitigate and to reduce vulnerability to climate change. The Private Financing Advisory Network (PFAN) is an existing public-private partnership, which was established to support the development of alternative energy and energy efficiency businesses by providing mentoring and access to sources of project finance to prospective project developers. PFAN identifies projects that may be suitable for private sector finance and then acts as a project finance coaching and consultancy service to guide these projects toward becoming bankable and securing investment. 2 PFAN was initiated by the Climate Technology Initiative (CTI) in cooperation with the United Nations Framework Convention on Climate Change (UNFCCC) Expert Group on Technology Transfer (EGTT) and is supported by a number of private sector companies in the financing sectors of the clean energy (CE) / renewable energy (RE) / energy efficiency (EE) industries. The CTI is responsible for coordinating the PFAN Program while the International Center for Environmental Technology Transfer (ICETT) is responsible for administering the program. USAID is a partner of the PFAN alliance and provides support to the program through a cooperative agreement. The agreement runs from Oct 2007 to Sept 2015, and has a budget of $5 million. PFAN activities started in 2006 as a pilot project and expanded in 2008. During the scale up, PFAN established regional networks in Latin America, Africa, Asia, and Eastern Europe, as well as dedicated country specific networks. Currently, the program is funded through various international partners, who also may offer support in the form of discounted service rates and fees. Ideally CTI-PFAN will become a stand￾alone entity and preparations are underway to facilitate this transition. The projects accepted into the program are showcased on the PFAN website in a project pipeline summary report and are tracked through financial close. As of March 2013, there were 164 projects in the development pipeline, representing $4.9 billion of investment. Of these, 38 projects closed, raising $432 million. Of the 10 financing fora that were held, 422 projects were identified, 116 selected, 84 showcased, and 24 closed, raising $251 million. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 3  Broaden the access to financing for climate friendly and technology transfer projects in the RE/ EE sectors in developing countries and economies in transition;  Get more renewable energy and climate friendly projects financed in the private sector and thereby to accelerate technology transfer under the UNFCCC. PFAN targeted the lack of access to financing by bridging the gap between financiers and project developers. By providing mentoring, PFAN helps project developers speak the same language as potential sources of financing and help project developers create more robust business plans. PFAN provides advice and guidance on:  Overall project structure  Finance structuring, and sourcing and procuring financing  Technical and engineering assistance  Preparation and presentation of investment proposals The second aspect of the program is the connection between potential investors and project developers through investor matchmaking, usually in conjunction with a regional workshop where project developers can present their proposals directly to members of the PFAN network. The program targets midsized projects in the $1 million to $50 million range. In order to qualify for PFAN support, a project must undergo a rigorous selection process based on an initial project description, proposal, and other relevant information. Projects accepted into the program go through three more stages of review based on the individual project and the coaching that it receives. In general, these three stages review the following elements of the project:  Project economics and viability;  Technical and engineering aspects;  Problem solving and marketing. Each review stage is summarized in a formal written memorandum to the project sponsor which provides an analysis of the strengths and weaknesses of the project proposal and suggests next steps for improvement. As stated in the CTI￾indicators are  Percent of projects engaged by PFAN will come to fruition 33 to 44 percent;  Total public and private dollars leveraged by USG for energy infrastructure projects $180 million; and  Number of commercially and concessionally financed projects as a result of USG assistance 24. PF anticipated from the support of PFAN:  Partnership, Methane to Markets: acceleration of the implementation of priority projects by the creation of expanded access to financing; GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 4  Promotion of sustainable development in developing countries in the areas of CE/RE and EE, achieved by the innovative PFAN approach which is explicitly designed to ensure capacity building (especially in financing) thereby further promoting economic and social stability;  In the course of their capacity building efforts the PFAN Consultants will be actively using and promoting the use of the UNFCCC Guidebook on preparing Technology Transfer Projects for Financing;  Promotion and acceleration of the technology transfer process in key areas to help developed and developing countries meet their UNFCCC obligations and to mitigate climate c 3 3.1 EVALUATION PURPOSE The purpose of the performance evaluation of the PFAN activity is to assess the performance, effectiveness, and sustainability of PFAN and its participants. This evaluation is to provide information for future programmatic and policy-related decision making, contextual learning for USAID and other involved partners and stakeholders, and to demonstrate accountability for resources. Recommendations on specific program elements in the evalu implementation, either now or as the program transitions to a stand-alone entity. The performance evaluation creates an opportunity for the PFAN activity and GCC to learn from a stakeholder perspective, whether or not developers felt that the barriers and challenges that they faced were addressed sufficiently by the activity, or, if not, how PFAN could address these issues further. The performance evaluation also creates an opportunity gain greater insight into the longer term impacts of the mentoring and financing support that PFAN provides. The PFAN activity collects information on project developers related to the anticipated financing, GHG benefits, location, and technology type. However, additional information, such as the current status of PFAN projects post￾other projects, and the co-benefits that developers and/or their communities receive, is not being collected by the current system that can be collected during the course of the performance evaluation. The results of the evaluation can be used to adjust the design of the PFAN program within the scope of its vide insights which showcase this program in USAID and CTI-PFAN communications products. Results from the evaluation will be shared with a variety of stakeholders, including the program coordinators at the CTI and the USAID/GCC office. The results of the evaluation may be distributed to other funding Energy for All Program, the Energy and Climate Partnership of the Americas, and the International Center for Environmental Technology Transfer. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 5 3.2 EVALUATION QUESTIONS Performance 1. What has been the cost effectiveness of PFAN in relation to:  Clean energy technology financing  Establishment and maintenance of lender/developer relationships  Leveraging private sector resources, such as in kind services and mentoring  CE technology deployment? 2. What are the strengths and weaknesses of the PFAN organizational and partnership model? Effectiveness 3. Why have certain PFAN projects been more or less successful in achieving financial closure? 4. In what ways and to what extent do project developers credit PFAN for their successfully securing financing? 5. Are there barriers which are not being addressed or that are not being addressed effectively? i.e., what additional assistance can PFAN offer under its mandate? Sustainability and Replication 6. What is the performance of PFAN participants post financial closure, including reaching and maintaining operational status, replicating or expanding business, and producing co-benefits for themselves or their community? 7. form or as the program transitions to a stand-alone entity? 3.3 RESEARCH DESIGN AND DATA COLLECTION METHODS 3.3.1 DATA COLLECTION AND METHODS Four data sources will be utilized in this evaluation. The two primary sources will be an anonymous electronic survey open to stakeholders and key informant interviews. The two secondary sources are document and literature reviews and a review of information provided by CTI-PFAN and on its website. The various data sources are described in further detail below. A strategy to address and assess gender within this evaluation will be incorporated into the evaluation plan document. In summary, questions will be asked in order to ascertain the rates of female participation, special challenges that they face, reasons why they might not be participating, and suggestions for getting more women involved. 3.3.2 DATA SOURCES Two primary sources of data will be utilized: electronic survey instrument and key informant interviews. Survey Instrument. The target audience for the survey instrument will be PFAN participants, non￾participants, implementers, and finance institutions in all PFAN countries. The survey questionnaire instrument will be administered online, incorporating both open- and closed-ended questions as necessary GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 6 to answer the detailed evaluation questions (See Evaluation Goals Above). Survey questions will be prepared and submitted to USAID and CTI for review prior to survey launch. Interviews. In-depth key informant interviews will be planned based on the recommendation of the conducted when possible to provide further insight into collected quantitative data in terms of attitudes and -PFAN) and conduct interviews with relevant operational stakeholders, mentors, project developers, and financial institution representatives. Stakeholder lists, interview questions, and other interview documentation will be prepared in advance and finalized in coordination with USAID and CTI.The information below provides an overview of the types of questions that will be asked in order to inform the evaluation questions above. PERFORMANCE 1. Assessing the extent to which there is evidence that the program has contributed to achieving the assistance objectives. This may include measures such as size (e.g., megawatts), value (total dollars of secured financing), and technology. 2. Estimating co-benefits of PFAN projects. 3. Determining the success rate, i.e. the ratio of projects that reach financial closure to the projects that are accepted by PFAN. 4. Determining whether stakeholders believe that PFAN has contributed to attainment of the UNFCC goals. 5. What are the results of PFAN assistance to date in terms of carbon dioxide offset, megawatts (MW) of clean energy, types of technology and financing leveraged? EFFECTIVENESS 1. Determining the extent to which PFAN helps to address identified obstacles for CE finance. 2. Determining the extent to which identified obstacles match on the ground realities and whether additional types of assistance are warranted or the extent to which the emphasis on assistance matches the ground level reality. 3. Identifying which project elements contribute to success in securing finance. 4. Determining the extent to which the project developers credit PFAN for successfully securing financing. 5. Determining the extent to which projects that received funding were completed and started operations. 6. Determining whether the PFAN mechanism reduces the amount of time and/or cost for a project developer to reach financial closure. 7. Determining whether PFAN participants are pursuing alternative financing assistance and if so, what is the nature of that assistance? – How does the alternative assistance compare to PFAN? – Who is providing it? – Why are they engaged if already participants of PFAN? GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 7 8. Determining the extent to which women and women-owned businesses are: – Participating in PFAN; – Successful in securing financing; – Achieving operationalization of projects; – PFAN mentoring successfully addresses the needs of women. SUSTAINABILITY AND REPLICABILITY 1. Assessing the success of PFAN funded projects in obtaining long-term operational status. 2. Determining the extent to which the developers have utilized skills and resources obtained through PFAN (through mentoring) successfully outside of the PFAN mechanism (future projects) and/or transferred those skills to other developers (became mentors). 3.3.3 EXISTING PERFORMANCE INFORMATION Desk review of documents. To complement the two primary data sources described above (3.3.2 survey instrument and key informant interviews), dTS will review relevant project specific documents such as proposals, reports, scope of works, etc. provided by CTI-PFAN. The evaluation team also will go through these as well as the official websites of PFAN and collect other relevant documents, reports, and data. USAID and CTI-PFAN representatives have provided dTS with various background documents and sources of data. The following is a comprehensive list of documents received or sourced to date: 1. The Request for Proposals (RFP) for the Regional Clean Energy Investment Project; 2. The original and subsequent modifications of the Program Statement for the PFAN Program; 3. The Cooperative Agreement between USAID and ICETT concerning PFAN; 4. An amended PFAN Scope of Work specifically related to PFAN activities within the Economic Community Of West African States and Regional Centre for Renewable Energy and Energy Efficiency; 5. CTI-PFAN presentation given at the UNFCCC Technology Needs Assessment (TNA) Workshop in Bonn, Germany June 2011; 6. Business Plans, Project Data Sheets, and Presentations for the following four CTI-PFAN Projects: – Tough Stuff Kenya – Cleanopolis Energy Systems India – Primavera City Philippines – SOIL Ghana 7. The CTI-PFAN Website which includes: – Project Development Pipeline – Partner information – Member information – Background and contextual information for CTI-PFAN – Success stories – Fact Sheets – Information about CTI-PFAN Services available GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 8 8. The USAID/Eco-Asia CDCP Evaluation; 9. The Clean Energy Independence Report (FY 2011); 10. UNFCCC Guidebook on preparing Technology Transfer Projects for Financing. 4 4.1 EVALUATION TEAM COMPOSITION TABLE The dTS evaluation team is comprised of a dTS team leader, monitoring and evaluation specialist, and potentially a data analyst. Name and Qualifications Role Responsibilities Matthew Addison, MS, Economics Mr. Addison has extensive experience in clean energy finance, clean energy policy and regulation, as well as clean energy evaluation. He has managed teams on a variety of clean energy initiatives for national governments, donor agencies, and private businesses. Team Leader – Provide clean energy technical expertise for evaluation design and implementation – Supervise survey instrument creation and ensure data quality – Oversight of the drafting of the all reports and deliverables – Provide leadership, coordination and supervision of team members on relevant evaluation activities, including documentation review, interviews, analyses and formulation of conclusions and models Rayne Loken, MA, MPH Mid-Level monitoring and evaluation specialist. She has significant experience in program evaluations, survey design and field-testing, and qualitative and quantitative data analysis. Additionally, she has extensive experience in drafting technical reports and deliverables for USG agencies. Monitoring and Evaluation Specialist – Draft all reports and deliverables, with feedback from Team Leader – Assist with creation of survey instrument; oversight of survey administration – Supervise the analysis of data and the translation of the data to findings, conclusions, and recommendations – Technical methodological inputs to relevant deliverables – Suggest revisions to the evaluation design to the Team Leader – Day-to-day/home office management of evaluation technical and operational Marc Shapiro, PhD, Political Science Dr. Marc Shapiro has extensive experience in project management, monitoring and evaluation (M&E). His technical specialization within M&E is in leading, designing, and implementing impact evaluations and quantitative methods. Senior Technical Advisor – Provide technical advice to the Monitoring and Evaluation Specialist – Serve as quality controller across evaluations GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 9 Name and Qualifications Role Responsibilities TBD, if needed Junior level data analysis associate. Will have 3+ years relevant experience in survey administration, data collection, analysis, and management. Data Analysis Associate – Survey administration and quality control – Data analysis 4.2 COORDINATION WITH USAID/GCC, MISSION, AND IMPLEMENTING PARTNER The evaluation will be coordinated through the USAID/GCC office. The main points of contact for this evaluation are:  Kate Faulhaber, M&E Specialist USAID   Peter Storey, PFAN Global Coordinator, Director of PPL International  Elmer Holt, PFAN Manager, Vice Chair of CTI  Taiki Kuroda, Chief of CTI Secretariat dTS will copy Ms. Faulhaber, Mr. Taylor, Mr. Storey, and Mr. Kuroda on all requests for data and information from CTI. 5 dTS will provide the following deliverables in electronic format. The final report will be delivered in both paper (15 copies) and electronic format. The following is a list and description of proposed deliverables. 5.1 CONTRACTUALLY-REQUIRED DELIVERABLES Evaluation Scope of Work. Evaluation Plan. An evaluation plan will be prepared and agreed upon in collaboration with USAID/E3/GCCO and CTI. Draft Evaluation Report/Final Evaluation Report. 1 Draft and Final Evaluation Reports will include all elements and sections required by the dTS contract2 performance evaluations.3 This includes: 1. research design 2. data collection plan 3. description of the methodology and schedule 1 Note that as this is a standard performance evaluation and no draft or final interim report will be provided. 2 Contract/Order AID-OAA-TO-12-00001 3 USAID Evaluation Report Template http://usaidsite.carana.com/content/usaid-evaluation-report-template, and How to Note on Evaluation http://www.usaid.gov/sites/default/files/documents/1870/How-to-Note_Preparing-Evaluation￾Reports.pdf GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 10 4. data source identification 5. data analysis plan 6. statement on anticipated data quality 7. anticipated survey instruments 8. evaluation team members and stated roles and responsibilities 9. list of additional documents requested from USAID and the implementing partner 10. any logistical or other support that dTS team members will need from USAID or CTI 11.timeline for the entire evaluation activity Oral Briefing and PowerPoint Presentations. A meeting will be held with relevant stakeholders before the finalization of the report to present and discuss key finding, conclusions, and recommendations. After the evaluation report is finalized, if desired, a summary brownbag PowerPoint presentation will be delivered at Flash drives with data and instruments. dTS will provide clean datasets including metadata in a format suitable for reanalysis. All clean datasets will meet professional standards and be in a format agreed upon with the USAID/GCCO COR, such as Excel, SPSS, or STATA. Final copies of all survey instruments and interview guides will also be provided. 5.2 OTHER DELIVERABLES Stakeholder Tracking Matrix. This matrix (in Excel format) will be provided to E3 and the IP in order for them to provide dTS with guidance on appropriate points of contact for the survey and interviews. After the matrix is completed, dTS will use it as a tool to track survey and interview completion and response rates. If required, the evaluation team will assign random number codes to each individual survey respondent in order to protect anonymity. Survey Questionnaires. The survey instrument will be drafted and provided to USAID/GCCO and CTI for comments and feedback before it is launched online. The evaluation team envisions a single master instrument which will use a skip logic format so that it is tailored for the distinct target respondent groups. These groups are: Operational Stakeholders, Mentors, Project Developers, and Financial Institutions. Interview Guide. The interview guide will be drafted and provided to USAID/GCCO and CTI for comments and feedback before it is utilized. Guidance will be sought from USAID and CTI-PFAN for determining the most appropriate stakeholders to be interviewed. As with the online survey, the interviews will target key informant within the following groups: Operational Stakeholders, Mentors, Project Developers, and Financial Institutions. All written deliverables will be submitted in accordance with USAID branding requirements.4 The final evaluation report will be submitted to the Development Experience Clearinghouse in accordance with all USAID requirements for submittal. 4 USAID Branding www.usaid.gov/branding and Graphics Standards Manual http://www.usaid.gov/branding/gsm GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 11 6 A timeline and level of effort is proposed below. dTS also will provide a budget estimate associated with a work planning Gantt chart. These documents will be provided by dTS after agreement with USAID and CTI-PFAN has been reached on this evaluation SOW. Other logistical considerations include: availability/responsiveness of project developers, language issues, and the protection of sensitive personally identifying or financial information. 6.1 SCHEDULE USAID anticipates that the period of performance of this evaluation will be from January 2014 to March 2015. The recommended timeline is as follows: Month Activity January- April 2014 Request, receive, and read initial program documents Initial draft of evaluation work plan May 2014 Stakeholder Tracking Matrix delivered May 2014 Evaluation SOW delivered June 10th 2014 Feedback received on SOW from E3 June 17th 2014 dTS to provide workplanning Gantt and budget after concurrence on this SOW June 25th 2014 Draft survey instrument, interview guide, and initial email for blast for review and comment; schedule call with Peter June 27th 2014 Draft Evaluation Plan for review and comment July 4th 2014 Feedback received on survey instrument, interview guide, Stakeholder Tracking Matrix (as appropriate), and initial email from E3 and CTI July 11 th 2014 Feedback received on Evaluation Plan from E3 July 23rd 2014 Final Evaluation Plan Submitted August 4th Final survey instrument, interview guide, and Stakeholder Tracking Matrix agreed to by dTS, E3, and CTI July-August 2014 Submit any additional data requests to PFAN country offices and review material September 1st Initial email blast introducing the survey sent out by CTI September October 2014 Data Collection (Survey administration and key informant interviews) GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 12 Month Activity September 5th (and every two weeks thereafter through October) dTS follows up email blast with call scheduling for interviews September 15th (and every two weeks thereafter through October) dTS/CTI follows up with email blast reminder for survey September 8th October 31st dTS hold calls for key informant interviews October November 2014 Data Analysis December 12th 2014 Evaluation Report Draft for review and comment January 12th 2015 Feedback received on Evaluation Report from E3, and CTI (if appropriate) February 13th 2015 Final Evaluation Report February 2015 Dissemination Meeting(s) in Washington, DC March 2015 Delivery of all data and upload of final report into the DEC GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 13 Name and Qualifications Role Responsibilities Matthew Addison, MS Mr. Addison has extensive experience in CE finance, CE policy and regulation, as well as CE evaluation. He has managed teams on a variety of CE initiatives for national governments, donor agencies and private businesses. Team Leader Provide CE technical expertise for evaluation design and implementation Supervise survey and KII instrument creation and ensure data quality Supervise oversight of survey administration and analysis of data Translate data into findings, conclusions and recommendations Primary author of all reports and deliverables Provide leadership, coordination and supervision of team members on relevant evaluation activities, including documentation review, interviews, analyses and formulation of conclusions and models Marc Shapiro, Ph.D. Dr. Shapiro has extensive experience in project management and monitoring and evaluation (M&E). His technical specialization within M&E is in leading, designing and implementing impact evaluations and quantitative methods. Senior Technical Advisor Provide technical advice to the team Supervise the translation of the data to findings, conclusions and recommendations Serve as quality controller across evaluations Farhat Rahman, MS, MPA Mid-level consultant with substantial experience working in survey administration, data collection, analysis and management. Data Analysis Specialist Survey administration and quality control Data analysis Data interpretation Participate in or conduct KIIs GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 14 Technical methodological inputs to relevant deliverables Elizabeth Stahl, BA Ms. Stahl has supported the GCC M&E Project for its duration, providing research, report writing and editing, recruiting, and coordination efforts. Senior Program Associate Day-to-day/home office technical and operational management of evaluation Assist with creation and conduct piloting of survey instrument and interview guide Facilitate and support survey administration and KIIs Support writing and editing of reports GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 15 Evaluation Questions Planned Measures/ Indicators Data Collection Methods Data Sources Limitations 1. What has been the cost effectiveness of PFAN in relation to: a. Clean energy technology financing; b. Establishment and maintenance of lender/developer relationships; c. Leveraging private sector resources, such as in-kind services and mentoring; and d. Clean energy technology deployment? Cost effectiveness is a ratio: Megawatt per dollar of USAID funding5 funds leveraged per dollar and metric ton of CO2e 6 avoided per dollar USAID funds spent per project, Dollars spent per Financial Institution, And prospective investment per dollar. Number of projects funded (by technology type) per dollar Cost per project, and MT of CO2e avoided per dollar. Document review PFAN documents and administrative records CTI-PFAN website Operational stakeholders (Funding partners and resource partners) IPs PFAN participants (project developers, mentors, financial institutions) International financial institution (IFI)/Donor CE program reports Principal limitations are: 1. PFAN has not collected data on applicants who are not accepted into PFAN; this limits the ability to develop strong counterfactual scenarios. 2. Data are not collected on participants after their projects achieve financial closure. 3. Pipeline data collected and reported by PFAN is incomplete. 4. There have been no previous evaluations or in-depth examinations of PFAN on which to build the current analysis. Online survey Online survey 1. Sample size includes 159 respondents with uneven distribution across stakeholder types. 2. Only a few project developers reaching closure chose to participate; 3. No commercial bank or financial partners participated. 4. PFAN has not collected data on applicants that are not accepted into PFAN; this limits the ability to develop strong counterfactual scenarios. 5 Throughout the term per dollar is used synonymously with per dollar of USAID funding. 6 CO2e is the carbon dioxide equivalent or a metric used to standardize the different greenhouse gases. It is standard industry nomenclature. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 16 Evaluation Questions Planned Measures/ Indicators Data Collection Methods Data Sources Limitations  Key informant telephone and email interviews  Telephone interviews augmented by email response to questions depending on respondent preferences The timeframe for telephone interviews was extended to try to increase response rate of under￾reporting groups on the survey: financial partners and project developers reaching closure. However, few additional project developers responded as available, and responses to questions asked by a different medium are not necessarily directly comparable. 2. What are the strengths and weaknesses of the PFAN organizational and partnership model? Level of satisfaction with technical assistance (TA) and CB received Number of stakeholders reached with CB and TA, disaggregated by gender Be utility of received/provided PFAN services by sub-group, type of TA, gender Strengths of PFAN observed, by sub-group, gender Weaknesses of PFAN observed by sub-group, gender Measurable level of cooperation between operational stakeholder organizations (OSO) and among OSOs and IP Effectiveness of and satisfaction with the: - Contributions and accountability of each participating USG agency - Current structure of inter-agency relationship under PFAN - The mode in which interagency relationships facilitate goal Online survey Key informant telephone interviews PFAN documents and administrative records CTI-PFAN website Operational stakeholders (Funding partners and resource partners) IPs PFAN participants (project developers, mentors, financial institutions) Online survey See above GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 17 Evaluation Questions Planned Measures/ Indicators Data Collection Methods Data Sources Limitations attainment in each agency - Coordination between agencies 3. In what ways and to what extent do project developers credit PFAN for having successfully secured financing for their projects? Respondents credit PFAN for results Finance entities credit PFAN for results Respondents undertake additional CE investments7 ; Finance entities move into non-PFAN CE funding8 Document review Online survey Key informant telephone interviews PFAN documents and administrative records CTI-PFAN Website Operational stakeholders (Funding partners and resource partners) Implementing partners PFAN participants (project developers, mentors, financial institutions) Online survey See above 4. Are there barriers that are not being addressed or that are not being addressed effectively? i.e., what additional assistance can PFAN offer under its mandate? CE penetration rates Ratio of PFAN projects funded Beneficiary responses Non-PFAN participant CE project developers or financier track record or responses Document review Online survey Key informant telephone interviews PFAN documents and administrative records Operational stakeholders (Funding partners and resource partners) IPs PFAN participants (project developers, mentors, financial institutions) IFI/Donor CE program reports Online survey See above 7 Too few closed projects responded to the survey to make this meaningful. 8 This was not applicable as all financial entities responding to the survey had already been engaged in clean energy prior to working with PFAN. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 18 Evaluation Questions Planned Measures/ Indicators Data Collection Methods Data Sources Limitations 5. What is the performance of PFAN participants, post financial closure, including reaching and maintaining operational status, replicating or expanding business and producing co￾benefits for themselves or their community? Percent of operating projects to closed projects; Reasons for not starting operations Percent developing new CE projects Percent expanding business Types and amounts of co benefits produced Document review Online survey Key informant telephone interviews Online survey See above 6. What are ways PFAN can change or update the implementation, either in its current form or as the program transitions to a stand-alone entity? Country representation, sustainability (based on a number of factors specified in the data collection instrument) Perceived degree of coordination and cooperation (based on a number of factors specified in the data collection instrument) Level of satisfaction with CB and TA received Partner beneficiary perceptions of strengths and weaknesses of current PFAN approaches Perceptions on the optimal lifespan of the PFAN program Online survey Key informant telephone interviews In situ in￾person/remote interviews with operational stakeholders See above GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 19 The dTS current evaluation team includes a team leader (Matthew Addison), senior program associate (Elizabeth Stahl), senior technical advisor (Marc Shapiro), and data analyst (Farhat Rahman). Team members were selected based on their experience in energy and evaluation. For example, the team leader is an acknowledged CE expert and has conducted numerous CE evaluations and designs. 1.1 CHARACTERISTICS OF SURVEY RESPONDENTS This section discusses the demographics of the three groups of survey respondents on which the project is mainly focused: developers, coaches, and FIs. Project developers9 are a primary focus of the evaluation since they receive the bulk of the technical assistance, business training, and support from PFAN. Understanding both the type of developer that PFAN attracts and those that participated in the survey and interviews is important in understanding the results of this evaluation. 1.1.1 Within the context of the PFAN evaluation, there are two distinct populations: (1) the project developers, where N = 468, and (2) the operational stakeholders, mentors/coaches, resource partners, and FIs, where N = 218. The population of project developers is distinct with no known overlaps with the other sub￾groups. However, there exists substantial overlap among potential respondents from the other four sub￾e evaluation retroactively determines separate EBPD sub-group response rates based on the self￾categorization of survey respondents during the survey process. In some cases, multiple representatives from an institution were contacted for the survey. Due to low sample sizes, each respondent from the same institution is given the same weight as a single respondent from another institution. Therefore, an institution with multiple respondents could be represented multiple times in the population. The evaluation team relied on CTI to provide candidates for the KIIs that are, to the extent possible, representative across the various countries and regions where PFAN is operating. PFAN is currently operating in 15 Asian countries, 24 African countries, 13 Latin American and Caribbean countries, four countries within the Commonwealth of Independent States and Central Asia, and 12 countries that do not fall into those categories. The online survey was emailed to all participants who had been identified by PFAN. The evaluation team sent out 687 emails that included the survey invitation and instructions, with 674 invitations delivered successfully, generating a total of 156 responses. Table 1 indicates the response rate relative to the overall population size. Table 1: Survey Characteristics 9 Project developers include individuals, companies, communities, and NGOs. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 20 PFAN Entity Number of Stakeholders receiving survey invitation Number of Survey Respondents Survey Response Rate Project Developers 468 111 23.7% Project Developers Reaching Financial Closure 44 2 11.4% Financial Institutions 68 5 7.4% Operational Stakeholders 57 20 35.1% Coaches 46 11 23.9% Resource Partners 48 9 18.8% Source: PFAN Survey Operational stakeholders included representatives from USAID, CTI, ICEETT, United States Department of State, the Renewable Energy and Energy Efficiency Partnership, the Energy and Climate Partnership of the Americas, International Development Research Centre, and other PFAN managerial and funding stakeholders. Representatives from these organizations were interviewed in order to obtain insights into the multi-lateral interagency partnership, management and operations, and program design and execution. The team interviewed only those that identified as willing to be interviewed. Coaches included both individuals and organizations that provide advisory and capacity building services to project developers and businesses. Coaches were interviewed in order to ascertain their perspectives on the efficacy of the PFAN approach to mentorship, usefulness of services, addressing barriers to securing finance, and overall successes and challenges of PFAN. Project developers included individuals and businesses that are seeking or have sought access to PFAN coaching and financial investment matching services. Project developers were interviewed in order to gain insight into perceived efficacy of PFAN services, including coaching and financial investor matching. Resource partners tend to be government agencies in developing countries that have been set up to support CE investments in their countries. They help to ensure that efforts are aligned with governments and project objectives. They provide local context and support in terms of investment climate and CE technology needs and gaps. Financial institutions include representatives from banks, investment funds, and lending organizations. Representatives from FIs were interviewed in order to gain further insight into the barriers to CE investment, the efficacy of the PFAN approach to addressing these barriers, and individual perceptions of the usefulness of PFAN overall as an approach to bridging the gap between CE projects and investment. 1.1.2 There are three broad categories of developers developers with closed projects, developers still in the pipeline and developers that were deemed not qualified very early on and excluded from PFAN assistance. PFAN reports some data for the first two categories but does not collect data projects that are rejected as not qualified. However, PFAN had the email addresses of many African and a few Asian projects that were GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 21 unsuccessful applicants in recent RFP processes for respective regional financing fora. Numerous developers in this category completed the survey. Their responses are not considered in this section and later where they do not have knowledge of an item. When their views are considered, it is noted separately from those of PFAN developers who have been accepted into the program. Based on the survey results, the typical PFAN developer that received USAID support is a male only-owned company (57 percent)10 that has pursued multiple CE projects (75 percent) and has sought financing from other sources before his company approached PFAN (74 percent). Slightly more than one out of three (37 percent) have been successful in obtaining financing outside of their participation with PFAN. Many of the developers did not report financial data, but based on the 44 developers reporting financial in the survey, the total amount of investment required is $1.5 billion, and the average PFAN project costs $34.8 million. These figures are outlined in Table 2, which provides the financial statistics of the overall pipeline and the survey respondents. Table 2: Summary Financial Statistics Survey Respondents Total Current Pipeline Number of Project Developers n = 44 n = 255 No. Reaching Financial Closure 4 49 Total Investment $1,529,383,945 $6,682,968,215 Average Investment $34,758,726 $26,310,898 Minimum Investment $20,000 $100,000 Maximum Investment $250,000,000 $830,000,000 Standard Deviation $69,986,507 $83,860,687 Source: PFAN survey Additionally, Table 3 provides a comparison between the survey and pipeline data based on technology (left three columns) and geography (right three columns). The reported results from the survey are quite different from PFAN operations to date in two primary ways. First, closed projects are under-represented. Forty-nine projects have reached closure and another 255 are in the pipeline. This means that at the time of this evaluation 16 percent of PFAN projects are closed. Only four survey respondents had closed. Second, the geographic representation is the survey is very different from the PFAN population. There is wide geographic difference between the survey population and the total population of PFAN project developers. Asia is significantly underrepresented while Africa countries have responded heavily. Table 3 presents summary statistics for the PFAN pipeline and the survey respondents. Table 3: PFAN Developer Demographics 10 Female-owned companies constitute 11% of the survey respondents, and the remainder (32%) are entities that are jointly owned between males and females. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 22 By Technology PFAN Pipeline Survey espondents By Region PFAN Pipeline Survey Respondents Biofuels 11.0% 10.3% S Asia 13.2% 18.1% Biomass 13.6% 20.6% Asia Pacific 0.5% 0% Biogas 11.8% 11.3% China 8.6% 0% W2E 4.4% 14.4% SE Asia 25.0% 9.0% Clean Transport 2.2% 1.0% CISCA 0.5% 1.3% Hydro 20.6% 7.2% East Africa 21.4% 24.2% Wind 4.4% 6.2% Southern Africa 19.1% 20.5% Solar 17.1% 17.5% West Africa 5.0% 17.9% EE 9.2% 11.3% Latin America Brazil 2.7% 0% Other 5.7% CAC 4.1% 9.0% 100.0% 100.0% 100.0% 100.0% It is inadvisable, based upon the data available, to draw conclusions about technology differences other than that they exist. Drivers for technology choice are natural resource availability, competition from other energy sources, enabling frameworks and incentives, and market needs. For example, it is expected that solar photovoltaic (PV) would have a better chance of succeeding in the CAC or East Asia region than in Africa because the average income in CAC or East Asia is much higher than in Africa, and the people of CAC or East Asia are generally already higher on the energy ladder than those in Africa.11 1.1.3 Only five representatives of FIs responded to the survey: two development finance institutions, one private investor, a private equity fund, and a broker organization. Commercial banks were completely missing from the survey but also appear to be only minimally involved in PFAN. The data provided by PFAN did not indicate any commercial banks involved in the financial closures to date. Four out of five respondents offer both loans and equity, and one provides only equity financing. All were involved in the CE finance business prior to participating in PFAN. When asked why they choose to participate with PFAN if they were already providing CE financing, the principal reason c attain financial closing. 11 Based on analysis by dTS team. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 23 1.1.4 Eleven coaches out of 46, representing 24 percent of the PFAN coaching population, participated in the survey. Female coaches represented 18 percent of the respondents, as compared to 9 percent of the total PFAN coaching population. Table 4 indicates the countries in which the respondent coaches have worked with PFAN and the percentage these countries represented in closed projects and pipeline projects. The responding coaches have experience in countries that account for 52 percent of closed projects and 57 percent of pipeline projects. Table 4: PFAN Coach Demographics Project Location Coach Working Location % Total Closed Projects in Country % Total Pipeline Projects in Country BW - Botswana 2 4% 0.7% CV - Cape Verde 1 - 0.3% ET - Ethiopia 1 - 0.7% GT - Guatemala 1 - 0.0% IN - India 2 10% 11.5% KE - Kenya 2 4% 7.6% MZ - Mozambique 1 2% 4.9% NA- Namibia 1 - 0.7% NP - Nepal 1 - 0.7% PH - Philippines 1 20% 14.9% TH - Thailand 1 2% 1.7% TZ - Tanzania 1 4% 1.4% UG - Uganda 1 6% 6.6% ZA - South Africa 1 - 5.2% Eleven coaches reported country location. 1.2 RESEARCH DESIGN The evaluation uses a convergent parallel mixed method design that allows for the collection of qualitative and quantitative data concurrently as well as analysis of the two data sets separately. This approach considers several perspectives and reduces gaps in information. The quantitative instruments are needed for obtaining and analyzing generalizable statistical data. Qualitative instruments are well suited for explaining processes and impacts. Together, the two parts provide both analytical and explanatory power, while validating the findings through triangulation of data from multiple sources. Given the challenges with data collection as well as the importance of learning lessons, as opposed to merely measuring differences, this mixed method approach is considered both appropriate and essential. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 24 Four data sources were used to collect the cross-sectional data. The two primary sources were qualitative KIIs centered on subjective data and a confidential, primarily quantitative, electronic survey open to most PFAN stakeholders. The two secondary sources were a document and literature review, and information provided by CTI and obtained from the PFAN website. The various data sources are described in further detail in the following sub-section. 1.3 DATA COLLECTION METHODS The evaluation design matrix (Error! Reference source not found.) below provides an overview of the evaluation questions, measures and indicators, methods, and data sources required to answer those questions. The list of indicators provides the background context on the types of information that the evaluation collects. 1.3.1 Administrative and institutional data include documents provided to dTS by either CTI or USAID that website and has collected other relevant documents, reports, and publicly available data. In addition to the administrative and institutional data described above, dTS carried out a desk review of relevant project-specific documents provided by USAID and CTI or collected by dTS. 1.3.2 The evaluation used two types of primary data collection instruments: an online survey instrument and a KII guide. Using SurveyMonkey, an online software tool, the evaluation team developed a survey for project developers, coaches, operational stakeholders, resource partners, and financial institution (FI) representatives.12 question directed the respondent to specific subsequent questions) and customized to gather information specific to each of the subgroups. The logic piping within the survey allowed for greater control over which questions each respondent saw and directed them to answer only the survey questions relevant to their sub-group. At the end of the survey instrument, respondents could choose to participate in a KII. The evaluation team contacted respondents who provided email addresses to schedule interviews. The KII guide consisted of general questions for all informants and the distinct and targeted interview guides for each of the sub-groups opinion of the strengths and weaknesses of PFAN, their perspectives on the effectiveness of PFAN in fulfilling its objectives to increase access to CE financing and accelerate technology transfer, and their suggestions for improving the program. 12 Coaches are individuals who provide business coaching to developers on a cost-share basis. Resource partners are organizations, such as non-governmental organizations or government-owned CE entities, which have a vested interest in promoting CE and see PFAN as assisting them in achieving their objectives. Operational stakeholders are donors, implementers, country coordinators, and other PFAN project personnel. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 25 1.3.3 An analysis map was developed that linked each evaluation question and/or sub-question to the corresponding survey question(s) for quantitative data and to the corresponding KII guide question(s) for related qualitative data. As question and item numbering varies somewhat across population sub-groups, there was a separate, but related, mapping for each sub-group in order to efficiently organize data collection and map correlations across the population as a whole. analyze data. Quantitative data from the online survey were analyzed for descriptive statistics (e.g. frequencies, means, and medians) and for correlations of key responses with exposure to specific PFAN activities, individual roles, and responsibilities. Qualitative data provided as part of the online survey was used primarily for descriptive, anecdotal information and further analysis, depending on the length of the narrative. Qualitative data collected as part of KIIs were analyzed in depth for emergent patterns of perceptions, representations, portrayals of utility and effectiveness, and aspects that suggest potential sustainability. 1.3.4 In￾d quantitative data in terms of attitudes and behaviors. The evaluators identified and conducted interviews with relevant operational stakeholders, coaches, project developers, resource partners, and FI representatives. The KIIs were used to provide additional detail in areas that could not be adequately dealt with in the survey. Seventy-one persons self-identified their stakeholder status as indicated in Table 5. Table 5: Number of Self-Identifying KII Operational Stakeholders Coaches Resource Partners Developers Financial Partners Total Number Volunteering 9 4 2 55 1 71 Number Contacted 9 4 2 48 2 65 Number Interviewed 4 4 2 22 2 34 Of the developers that responded to the request for interviews, two had reached financial closure, and a third was listed in the pipeline as having reached financial closure but funds had not yet been disbursed? 1.4 DATA QUALITY nstruments and methodologies in order to meet the expectations for data quality specified in the USAID Automated Directives System 203, Assessing and Learning, namely validity, integrity, reliability, precision, and timeliness. uation team to promote data quality include pre-testing of survey instruments, GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 26 performing multiple mock KIIs, training and close supervision of enumerators as needed, data entry controls, qualitative data recording, summarizing, transcribing, and use of mixed methods. 1.5 STRENGTHS AND LIMITATIONS OF EVALUATION METHODS 1.5.1 Within the scope of the PFAN performance evaluation, there are three potential conflicts of interest (COIs) that have arisen and are being mitigated. The first COI is that two members of the evaluation team routinely work in the CE field and may have access to proprietary data. To avoid this potential COI, these members have signed nondisclosure agreements and do not have access to data with identifiers such as name or email. Relevant data are stripped of this information except (1) where respondents have self￾identified that they are willing to be interviewed and thus the results of the interview would be available; and (2) where PFAN has identified other interview participants. Outside of the team, there are two other COIs. One potential COI is that USAID is a funding partner of PFAN as well as a managing client of dTS, specifically the GCC M&E project that is carrying out this evaluation. In this role, USAID has the opportunity to provide both technical and managerial direction to the evaluation team and their work products. The second potential COI is that CTI, through directive from USAID, has been provided the opportunity to ork products including the survey instrument and key informant interview guide. In addition, CTI is responsible for providing the evaluation team with contact information for all PFAN stakeholders and participants who have been involved with the pipeline projects. The evaluation team is unable to independently verify the data provided by CTI. To mitigate any potential COI that may affect research or findings, the evaluation team has not altered any document, report, instrument, or guide in a way that substantively changes the data to be collected and/or the findings to be disseminated based on feedback from either USAID or CTI. The evaluation team was not able to independently verify the population characteristics within each stratum, as some individuals may have been included in more than one category. Therefore, it cannot be stated with certainty that the probability sample derived from each group is representative of the population. As discussed in the next sub-section, precision of the estimates for online surveys may also be an issue. The specific findings of the evaluation may not apply to the diverse PFAN population. However, since a mixed-methods approach is used, a degree of certainty can be attached to more general findings and recommendations presented to CTI and USAID because they will be supported through more than one data source. Many of the metrics in this report will not accurately reflect the long-term potential since PFAN is still an on-going project. Two examples of this discrepancy include the following. Cost-effectiveness. The cost effectiveness of the activity, in terms of investment leveraged, MWs, and GHGs avoided, is provisional in that it measures investment leveraged and MWs at reported financial close13, not at project operation. GHGs avoided are based ex ante on assumptions for capacity factors and using default emission factors, and do not represent verified ex post reductions. Additional project benefits 13 PFAN does not report at financial closure but rather when their assistance to project developers end; when there is an agreement to finance but not when actual financing is consummated. GCC M&E Appendices to the Performance Evaluation Final Report: CTI PFAN 27 not currently represented in the pipeline data could result from scale up and operation, resulting in an Sustainability and Replicability. It is difficult to measure sustainability and replicability in a reliable manner in the short run. Many of these projects have been undertaken by new businesses and it is hard to determine now whether they will weather the course of time and be sustainable. Measuring replicability is complicated by the fact that it takes time to build capacity, change minds, penetrate markets, and have an impact on business and financial institutions. 14 TEMPLATE: INFORMED CONSENT FOR INTERVIEW Same for all the following interview guides; included in this document only once. ROLE or DESIGNATION: ________________________ NAME: _________________________ Advisory Network (PFAN) for the purpose of finding out how well the program has been meeting the needs of internal and external stakeholders like you. You are invited to participate, on a free and voluntary basis, in an interview on how various aspects of the PFAN are working. This interview is not compulsory and you may withdraw your consent to participate at any time before or during the interview without negative consequences. You can choose not to answer any or all questions. If you agree to participate, you will be asked a number of open-ended questions. This interview should not take more than 60 minutes to complete. We can discuss any of your concerns during your interview. Following the interview, we may contact you again in several days to confirm information. We will need to record the session in order to capture points made in response to the questions. The discussion will be transcribed, all digital recordings and notes will remain confidential, and will be kept in a secure place. The recording of your verbal responses will be used for analysis only and will not be distributed beyond the evaluation team. Your name will not be identified or linked to any quotes in any public reports summarizing the findings of these interviews, unless you request it. Unless specified otherwise, a list of all interviewees will be included as an annex. Do you prefer to remain anonymous? Yes  No  If you have further questions about this project, the results of this study, or if you have a research-related problem, you may contact any one of researcher team members listed here: Rayne Loken (RLoken@onlinedts.com); Matthew Addison (mwaddison@cox.net). The evaluation report is expected to be completed in January 2014 and will be circulated to the PFAN participating agencies, USAID/GCCO, Climate Technology Initiative (CTI), and other stakeholders shortly thereafter. The report can also be shared with all those interviewed, upon request. Do you agree to participate in this study? Yes  No  [For the interviewer/administrator] I have followed the agreed evaluation protocol to obtain consent from the participant. S/he apparently understands the nature and the purpose of the study and consents to participate. S/he has been given the opportunity to ask questions, which have been answered satisfactorily. signature: _____________________________________ DATE ______2014/2015 14 This appendix satisfied the contractual requirement for documentation of tools/methods used for estimation/calculation of GCC outcomes. PROJECT DEVELOPER KEY INFORMANT INTERVIEW GUIDE For developers with projects accepted to the pipeline  Which sector did your project target? a.) Waste to Energy b.) Clean Transport c.) Biofuels d.) Biogas e.) Solar f.) Hydro g.) Energy Efficiency h.) Biomass i.) Other:___________________________________________________________________  Please list the size and unit of measure of this project. Possible units of measure include: MW, KW, cubic meters, cubic feet, tonnes per year, and litres per year. Size___________________ Unit of Measure___________________ What was the total cost of this project in US Dollars?________________________________________  What was the value of your equity contribution in US Dollars?________________________________  When did your project reach closure (Month and Year)?  How long did it take for you to reach closure once you had been accepted into the PFAN pipeline (in months)?  At what stage was your project when it was proposed to PFAN? a. Concept b. Business plan developed c. Some investment committed d. Other: Please Describe?  Why do you believe that your project was accepted into the PFAN program?  Do you think the selection process could be improved or are you happy with the time and information required to be considered and then selected to participate?  What do you think could make the selection process better?  In your experience, how effective have PFAN efforts been for you in terms of: a. Technical and engineering assistance b. Preparation and presentation of investment proposals c. Establishment and maintenance of lender/developer relationships d. Finance structuring, and sourcing and procuring financing  Was the mentor an appropriate match for your project?  In what ways and to what extent do you credit PFAN for successfully being matched with an investor?  What are the three most important barriers to clean energy investment? (1) ______________________________, (2) _________________________, (3) ______________  What are the barriers to clean energy investment that PFAN is successfully addressing? (1) ______________________________, (2) _________________________, (3) ______________ (4)  What are the barriers to clean energy investment that PFAN is NOT successfully addressing? (1) ______________________________, (2) _________________________, (3) ______________  In what ways and to what extent do you credit PFAN for assisting you to successfully secure financing?  What are likely to be the lasting impacts of PFAN on your project?  What are likely to be the lasting impacts of your project as a result of PFAN?  Would you recommend using the PFAN resources to someone else who is trying to secure financing for a CE project? Why or why not? In your experience, were private sector financial institutions engaged in an effective manner? Are there ways to increase their participation in PFAN?  Access to finance has been identified by multiple organizations as one of the primary barriers to clean energy technology deployment. To what extent do you believe PFAN has successfully addressed this barrier?  Are there barriers (to securing financing) which are not being addressed or that are not being addressed effectively? i.e., what additional assistance can PFAN offer under its mandate?  PFAN is looking at ways to keep it going once donor funding has stopped. Would you be willing to pay a success fee for the PFAN services?  If so, how much would you be willing to pay?  Would you be willing to pay for coaching services?  Would you be willing to pay for the Investor networking services?  Does your company have any programs that are designed to deal with gender issues or promote gender is hiring?  In designing your project, did you take into account the needs of women, children or the elderly? For example, a cookstove project will generally benefits women and children more than men.  To summarize, what would be some specific recommendations and lessons you might suggest in order to enhance PFAN performance moving forward?  On what date (Month and Year) did your project start commercial operations?  Is it still operating?  If not, why not?____________________________________________________________________  If it has not started commercial operations, why not?______________________________________  Have you or will you undertake another clean energy project?  If you have already begun another clean energy project, what is the sector, size and units and total costs?  Is there anything else you want to tell us about PFAN? PROJECT DEVELOPER KEY INFORMANT INTERVIEW GUIDE ALTERNATIVE For developers with projects deemed not qualified/not accepted to the pipeline  What is your position within that organization?  Was your proposal unsolicited or did you send it in response to a call for proposals?  Was your project greater than US $1 million and less than US $50 million?  At what stage was your project when it was proposed to PFAN? e. Concept f. Business plan developed g. Some investment committed h. Other: Please Describe?  Why was your project not accepted into the PFAN program?  Based on your experience with PFAN, what do you think your organization/project could have done differently in order to successfully secure financing in the future?  What do you believe PFAN should do differently to better address barriers facing projects like yours?  Have you been able to find financing?  Did your participation in the PFAN process, help you improve your project?  If you have not found financing, will you improve your project and resubmit to PFAN?  If not, why not?  Would you recommend using the PFAN resources to someone else who is trying to secure financing for a CE project? Why or why not?  Do you have any suggestions as to how the PFAN program can changed or updated in order to better suit the needs of participants?  To summarize, what would be some specific recommendations and lessons you might suggest in order to enhance PFAN performance moving forward?  Is there anything else you want to tell us about PFAN? FINANCIAL INSTITUTION KEY INFORMANT INTERVIEW GUIDE We are interested in your experience with PFAN.   Objective 1: Broaden the access to financing for climate friendly and technology transfer projects in the RE/ EE sectors in developing countries and economies in transition.  Objective 2: Get more renewable energy and climate friendly projects financed in the private sector and thereby accelerate technology transfer.  F , t  From your perspective, do you think PFAN successfully addresses country-specific needs for clean energy technology? Does PFAN successfully address country-specific needs for clean energy financing? Could you provide examples or explain why not?  Do you have any suggestions as to how the PFAN program design or implementation strategy can changed or updated in order to better suit the needs of participants?  To summarize, what would be some specific recommendations and lessons you might suggest in order to enhance PFAN performance moving forward?  Is there anything you have observed or that you anticipate might hinder progress toward achievement  Can you provide examples?  What about the scope of the program? (Thematic and geographical scope)  Is there anything you have observed or that you anticipate might facilitate/support the achievement objectives?  Can you provide examples?  What about the scope of the program? (Thematic and geographical scope)  Do you have experience engaging with other PFAN partners/agencies?  If Yes, were the relationships well managed?  What was the effect of the multilateral approach on efficiency and effectiveness of projects working with PFAN?  In your experience, how effective have PFAN efforts been for you in terms of: a. Establishment and maintenance of lender/developer relationships b. Finance structuring, and sourcing and procuring financing c. Preparation and presentation of investment proposals  In what ways and to what extent do you credit PFAN for successfully being matched with a project developer? In your experience, were private sector financial institutions engaged in an effective manner? Are there ways to increase their participation in PFAN?  Access to finance has been identified by multiple organizations as one of the primary barriers to clean energy technology deployment. To what extent do you believe PFAN has successfully addressed this barrier?  Specifically, the single most identified barrier was collateral was too high? What can be done to reduce collateral requirements?  Do you have the same collateral requirements for projects that are part of the PFAN program as you would/do for those that are not?  Among Financial Institutions, the lack of favorable government policies was the most important barrier. Do you agree and if so, can you explain what these are and how PFAN might address these?  Have the opinions of financial institutions been taken into consideration in PFAN design and implementation? Could you provide examples or provide your thoughts on why or why not?  Will your investment contribute to positive advances in clean energy technology?  Beyond the additional projects that were brought to you through PFAN, what additional value has it been to your entity?  In the countries where you have participated with PFAN, do you seen any lasting changes resulting from t will continue to promote clean energy finance once PFAN is closed?  Would you be willing to pay PFAN a success fee to help cover its costs?  Will your company continue to engage with PFAN? If not, why not?  Does your company have a gender program?  Has it  Is there any preference in lending to woman owned projects?  Are any of your PFAN projects, woman owned?  Are any of your professional staff women? OPERATIONAL STAKEHOLDER KEY INFORMANT INTERVIEW GUIDE We are interested in your experience with PFAN.   What is your position within that organization?   Objective 1: Broaden the access to financing for climate friendly and technology transfer projects in the RE/ EE sectors in developing countries and economies in transition.  Objective 2: Get more renewable energy and climate friendly projects financed in the private sector and thereby accelerate technology transfer under the UNFCCC.  What role have you played in accomplishing specific PFAN objectives in the following areas? i. Capacity Building and mentorship: Workshops/training ii. Clean Energy Project Development iii. Financial investment iv. PFAN Program administration, planning, and development v.   To what degree have program accomplishments been captured and communicated to internal and external stakeholders?  From your perspective, do you think PFAN successfully addresses country-specific needs for clean energy technology? Does PFAN successfully address country-specific needs for clean energy financing? Could you provide examples or explain why not?  Do you have any suggestions as to how the PFAN program design or implementation strategy can changed or updated in order to better suit the needs of participants?  To summarize, what would be some specific recommendations and lessons you might suggest in order to enhance PFAN performance moving forward?  Are there any other key stakeholders that you recommend we talk to, who could provide an insight and valuable input into the PFAN evaluation? (Request name, Agency/Affiliation/Contact information)  Access to finance has been identified by multiple organizations as one of the primary barriers to clean energy technology deployment. To what extent do you believe PFAN has successfully addressed this barrier?  Have the opinions of technical experts and/or host country and regional leadership have been taken into consideration in PFAN design and implementation? Could you provide examples or provide your thoughts on why or why not?  How effective has the Climate Technology Initiative (CTI) as the coordinating body, been at developing implementation strategies, monitoring progress, documenting lessons learned, general planning, etc.?  Why do think that is? Could you provide examples to illustrate?  From your perspective, how effective have PFAN efforts been in achieving: a. Clean energy technology financing b. Establishment and maintenance of lender/developer relationships c. Leveraging private sector resources, such as in kind services and mentoring d. CE technology deployment?  Why do you believe certain PFAN projects been more or less successful in achieving financial closure?  Is there anything you have observed or that you anticipate might hinder progress toward achievement  Can you provide examples?  What about the scope of the program? (Thematic and geographical scope)  Is there anything you have observed or that you anticipate might facilitate/support the achievement  Can you provide examples?  What about the scope of the program? (Thematic and geographical scope)  In your view, how effective is the multilateral approach to funding for PFAN? (By this it is meant multiple entities that finance PFAN, each with different and overlapping objectives) Does this relationship seem to promote or hinder PFAN implementation?  How do you view the existing interagency structure and coordination? (By this it is meant the role of the various funding entities)  hink that is?  Has anything unexpected (positive or negative) come out of the current structure?  How well has the administrative structure worked (Interagency committee, technical team, country teams)?  Do you have experience engaging with other PFAN partners/agencies such as ADB, REEP or USAID, ?  If Yes, were the relationships well managed?  What was the effect of the multilateral approach on efficiency and effectiveness of projects working with PFAN?  [FOR USAID] What have been the USAID-specific challenges with engaging with other partners and how have these impeded overall project accomplishments (if they have)?  To what degree do the roles of PFAN USG partners in decision-making and implementation relate to  Why do think that is? Could you provide examples to illustrate?  design process and during implementation?  To your knowledge, does PFAN currently monitor progress and track performance? To this end, have benchmarks or targets for success been established and are data collected on the indicators to track progress? If so:  Can you provide some examples of indicators or targets that are being tracked?  Can you explain how these data are being collected and managed?  What agency(ies)/partners provide oversight to this process?  Is a centralized information management system in place where activity/project data (performance metrics), financial records, grantee information, or administrative affairs for the whole of PFAN in place?  If Yes, which partner(s) manage this system?  If No, do you feel that this could be helpful to ongoing program management?  Do you have any comments or suggestions concerning this system?  Have the lessons learned during PFAN implementation been used for decision-making to improve program implementation? Have these lessons been recorded? How effective would you say this has been? Why or why not?  In your experience, were private sector financial institutions engaged in an effective manner? Are there ways to increase their participation in PFAN?  Are there systems in place to support a transition for PFAN to become a stand-alone entity (absent of external support)?  What are the strengths and weaknesses of the PFAN organizational and partnership model? Gender-Specific Questions  Has PFAN developed a gender sensitive program approach to assist alternative energy and energy efficiency project developers, businesses and consumers?  If so, please explain:  Empowerment?  If so, please explain:  Has PFAN reduced gender gaps relating to: o ? o PFAN assisted businesses that have been successful in getting financing?  Have businesses that have benefited from PFAN developed gender sensitive personnel, product development and marketing systems as a result of PFAN assistance?  Are businesses that have benefited from PFAN meeting gender sensitive consumer needs, demands and standards as a result of PFAN assistance?  Going forward, what actions should PFAN take to achieve gender equality results? MENTOR/COACH KEY INFORMANT INTERVIEW GUIDE We are interested in your experience with PFAN.  What is your position within the organization?   Objective 1: Broaden the access to financing for climate friendly and technology transfer projects in the RE/ EE sectors in developing countries and economies in transition.  Objective 2: Get more renewable energy and climate friendly projects financed in the private sector and thereby accelerate technology transfer under the UNFCCC.   From your perspective, do you think PFAN successfully addresses country-specific needs for clean energy technology? Does PFAN successfully address country-specific needs for clean energy financing? Could you provide examples or explain why not?  Do you have any suggestions as to how the PFAN program design or implementation strategy can changed or updated in order to better suit the needs of participants?  As a mentor, did you participate in the direct capacity building efforts of PFAN? Can you describe what those efforts were, where they took place, and with whom?  What observations and impressions can you share?  Do you think, given your expertise and experience, your match to this project developer was appropriate?  In what ways and to what extent do you credit PFAN for successfully being matched with a project developer?  Why do you believe certain PFAN projects been more or less successful in achieving financial closure?  Have the opinions of technical experts and/or host country and regional leadership have been taken into consideration in PFAN design and implementation? Could you provide examples or provide your thoughts on why?  Coaches participating in the online survey indicated that the most important barrier was inexperienced project developers. Clearly, coaches are one thing that PFAN can do to correct this. What else could PFAN do to reduce or eliminate this barrier?  Access to finance has been identified by multiple organizations as one of the primary barriers to clean energy technology deployment. To what extent do you believe PFAN has successfully addressed this barrier?  Specifically, the single most identified barrier was collateral was too high? What can be done to reduce collateral requirements?  Is there anything you have observed or that you anticipate might hinder progress toward achievement  Can you provide examples?  What about the scope of the program? (Thematic and geographical scope)  What about the multi-agency composition of PFAN?  Is there anything you have observed or that you anticipate might facilitate/support the achievement  Can you provide examples?  What about the scope of the program? (Thematic and geographical scope)  What about the multi-agency composition of PFAN?  Has PFAN developed a gender sensitive program approach to assist alternative energy and energy efficiency project developers, businesses and consumers? If so, explain.  Empowerment? If so, explain.  Has PFAN reduced gender gaps relating to:   PFAN assisted businesses that have been successful in getting financing?  Have businesses that have benefited from PFAN developed gender sensitive personnel, product development and marketing systems as a result of PFAN assistance?  Are businesses that have benefited from PFAN meeting gender sensitive consumer needs, demands and standards as a result of PFAN assistance?  Going forward, what actions should PFAN take to achieve gender results? PFAN-Sourced Documents Request for Proposals (RFP) for the Regional Clean Energy Investment Project; Complete list of PFAN stakeholders and contact information; PFAN presentations; Business Plans, Project Data Sheets and Presentations for the following three PFAN Projects: Cleanopolis, Biofuel Soil Ltd Ghana, and Primavera City; PFAN Call for Proposals (Various); The PFAN Website; PFAN Quarterly Reports; PFAN Project Pipeline; Other Documents Brown, M. et al. (SWEEP) (2009). Recent Innovations in Financing for Clean Energy. EBRD. (2010). Sustainable Energy Financing. Facilities EBRD. (2010). Sustainable Energy Initiative. E+Co International Finance Corporation Justice, S. (SEFI UNEP). (2009). Private Financing of Renewable Energy -. A Guide for Policy Makers. Maclean, J. et al. (SEFI UNEP). (2008). Public Finance Mechanisms to Mobilise Investment In Climate Change Mitigation. An Overview of Mechanisms being used today to help scale up the climate mitigation markets, with a particular focus on the clean sector. Assessment & Evaluation in Higher Education. 33 no. 3 (2008): 301 314. Penwarden, Rick, Response Rate Statistics for Online Surveys -What Numbers Should You be Aiming For? FluidSurveys, (October 8, 2014). Market Research and MCH Strategic Data, (February 2012). 12. Poyry. (2008). International Finance Corporation. Mid-term Evaluation, Russia Sustainable Energy Finance Program. Stanbic Bank s.a./Celtel Uganda. (2008). Investment Review Memorandum. SWEEP. (Brown&Conover). (2009). Recent Innovation in Financing for Clean Energy. UNEP. (2004). Financial Risk Management Instruments for Renewable Energy Projects. Summary Document. UNEP. (2004). Financial Risk Management Instruments for Renewable Energy Projects. UNEP. (2007). Assessment of Financial Risk Management Instruments for Renewable Energy Projects. UNEP Working Group 1 Study Report. UNEP. (2008). Public Finance Mechanisms to Mobilise Investment in Climate Change Mitigation. UNEP. (2009). Catalysing low-carbon growth in developing economies. Public Finance Mechanisms. UNEP. (2009). Private Financing of Renewable Energy A Guide for Policy Makers. UNEP. (2009). Why Clean Energy Public Investment Makes Economic Sense The Evidence Base. UNEP. (2010). Global Trends in Sustainable Energy Investment 2010. UNEP. (2010). Publicly Backed Guarantees As Policy Instruments to Promote Clean Energy. USAID. (2000). Credit Guarantees. Promoting Private Investment in Development. 13.USAID, (2013). The USAID/Eco-Asia Clean Development and Climate Program Evaluation. World Bank. (2008). Financing Energy Efficiency. Lessons from Brazil, China, India and Beyond. World Bank. (2010). Project Performance Assessment Report. China Energy Conservation World Bank. (2014) Pacific Islands - Sustainable Energy Finance Project: Restructuring Paper. Washington DC: World Bank Group Within the scope of the PFAN performance evaluation, there are three potential conflicts of interest (COIs) that have arisen and are being mitigated. The first COI is that two members of the evaluation team routinely work in the CE field and may have access to proprietary data. To avoid this potential COI, these members have signed nondisclosure agreements and do not have access to data with identifiers such as name or email. Relevant data are stripped of this information except (1) where respondents have self￾identified that they are willing to be interviewed and thus the results of the interview would be available; and (2) where PFAN has identified other interview participants. Outside of the team, there are two other COIs. One potential COI is that USAID is a funding partner of PFAN as well as a managing client of dTS, specifically the GCC M&E project that is carrying out this evaluation. In this role, USAID has the opportunity to provide both technical and managerial direction to the evaluation team and their work products. The second potential COI is that CTI, through directive from USAID, has been provided the opportunity to informant interview guide. In addition, CTI is responsible for providing the evaluation team with contact information for all PFAN stakeholders and participants who have been involved with the pipeline projects. The evaluation team is unable to independently verify the data provided by CTI. To mitigate any potential COI that may affect research or findings, the evaluation team has not altered any document, report, instrument, or guide in a way that substantively changes the data to be collected and/or the findings to be disseminated based on feedback from either USAID or CTI. Following are disclosures of potential conflicts of interest from the evaluation team members participating in data collection activities. Given the nature of the PFAN collaborative agreement, there are significant opportunities for improved transparency and documentation which would increase PFAN as a clean energy developer. This report has mentioned many of these. USAID should consider treating PFAN like a typical contract in funding data collection and monitoring and evaluation. At this point, PFAN collects only the data it thinks it needs to comply with policies and achieve a target level of financial closures. Suggestions for types and sources of additional data include: Comprehensive data on applicants that will not be enrolled in PFAN, including retaining contact information. Some applicants heretofore labeled not qualified have in fact simply been turned away because PFAN did not have enough funding to handle them. Several benefits can be obtained by collecting this information. - First, disaggregating these applicants could help PFAN and others understand how much larger the the PFAN pipeline alone. Donors might be more inclined to increase funding if the pool of potential fundable projects is larger than can be seen from the pipeline. - Second, it helps PFAN communicate to qualified, but not accepted, applicants that they were not disqualified but rather not accepted due to a lack of funding. PFAN currently informs them they are not qualified. While this does not have consequences for PFAN, it can be quite a difference to a developer and help to encourage them if their project is promising but simply excluded for reasons of PFAN capacity. This group could be tracked for success as a counterfactual group for evaluation purposes - Third, it might help PFAN target future outreach and communications to reduce the number of applicants that are likely to be not qualified. Data on financial closures closer to, if not at, the time of closing. PFAN currently relies on information from the coaches on the intent of the financial institution to close as the marker for closed projects. There is no way to know how many of these closures actually take place since neither the financial institution nor the developer reports back to PFAN, and PFAN does not collect data once their assistance stops, which occurs before the actual closure. Funding PFAN to provide coaching services through financial closure will not only increase the accuracy of financial closings, but it will also provide much needed assistance, as discussed in this evaluation. Data on status and select key indicators for formerly PFAN-supported projects currently in operations. Data collection from projects funded by PFAN that are operating (or not) will help USAID understand the impact of its funding on the final beneficiaries of the project. There are other areas where greater data collection will benefit USAID, PFAN, and developers but all of this requires additional or reallocated funding. Data from this survey should not be posted nor made public without significant scrubbing of data fields, as it might be possible to identify individual respondents by the information they provided in their survey and interview responses. Please see the attached PDF of the SurveyMonkey online survey instrument,