MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE-BiH) THE WHOLE-OF-PROJECT EVALUATION OF USAID/BiH PROJECT 2.2: IMPROVED ECONOMIC ASPECTS OF GOVERNANCE RELEVANT TO BUSINESS ACTIVITY JUNE 2018 This document is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of authors and do not necessarily reflect the views of USAID or the United States Government. MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE-BiH) THE WHOLE-OF-PROJECT EVALUATION OF USAID/BiH PROJECT 2.2: IMPROVED ECONOMIC ASPECTS OF GOVERNANCE RELEVANT TO BUSINESS ACTIVITY June 2018 Prepared under the USAID’s Bosnia and Herzegovina Monitoring and Evaluation Support Activity (MEASURE-BiH), Contract Number AID-168-C-14-00003 Submitted to: USAID/Bosnia Herzegovina, June 2018 Contractor: IMPAQ International, LLC USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 1 CONTENTS CONTENTS............................................................................................................................................................1 TABLE OF EXHIBITS ........................................................................................................................................2 ACKNOWLEDGMENTS...................................................................................................................................3 ACRONYMS...........................................................................................................................................................4 EXECUTIVE SUMMARY...................................................................................................................................5 EVALUATION PURPOSE AND QUESTIONS ............................................................................................................................................................ 5 EVALUATION METHODS AND LIMITATIONS....................................................................................................................................................... 5 FINDINGS AND CONCLUSIONS .................................................................................................................................................................................. 6 MAIN RECOMMENDATIONS........................................................................................................................................................................................... 7 I. EVALUATION PURPOSE, DESIGN AND LIMITATIONS ...................................................8 II. BACKGROUND INFORMATION ON PROJECT 2.2 ..............................................................9 III. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS........................................... 11 A. FINDINGS (RQ1).......................................................................................................................................................................................................11 B. CONCLUSIONS (RQ1)...........................................................................................................................................................................................19 C. RECOMMENDATIONS (RQ1).............................................................................................................................................................................19 D. FINDINGS (RQ2 – EIA)...........................................................................................................................................................................................21 E. CONCLUSIONS (RQ2 – EIA) ..............................................................................................................................................................................29 F. RECOMMENDATIONS (RQ2 – EIA)................................................................................................................................................................31 G. FINDINGS (RQ2 - FSRA)........................................................................................................................................................................................35 H. CONCLUSIONS (RQ2 – FSRA)...........................................................................................................................................................................42 I. RECOMMENDATIONS (RQ2 – FSRA) ............................................................................................................................................................44 J. FINDINGS (RQ3).......................................................................................................................................................................................................48 K. CONCLUSIONS (RQ3)...........................................................................................................................................................................................49 L. RECOMMENDATIONS (RQ3).............................................................................................................................................................................50 ANNEX I: EVALUATION PURPOSE, DESIGN AND LIMITATIONS........................................ 51 I. EVALUATION PURPOSE........................................................................................................................................................................................51 II. EVALUATION QUESTIONS.................................................................................................................................................................................51 III. DATA COLLECTION AND ANALYSES .........................................................................................................................................................53 IV. EVALUATION LIMITATIONS ..............................................................................................................................................................................58 V. BACKGROUND INFORMATION ABOUT CONSTITUENT ACTIVITIES......................................................................................59 ANNEX 1I: MAIN DATA COLLECTION INSTRUMENTS............................................................. 65 ANNEX II.I: DOCUMENTS/DATABASES REVIEWED..........................................................................................................................................67 ANNEX II.II: SEMI-STRUCTURED KEY INFORMANT INTERVIEW GUIDE FOR PROJECT-RELATED ISSUES........................70 ANNEX II.III: MINI ONLINE SURVEY QUESTIONNAIRE FOR ENTERPRISES .........................................................................................77 ANNEX III: KEY INFORMANT INTERVIEWEES ............................................................................... 80 ANNEX IV: THE EVALUATION TEAM RESPONSES TO COMMENTS FROM THE IMPLEMENTING PARTNER........................................................................................................................ 82 USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 2 TABLE OF EXHIBITS Exhibit 1. USAID/BiH 2012–2016 CDCS Results Framework, as of January 2012................................. 9 Exhibit 2. Key Elements of PAD 2.2......................................................................................................... 11 Exhibit 3. Changes in the Project 2.2 Logical Framework at the Sub-Purpose Level ......................... 13 Exhibit 4. Assumptions Indirectly Outlined in the PAD......................................................................... 13 Exhibit 5. Doing Business Ranking of BiH................................................................................................ 16 Exhibit 6. Score on the Regulatory Quality Dimension of the Worldwide Governance Indicators ... 16 Exhibit 7. Addressing Obstacles to Investments in the BiH Energy Sector ......................................... 16 Exhibit 8. Indicator of Sub-Purpose of Fiscal Policy Reforms ................................................................ 17 Exhibit 9. In your opinion, how important is BiH's improvement in economic governance? ............. 17 Exhibit 10. Sectors Important to Private Sector Companies ................................................................ 18 Exhibit 11. List of KIIs per Type of Key Informant ................................................................................. 54 Exhibit 12. Evaluation Matrix ................................................................................................................... 55 Exhibit 13. Map of EIA and FSRA Beneficiaries ...................................................................................... 57 Exhibit 14. Timing and Cost of the Project 2.2 Activities ...................................................................... 59 USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 3 ACKNOWLEDGMENTS Guided by the U.S. Agency for International Development (USAID) Evaluation Policy, USAID/Bosnia and Herzegovina (BiH) commissioned IMPAQ International within the USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH) to conduct the Whole-of-Project Evaluation of USAID/BiH Project 2.2: Improved Economic Aspects of Governance Relevant to Business Activity. Two activities contributing to this project— the Energy Investment Activity (EIA) and the Fiscal Sector Reform Activity (FSRA)—have been implemented by USAID/BiH within the current Country Development and Cooperation Strategy (CDCS) 2012–2016 period examined in this evaluation, beginning March 2014, when the 2.2 Project Appraisal Document was finalized. The evaluation team members include Davorin Pavelic (Team Leader and MEASURE-BiH Chief of Party), Edis Brkic (evaluation team member and Deputy Chief of Party); Lejla Lazovic-Pita (evaluation team member and MEASURE-BiH Research Fellow), Fuad Strik (evaluation team member), Anesa Hadzic (evaluation team member and MEASURE-BiH Research Analyst), and Anela Kadic Abaz (evaluation team member and MEASURE-BiH Research Analyst). The authors wish to thank those who contributed to the process of developing this evaluation report. At USAID/BiH, the following people were instrumental in designing the evaluation: Elma Bukvic Jusic, MEASURE-BiH Contracting Officer’s Representative (COR); Andrew Boegel, Economic Development Office (EDO) Director; Ankica Gavrilovic, Energy Investment Activity COR. Samir Dizdar, FSRA COR and Project 2.2 Manager helped us understand the design of Project 2.2 and its activities and management. We also thank the EIA and FSRA activity management teams for their assistance with activity documentation and databases. Any errors in this report are the sole responsibility of the authors. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 4 ACRONYMS ADS Automated Directives System BiH BMIS Bosnia and Herzegovina Budget Management Information System CDCS Country Development and Coordination Strategy COP COR Chief of Party Contracting Officer Representative DO Development Objective EDO Economic Development Office EIA EnC EU Energy Investment Activity Energy Community European Union FBiH Federation of Bosnia and Herzegovina FSRA Fiscal Sector Reform Activity GIZ German Agency for International Cooperation (Deutsche Gesellschaft für Internationale Zusammenarbeit) IMF International Monetary Fund IR Intermediate Results KI Key Informant KII Key Informant Interview M&E Monitoring and Evaluation MEASURE-BiH USAID/BiH Monitoring and Evaluation Support Activity MoFTER Ministry of Foreign Trade and Economic Relations NGO NEAP Non-Governmental Organization National Environment Action Plan PAD Project Appraisal Document PARCO Public Administration Reform Coordinator’s Office PECI Projects of Energy Community Interest RS Republika Srpska TPP Thermal Power Plants TS Treasury System USAID US Agency for International Development USAID/BiH US Agency for International Development in Bosnia and Herzegovina USG US Government USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 5 EXECUTIVE SUMMARY IMPAQ International has been commissioned by USAID/BiH, as part of the USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH), to conduct the Whole-of-Project Evaluation of USAID/BiH Project 2.2, Improved Economic Aspects of Governance Relevant to Business Activity. Two activities contributing to the purpose of this project— the Energy Investment Activity (EIA) and the Fiscal Sector Reform Activity (FSRA)—have been implemented by USAID/BiH within the current Country Development and Cooperation Strategy (CDCS) period examined under this evaluation, beginning March 2014, when the 2.2 Project Appraisal Document (PAD) was finalized. The Results Framework of the USAID/BiH 2012–2016 CDCS includes two Development Objectives (DOs)/Project Goals. The second DO states: “A competitive, market-oriented economy providing better economic opportunities for all its citizens governs the Mission’s economic development interventions.” Within this DO, USAID/BiH has a two-pronged approach through two CDCS Intermediate Results (IRs)/Project Purposes. The first IR aims to help enterprise-level competitiveness (notably in agriculture and small and medium-sized enterprises in key sub-sectors), while the second IR aims to support priority regulatory and policy reform to improve the business environment in BiH. The latter is addressed through Project 2.2 and is the subject of this evaluation report. EVALUATION PURPOSE AND QUESTIONS The purpose of the whole-of-project performance evaluation (WOPE) is to examine the entire USAID/BiH Project 2.2, including all of its constituent activities, and progress toward the achievement of the project purpose. In order to assess Project 2.2 as a whole, along with its constituent activities, this evaluation answers the following three research questions: (1) To what extent were the programmatic and contextual assumptions identified in the project theory of change sufficient to achieve the purpose of Project 2.2? (2) To what extent have the activities implemented under this project achieved their expected results, and to what extent have they been able to address and collectively contribute to the purpose of Project 2.2? (3) To what extent have the activities implemented under this project been coordinated and how have internal and external activity/project management practices supported progress toward the purpose of Project 2.2? EVALUATION METHODS AND LIMITATIONS The evaluation team used a mixed-method approach based on an extensive desk review of implementation documents and databases received from USAID/BiH as well as additional documents collected from the EIA and FSRA activities, semi-structured interviews with 63 key informants (KIs), and a mini online survey of a sample of enterprises in BiH. The main limitations of this evaluation relate to possible selection bias, response bias, and recall bias. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 6 FINDINGS AND CONCLUSIONS The number of implemented activities was not sufficient to achieve the Project 2.2 purpose. Due to its late start, changes in the Mission’s management, and lack of funding, Project 2.2 was only partially implemented and lacked a cohesive approach. The Project Appraisal Document for Intermediate Result (PAD) 2.2. was designed and approved two years after adoption of the CDCS. Only three of eight planned activities were implemented: two activities (EIA and FSRA) are at the late stage of implementation, while the third one (FINRA) has only begun. The small number of implemented activities made project-level management almost unnecessary. With regard to the project’s implemented activities, a majority of interviewed beneficiaries expressed their satisfaction with the technical assistance provided by EIA and FSRA and felt that the interventions implemented under these two activities are of highest priority and relevant for these two sectors. With regard to the FSRA’s expected results, however, not much has been achieved in terms of improved coordination at the level of the Fiscal Council of BiH and consistent reporting to the European Union (EU). Although analytical capacity of the entity’s ministry of finance in Republika Srpska (RS) is considered a success, the evaluation team could not confirm that the analytical capacity of two ministries of finance has improved. The Budget Management Information System (BMIS) is expected to achieve most results by the end of the activity. However, the implementation of the treasury system (TS) in local communities in the Federation of Bosnia and Herzegovina (FBiH) requires further actions through the end of the activity. Introduction of the TS to the RS Health sector was a premature intervention given the current state of the health sector in RS. However, certain progress has been made. Strengthening of the Public Debt Management system in BiH was cancelled in October 2017, but some lessons were learned for the future. Although the expected quantitative results related to tax administration in BiH were confirmed in RS, those in FBiH are expected by October 2018. Implementation of the single registry of business accounts in FBiH was assessed as needed, and the implementation was conducted very quickly. Further actions are planned by the end of the activity. Regarding achievement of the EIA’s expected results, an inadequate regulatory framework still renders the business environment in the electricity sector ill-suited to investments. Despite tremendous efforts of the EIA to identify all the impediments to investments, prepare comprehensive sets of recommendations for their removal and the requisite policy documents, and build institutional capacities, there has been no major breakthrough to improve the investment climate. Although the energy retail market in BiH is functional, the EIA still needs to achieve Result 2.1, Distribution System Operator (DSO) as an independent market facilitator, which is one of the preconditions for the development of other elements of an open electricity retail market. An analysis of coordination processes at the project level, and interaction among the constituent activities, showed that interaction between EIA and FSRA activity is weak. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 7 MAIN RECOMMENDATIONS This report provides several recommendations for USAID/BiH’s consideration based on lessons learned from the evaluation:  First, a project-level management function and a project-level coordination mechanism should be established, formalized, and institutionalized.  The participatory approach to planning development interventions should be further strengthened in order to gather a more realistic overview of the context in which activities are implemented. For policy-level interventions, USAID/BiH should bring the relevant stakeholders to the table early in the design process. USAID/BiH should ensure and confirm, through an active political process and a request for a guarantee and proof of local stakeholders’ commitment that programmatic and contextual assumptions are in place before beginning technical preparations for the implementation of activities. Where guarantees and strong commitments from local decision makers cannot be obtained, USAID/BiH should consider discontinuing further assistance.  USAID should not expect the implementing partner to resolve problems resulting from a political arena that has not been well prepared for the technical implementation of activities. Contracts should be awarded to implementing partners with strong experience and expertise in both the technical and political segments of the implementation of development initiatives.  Regarding energy sector reforms, USAID should be flexible and realistic in setting targets for its activities because energy sector reform is a long-term and complex process, especially given the constitutional complexity regarding the jurisdiction of BiH national and entity levels.  USAID should assess support for the establishment of an electricity stock exchange in BiH for further development of the electrical energy market. Some experts consider such an institution to be an urgent requirement. With three existing electric power companies and a number of registered energy suppliers, BiH has the necessary preconditions for the development of a stock exchange for electric energy and its integration into an evolving regional stock exchange network.  To strengthen the fiscal sector, further technical support related to macroeconomic analysis and macro-econometric modelling would be beneficial for building stronger analytical capacities in all ministries of finance, which should result in more reliable projections.  Also, more training and education related to implementation of BMIS (program budgeting, budget beneficiaries, etc.) would be beneficial by the end of the activity. In the future, BMIS should be connected with the treasury system and potentially introduced to local communities. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 8  Interventions related to a single registry of business accounts in BiH should be continued. USAID should persist in interventions related to registries of parafiscal fees, especially in terms of elimination and reduction of these fees in both entities and Brcko District. I. EVALUATION PURPOSE, DESIGN AND LIMITATIONS The goals of the whole-of-project performance evaluation of USAID/BiH Project 2.2, Improved Economic Aspects of Governance Relevant to Business Activity, are (i) to examine the progress of the entire project toward its stated purpose, and the progress of its associated constituent activities toward project sub-purposes and outcomes/outputs; (ii) to describe project and constituent activities coordination and implementation; and (iii) to provide recommendations for possible future USAID/BiH interventions. The evaluation questions are broadly based on the guidance from Automated Directives System (ADS) 201, Additional Help for Whole of Project Evaluation. The first question studied the strategic planning process of the Mission at the project level by examining strengths and weaknesses of the project￾designed theory of change. The second question examined the project’s performance and the results that were achieved. The third examined the strategic coordination process of the Mission at the project level by looking at the interactions among constituent activities as they contribute to the project purpose. The evaluation was based on a rigorous evaluation design and methodological approach to address the evaluation questions and to provide insights into the progress towards expected results at the project level, as well as at the level of the two constituent activities included in this project, by connecting the findings to conclusions, lessons learned, and recommendations. The research design employed mixed methods so that the gathered information could be triangulated in addressing each of the evaluation questions using different data sources: documents, key informant interviews, and a mini online survey of a sample of enterprises in BiH. The data from these sources were triangulated to address the same questions/sub-questions from multiple perspectives whenever possible. Comparing data helped the team to gain a more complete understanding of the issues and provided more confidence in the findings. The main limitations of the evaluation include response bias, recall bias, and selection bias. These limitations were mitigated by drawing on multiple sources of information, by carefully designing and conducting data collection, and by making sure to include the main stakeholders and beneficiaries of the activities. More information on the evaluation purpose, design, and limitations is presented in Annex II. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 9 II. BACKGROUND INFORMATION ON PROJECT 2.2 The Results Framework of the USAID/BiH 2012–2016 Country Development and Coordination Strategy (CDCS) is presented in Error! Reference source not found. 1. 1 The framework includes two CDCS Development Objectives (DOs)/Project Goals, one of which—DO 2: A competitive, market-oriented economy providing better economic opportunities for all its citizens—governs the Mission’s economic development interventions. Under this DO, USAID/BiH has two-pronged approach through two CDCS Intermediate Results (IRs)/Project Purposes to promote enterprise-level competitiveness (notably in agriculture and small and medium-sized enterprises in key sub-sectors) and to support priority regulatory and policy reform to improve the business environment in BiH. The latter is addressed through Project 2.2, Improved Economic Aspects of Governance Relevant to Business Activity. Exhibit 1. USAID/BiH 2012–2016 CDCS Results Framework, as of January 2012 At the time of developing the CDCS (January 2012), USAID/BiH identified financial sector reform and capacity building as priorities for assistance under this IR because of the importance of financial sector 1 Exhibit 1 shows the Results Framework included in CDCS. The Mission subsequently modified the Results Framework in 2014, including replacing IR 2.2.1 with 2.2.1 Fiscal policies reformed; 2.2.2 Employment, social and disabilities policies reformed; 2.2.3 Financial sector in line with international standards; 2.2.4 Energy sector's sustainable contribution to economy increased; 2.2.5 Green development introduced; and 2.2.6 Improved public service delivery efficiency and enhanced broad stakeholder involvement in public service by high-quality e-Governance. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 10 stability in attracting investment to BiH. Specifically, the CDCS aimed at supporting the growth of competitive financial systems while strengthening transparency and oversight to enhance the stability of financial markets, as well as possibly directing assistance also toward insurance and pension reform. Finally, the CDCS discussed planned support for energy sector reform in BiH, given the importance of the energy sector for exports, revenue generation for BiH, and attractiveness for foreign investment (i.e., power stability). To build on earlier USAID/BiH energy interventions, continued assistance was planned in developing transparent, functional and responsive energy regulatory bodies, to help lead the energy market in a more market-driven direction and promote the integration of the BiH energy sector into the larger Southeast and European Energy Market and energy trade bodies. As of September 2014, USAID/BiH began implementing two activities during this CDCS period: the Energy Investment Activity (EIA) and the Fiscal Sector Reform Activity (FSRA). The third activity under this Project—the USAID Financial Reform Agenda Activity (FINRA)—commenced in October 2017 and is thus not part of this evaluation.  The Energy Investment Activity (EIA) is a $5.47 million activity for the period from September 2014 to September 2019, that is being implemented by Advanced Engineering Associates International. The purpose of the EIA, as specified in the award, is to increase investments in the energy sector in BiH and advance the BiH accession process related to energy requirements by working on the following tasks: improving coordination, management, and transparency at all levels of BiH’s regulatory framework, making a simplified energy investment environment, and targeting technical assistance to integrate BiH’s energy sector into regional and EU markets.  The Fiscal Sector Reform Activity (FSRA) is a $5.0 million activity for the period December 2014 to September 2019, that is being implemented by FINIT Consulting, a local contractor. As specified in the award, the FSRA supports efforts to facilitate gains in fiscal sector and public investment, which in turn should result in an increase in private domestic and foreign investments in BiH. The focus of the activity is fiscal discipline, which entails a comprehensive set of initiatives tackling coordination between the different levels of the BiH governments, and additional work with governmental institutions such as the tax administrations. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 11 III. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Evaluation Question 1: To what extent were the programmatic and contextual assumptions identified in the project theory of change sufficient to achieve the Project 2.2 purpose? What are the lessons learned for USAID/BiH future planning at the project level? A. FINDINGS (RQ1) Finding 1: The number of implemented activities and the number of sectors covered are not sufficient to support achievement of the Project 2.2 purpose. The Project Appraisal Document (PAD) for Intermediate Result 2.2 was designed and approved in March 2014, two years after adoption of the CDCS. Both of the activities implemented under this PAD began in the last quarter of 2014 and will be completed 2019, one year after the expiration of an extension of the CDCS’s implementation. Of the eight activities planned for implementation, only two are in the late stages of implementation, while a third has just begun. Finding 2: There are several issues with the PAD’s narrative:  The PAD narrative describes a rather complex logical framework consisting of the goal, ultimate goal, purpose, and results (see Exhibit 2), and the LogFrame table does not reflect the PAD’s description.  There is no clear hierarchy between “ultimate project’s goal” and the actual project goal.  The definition of the project purpose “Improved economic aspects of governance relevant to business activity” is not clear and has resulted in many different interpretations.  The project purpose outlined as an intermediate result (IR) 2.2 is different in the PAD and the LogFrame.  It is stated that IR 2.2. will have a specific focus on sub-IR 2.2.1 (Regulations and policies that foster economic space and private sector development and investments), which is a synthesis of the project’s goal and the project’s “ultimate goal.”  Sub IR-2.2.1 is mentioned several times in the PAD, and each time it is defined differently. Exhibit 2. Key Elements of PAD 2.2 PAD LogFrame Assumptions Project’s ultimate goal Create a single economic space and business friendly environment for economic growth. N/A N/A Project’s goal Improve economic aspects of governance relevant to the business -enabling environment through improvement of the commercial legal framework, fiscal policies and delivery of goods and services provided by the government to citizens and businesses, N/A N/A IR 2,2 Project’s purpose Achieve more functional and accountable institutions and actors to meet citizens’ needs in BiH. Improved economic aspects of governance relevant to business activity N//A IR 2.2.1. Regulations and policies that foster a single economic space and private sector development and investments Fiscal policies reformed N/A USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 12 Finding 3: None of the five studies and analyses 2 planned to support and inform design of specific activities were completed. The absence of these analyses may have affected the design of activity-level theories of change. Finding 4: The key informant interviews indicate that the changes at the activity level were extensive, influenced by lack of funding and changes in management, and had a negative effect on the initially envisioned theory of change. All planned activities were interconnected and sequenced so that full completion of one activity would create the necessary preconditions for the implementation of another. Not implementing certain planned activities undermines the results framework and the envisioned theory of change. ADS 201 states that PAD only provides an organizing framework, which should be adapted when new evidence emerges or circumstances change. The Project’s LogFrame has changed significantly over the last five years. This is especially true at the Project’s sub-purposes level, which changed in terms of numbers and sectoral focus (see Exhibit 3). The following project sub-purposes were planned as part of Project 2.2’s purpose in PAD 2.2: 2.2.1. Fiscal sector strengthening activities (to result in improved coordination of the budget and treasury systems) 2.2.2. Adoption of employment, social, and disability policies related to economic growth (to result in introduction of a workman’s compensation system) 2.2.3. Financial sector providing stability and serving private sector needs (to result in financial sector reform – adoption of international standards) 2.2.4. Energy reform policies (to result in an increased sustainable contribution of the energy sector to the economy) 2.2.5. Anticorruption—Focus on improved control over public resources/funds (not included in the LogFrame associated with PAD) 2.2.6. Green development (to result in the introduction of green development) 2.2.7. E-governance reform (to result in improved utilization of e-governance). 2 (1) Detailed feasibility studies for activities envisioned during the second phase of the project, such as pension and disability insurance fund reform, analyzing the overall environment for the reform, possible risks, cost-effectiveness, and expected results; (2) detailed analysis of the disability insurance system; (3) detailed analysis focusing on labor market rigidities and possible reforms; (4) detailed analysis focusing on pension system reform; (5) analysis focusing on green development reform. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 13 Exhibit 3. Changes in the Project 2.2 Logical Framework at the Sub-Purpose Level USAID/BiH CDCS January 2012 USAID/BiH Project PAD 2.2 March 2014 USAID/BiH Project 2.2 LogFrame March 2014 USAID/BiH Project 2.2 LogFrame December 2017 Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.1 Regulations and policies foster private sector development and investment Fiscal sector strengthening activities Improved coordination of budget and treasury systems Fiscal policies reformed Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.2 / Adoption of employment, social, and disability policies related to economic growth Financial sector reform— adoption of international standards Employment, social, and disabilities policies reformed Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.3 / Financial sector providing stability and serving private sector needs Workman's compensation system introduced Financial sector in line with international standards Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.4 / Energy reform policies Energy sector's sustainable contribution to economy increased Energy sector's sustainable contribution to economy increased Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.5 / Anti-corruption—Focus on improved control over public resources/funds Utilization of e￾governance improved Improved public service delivery efficiency and enhanced broad stakeholder involvement in public service by high-quality e￾governance Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.6 / Green development Green development introduced Green development introduced Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.7 / E-governance reform / / Finding 5: Programmatic and contextual assumptions were not explicitly defined for PAD 2.2. However, certain programmatic assumptions can be identified through the PAD 2.2 narrative (see Exhibit 4). Four identified contextual assumptions are necessary for achievement of the Project 2.2 purpose, but they are not sufficient for achievement of that purpose. It should be noted that none of these assumptions were examined or tested. Exhibit 4. Assumptions Indirectly Outlined in the PAD # Critical Assumption Satisfied (Yes/No) Comment 1 The project will benefit from coordination and cooperation with USAID’s civil society organizations and independent media cross-cutting programs and the relevant departments of the US Embassy in Sarajevo (Economic and Political Sections) to facilitate rapid and smooth adoption of the regulations and policies targeted by Partially Activities implemented under PAD 2.2 established good cooperation with some of the cross-cutting programs of USAID’s civil society organizations. However, adoption of the regulations and policies targeted by the project has not been rapid or smooth. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 14 # Critical Assumption Satisfied (Yes/No) Comment the project. 2 Entity governments and state-level counterparts will readily accept USAID assistance in the identified areas (PAD 2.2) Partially Not all areas of USAID’s assistance were accepted by entity-level governments. This was especially the case with RS, which refused to work on FSRA’s fiscal council component and EIA’s energy efficiency obligation schemes. 3 It is expected that all three levels of government will be USAID’s champions in the implementation phase. No Very few, if any, of the partnering institutions could be considered champions of USAID’s supported reforms under PAD 2.2. 4 USAID will seek to partner with other principal donors in BiH to amplify its call for host country buy-in and to pursue opportunities for additional funding. Partially USAID established partnerships with other principal donors in BiH, but no additional funds were raised. Finding 6: The commitment of BiH authorities to champion necessary reforms, one of the critical assumptions outlined in the PAD, was not satisfied (please see the European Commission 2018 report on Bosnia and Herzegovina3). Lack of political will on the part of decision makers to pursue necessary reforms was also confirmed in interviews with the key informants. Almost one-third of all interviewees identified a lack of political will as the main obstacle to the successful implementation of EIA and FSRA activities. Finding 7: Understaffed institutions and their low capacity to efficiently implement USAID’s recommendations is one of the important assumptions that was not taken into consideration during the design activities. Several issues continually surfaced during the key informant interviews: inadequate human resources, both in terms of numbers and competencies, poorly equipped institutions, insufficient financial resources available for the operation of these institutions, and absence of communication and coordination of work (vertical and horizontal) between and within institutions. Finding 8: The document review and the USAID/BiH staff interviews indicate that during the PAD concept development phase, USAID consulted host government officials, who recommended the majority of activities under PAD 2.2; however, the evaluation team 3 European Commission. Key Findings of the 2018 Report on Bosnia and Herzegovina; httsp://ec.europa.eu/rapid/press￾release_MEMO-18-3408_en.pdf “(i) with the exception of a few reforms and the notable adoption of the excise legislation, delivery on a number of reforms was delayed by lack of agreement within the ruling coalition members. Countrywide strategies in areas such as energy, employment or public financial management are yet to be adopted. (ii) A national programme for the country's legal approximation with the EU acquis has yet to be adopted; (iii) Bosnia and Herzegovina is at an early stage with the reform of its public administration and no progress has been achieved in the past year.” —Excerpt from the European Commission 2018 Report on BiH USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 15 found that some of EIA’s and FSRA’s stakeholders and beneficiaries were consulted and informed only after the contracts for these two activities had been awarded. During the development of PAD 2.2, major donors active in sectors of interest to USAID (EU, UNDP, and GIZ) were consulted. Finding 9: Out of thirty (30) FSRA key informants, only six (6) stated that they were consulted during the activity’s intervention design process. All six KIs that were consulted are relevant and hence were included in the creation of specific interventions under the FSRA, with the possibility of disseminating related information through appropriate internal mechanisms within their respective institutions. These KIs all responded that they were included in previous USAID/BiH fiscal sector activities, that they were informed in a timely manner about the FSRA, and that they were directly included in the creation or signing the Memorandum of Understanding between interested parties that was signed after the award of the contract in May 2015. Finding 10: Eight (8) major stakeholders from the BiH energy sector, the state and entity￾level ministries in charge of the energy sector, regulators in both entities, and three electric power authorities were informed about the EIA in September 2014 after the award of the contract. Eighteen (18) of thirty-seven (37) interviewed stakeholders confirmed that they were informed about the interventions USAID intended to support related to energy sector reforms within EIA. They also stated that they received a presentation by a project implementer before activities started or were contacted in advance and included in the period of activity implementation in 2015. Finding 11: Although one USAID staff member is in charge of the management of Project 2.2, that position was never formalized within the Mission. According to ADS 201, the Project Manager is responsible for project implementation oversight, monitoring and evaluation, and learning and adoption. However, none of these project management functions were ever applied to the management of Project 2.2. The entire management function was reduced to a coordination role. Initially, coordination started with regular monthly meetings, but even that practice ceased six months after the start of EIA and FSRA. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 16 Finding 12: The project’s goal-level indicators, the World Bank’s Doing Business ranking and Distance to Frontier, which are measurements of the quality of the business regulatory environment and set to track the progress of DO2 achievement, are equally applicable to the measurement of progress in achieving the Project 2.2 purpose. Changes in these two indicators (see Exhibit 5) show that little or no progress was made during the last four years. The World Bank indicator of regulatory framework quality and progress of reforms shows that BiH’s ranking has been continuously worsening since 2015. The distance to frontier, 4 as a measure of how BiH is performing in advancing its business regulatory environment, shows that there was no significant progress over the last four years. Finding 13: The project-level indicator related to the regulatory quality of the Worldwide Governance indicator shows that no progress was made over the last four years. Changes in this indicator show that the business environment and investment climate was worse in 2017 than it was in 2014 (see Exhibit 6). Finding 14: The poor investment and business environment in BiH, specifically in the energy sector, was confirmed through the KIIs. Investing in the energy sector in BiH is a complex, time consuming, and expensive process with an unpredictable outcome. This is equally valid for foreign investors and for private and public sector investors from BiH. Also, the majority of private sector 4 The distance to frontier measure shows the distance of each economy to the “frontier,” which represents the best performance observed on each of the indicators across all economies in the Doing Business sample since 2005. An economy’s distance to frontier is created on a scale from 0 to 100, where 0 represents the lowest performance and 100 represents the frontier. The ease of doing business ranking ranges from 1 to 190. The ranking of 190 economies is determined by sorting the aggregate distance to frontier scores, rounded to two decimal places. 51.7 51.0 47.6 48.6 2014 2015 2016 2017 47% 45% 7% 1% No improvement Small improvement Better Significant improvement Exhibit 5. Doing Business Ranking of BiH Exhibit 6. Score on the Regulatory Quality Dimension of the Worldwide Governance Indicators Exhibit 7. Addressing Obstacles to Investments in the BiH Energy Sector USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 17 companies that responded to the mini-survey thought that during the last three years, no progress (47%) or only small improvements (45%) were made in addressing obstacles to investments in the energy sector (see Exhibit 7). Finding 15: There has been no significant progress at the sub￾purpose level in fiscal policy reforms. Regarding the tax burden and size of the government indicator, there was almost no change over the last four years. There were some small improvements in the paying taxes indicator (see Exhibit 8) Finding 16: A majority of private sector companies (83.87%) believe that improvements in economic governance are important for BiH’s further development (see Exhibit 9). PAD 2.2 defines improvement of economic aspects of governance relevant to business activity as the purpose of the project, which further reduces the project’s focus on an improved regulatory and policy framework that will foster private sector development and investments. The significance of such a purpose is supported both by the key informants and the findings of the mini-survey conducted with private companies. 82.9 82.9 83.9 83.5 58.2 55.2 60.8 60.4 2014 2015 2016 2017 Score in the Tax Burden dimension of the Government Size pillar within the Economic Freedom Index Score in Paying Taxes dimension of Doing Business (measuring time and costs saved for businesses), distance to frontier 83.87% 13.15% 2.48% 0.25% 0.25% Very important Mostly important Partially important Not important nor not important Not important Exhibit 8. Indicator of Sub-Purpose of Fiscal Policy Reforms Exhibit 9. In your opinion, how important is BiH's improvement in economic governance? USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 18 Finding 17: USAID/BiH’s selection of priority sub-purposes that were actually funded (energy and fiscal sectors) is not in line with the priority sectors (sub-purposes) identified by private sector companies. Private sector companies gave the highest priority to sub-purposes related to anti-corruption, stability of the financial sector and e-governance (see Exhibit 10). The priority ranking given by private sector companies is logical and covers three important parts of enabling the business environment. Anti-corruption will give the same opportunities to all businesses in the market, a stable financial sector will provide businesses with better access to finance, and e-governance will save time and cost to businesses when they are dealing with public administration and bureaucracy. Finding 18: A majority of key informants consider that the two activities, FSRA and EIA, directly contribute to the fulfilment of the Project 2.2 purpose. The design, objectives and expected results of the two USAID/BiH’s activities evaluated under WOPE are sound and could significantly contribute to the fulfilment of Project 2.2. However, full achievement of the project’s purpose requires implementation of additional interventions in other areas (anticorruption activity, less costly and more efficient and business-oriented public administration, etc.). Finding 19: The initial LogFrames of both activities (EIA and FSRA) were changed or adjusted during the implementation period. The implementing partners indicated that the first LogFrames did not adequately capture the essence of the activity’s interventions. “The new LogFrame resolved that issue. Now we have a system where lower activities move upwards and reflect exactly what we do. The first LogFrame was not adequate in terms of describing what actually we do. “ -EIA Exhibit 10. Sectors Important to Private Sector Companies USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 19 B. CONCLUSIONS (RQ1) Conclusion 1: The 2012–2016 CDCS has been extended through November 2018 and still provides a sound strategic framework for the programming and design of specific activities. Conclusion 2: The main elements of PAD 2.2, such as the theory of change, programmatic and contextual assumptions, the project’s purpose, and the hierarchy of results are either only partially outlined and dispersed throughout PAD 2.2 or they are missing or not adequately defined. Conclusion 3: Significant delays in the process of the PAD 2.2 design and the ratio of planned to implemented activities, make PAD 2.2 barely relevant for achievement of the CDCS’s goal and objectives. Conclusion 4: Due to the small number implemented activities, project-level management was nonexistent and almost unnecessary. Conclusion 5: Not much has been achieved regarding accomplishment of the DO2 and the Project 2.2 purpose over the last four years. None of the project-level indicators show significant improvement. Conclusion 6: At the sub-purpose level, programming was rather flexible, iterative, and responsive to contextual changes. However, the selection of sub-purposes was not based on the highest priority area, but rather was chosen because of long-term USAID/BiH involvement in the specific sectors. Conclusion 7: The design, objectives, and expected results of the two USAID/BiH activities evaluated under WOPE are sound and could significantly contribute to the fulfilment of the Project 2.2 purpose. However, EIA and FSRA by themselves are considered insufficient to achieve that purpose. Full achievement would require the implementation of additional interventions in other areas. Conclusion 8: The combination of the long-term presence and assistance in fiscal and energy sectors, the absence of commitment and political will of local stakeholders to implement necessary reforms, and the statements by local stakeholders that USAID/BiH should continue with assistance in these two sectors reinforces the aid dependency trap. C. RECOMMENDATIONS (RQ1) Recommendation 1: USAID/BiH should invest in the training of its staff in project design in order to increase the quality of documents produced, such as the Project Appraisal Document. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 20 Recommendation 2: USAID/BiH should further strengthen a participatory approach to planning development interventions in order to gather a more realistic overview of the context in which activities are implemented. In addition, more comprehensive analyses of programmatic and contextual assumptions and risks should be conducted. For policy-level interventions, USAID/BiH should bring the relevant stakeholders to the table early in the design process. Moreover, USAID/BiH should ensure and confirm, through an active political process and a request for guarantees and proof of local stakeholders’ commitment, that programmatic and contextual assumptions are in place before beginning technical preparations for the implementation of activities. Declared support by local decision makers and the signing of memoranda of understanding should not be considered sufficient guarantees of local stakeholders’ commitment. Where guarantees and strong commitments from local decision makers cannot be obtained, USAID/BiH should consider ceasing further assistance. Recommendation 3: USAID/BiH should carefully weigh trade-offs between political objectives and the costs of maintaining a presence in specific sectors through business￾related activities that are expensive and difficult to implement. Long-term presence and involvement in certain sectors should not be a driving factor when selecting areas/sectors for interventions. Development needs and the solid commitment of all relevant stakeholders should be the two main criteria in the planning process. Recommendation 4: USAID should not expect the implementing partner to resolve problems of a poorly prepared political arena for the technical implementation of activities. Otherwise, contracts should be awarded to implementing partners with strong experience and expertise in both the technical and political aspects of implementing development initiatives. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 21 Evaluation Question 2: To what extent have the activities implemented under this project achieved their expected results and to what extent have they been able to address and collectively contribute to the Project 2.2 purpose? How have constituent activities been implemented and what were the main challenges and lessons learned from stakeholders’ perspectives for any potential future interventions? D. FINDINGS (RQ2 – EIA) Energy Investment Activity Finding 21: The key informants (KIs) consider the interventions implemented by EIA highly relevant to energy sector reforms in BiH because investors are facing a range of important challenges. The twenty-eight (28) interviewed stakeholders all noted that there are a range of challenges in the implementation of the interventions aimed at energy sector reforms in BiH. One of the most often mentioned challenges is contradictions between laws on different institutional levels or different interpretations of the same law and unclear jurisdiction. The absence of a development strategy for the energy sector on the state level (due to the political resistance against state-level strategies) is recognized as a long-term challenge for energy sector regulation on the state level. Lack of actual and detailed spatial planning documents on all levels relevant to electricity power sources and renewable energy is also considered a challenge. All KIs viewed political unwillingness to adopt and implement solutions and improvements as a main challenge. For this reason they have concerns about the final implementation and practical adoption of the documents, rules, or recommendations developed under EIA. KIs who were members of EIA working groups emphasized that they are powerless with regard to the full implementation of working group interventions. They also believe that EIA implementers cannot have significant influence on the adoption of the regulations or implementation of the agreements because higher-level political interests are the dominant influence. Finding 22: The EIA faced difficulties in getting support and cooperation from certain RS institutions and ministries for implementation of the interventions. Support was lacking especially for certain components of the EIA, such as the introduction of obligation schemes. Finding 23: Even the largest state-owned companies involved in electricity production could not successfully implement large investments in electricity production facilities. Jurisdiction for carrying out processes and steps within the authorization framework for the development of energy projects is divided among different government levels in Bosnia and Herzegovina (the state, the entity, and Brcko District). Consequently, the permitting procedure in FBiH is conducted at the FBiH and/or cantonal and municipal level. In contrast, the permitting procedure in the RS is more centralized at the entity level, but may include local authorities, depending on the capacity of the planned power plant. As can be seen, the permitting, licensing, and other authorization procedures in Bosnia and Herzegovina are very complex and are not harmonized. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 22 Inefficiency, lack of transparency, and complex permitting procedures are slowing down the development of energy sector, as can be seen in the example of EP BiH and EP HZHB. Even government-owned companies like these cannot go through this process without obstacles and forced delays and stoppages. For example, EP BiH has six major projects that have been in the design phase for more than eight years. In its struggle to finish the Mesihovina wind power plant project, EP HZHB has been required to obtain 336 licenses and permits, from application to final operational approval. In answer to this problem, significant work has been done to improve and simplify permitting procedures. The work was initiated by EIA in cooperation with GIZ, which is performing some of the interventions through its project “Promotion of Renewable Energy Sources in BiH.” The activity is supported by all government agencies and authority representatives directly involved in permitting procedures, with the aim of making the process easier for investors and attracting more investment into the BiH energy sector. Finding 24: Eleven (11) of eighteen (18) institutional stakeholders believe that impediments to investments in the energy sector will be addressed adequately and assessed precisely in the “Guide for Investors in the Electrical Energy Sector in BiH.” To improve the energy infrastructure permitting process, an extensive research and legal analysis has been performed under EIA that included the revision of 48 laws and regulations governing the permitting procedures for the construction of energy facilities (generations/transmission). This comprehensive list includes legislation in the field of concessions, spatial planning and land use, construction, environment, energy, use of water, forestry, agricultural land, land registry, and expropriation. These extensive research activities and legal analyses resulted in the completion of the “Draft Report on Permitting Regimes and Obstacles to Investment in BiH.” This report was the first such document on the permitting procedure for the construction of generation facilities in BiH, and thus its content is unique. In addition, EIA conducted extensive research and an analysis on international practices and policies regarding energy infrastructure permitting and developed the “Report on International Best Practice in Energy Infrastructure Permitting.” The report is a comprehensive study of the EU-required authorization framework and highlights the key challenges in permitting procedures. Although the focus is on best practices in the EU countries. The report also illustrates best practices in countries outside of the EU, including Turkey and the United States. Finally, it offers a set of recommendations to streamline permitting procedures and make them more efficient and transparent. With these submissions EIA implemented Result 1.1.2, Simplified guidelines for investors developed. During the interview process, eighteen (18) of the twenty-eight (28) KIs confirmed the active participation of their representatives in the working groups preparing the “Guide for Investors in the Electrical Energy Sector in BiH,” which was presented at the Fourth Energy Summit , in 2018, in Neum. Twelve (12) KIs out of twenty-eight (28) considered improvement in the investment environment in the BiH energy sector through the elimination of impediments as the most important contribution to the economy because of its multiplicative effects and the significant impact of energy production to GDP. Finding 25: Nine (9) out of eighteen (18) KIs strongly believe that the “Guide for Investors in the Electrical Energy Section in FBiH” will contribute to a better understanding of the USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 23 complexity of the problems for investors and will motivate decision makers to improve the permitting process. According to these key stakeholders, neither the investors nor the institutional representatives involved in the process have a clear picture of the permitting process, and for this reason the Guide for Investors has been well received. Work on the guide included developing a set of documents that included an analysis of the permitting regime, guidelines for biomass technologies, guidelines for investors, and real case studies of energy investment projects. This work was aimed at creating a better environment for investment in the energy sector and removing obstacles that are pushing investors away from the BiH energy sector. At this time, EIA has completed the reports “Permitting Regimes and Obstacles and International Best Practice in Energy Infrastructure Permitting,” completed the “Guidelines for Biomass Power Plant Technologies,” and drafted the FBiH Guidelines for Investors and submitted it to the Federal Ministry of Energy, Mining and Industry (FMERI). Eight (8) of eighteen (18) institutional stakeholders expressed concern about institutional intentions and efficiency in the implementation of the recommendations in the Guide for Investors. Five (5) did not believe institutions were ready to accept the recommendations and remove barriers in the BiH energy sector. Although this might be interpreted as EIA failing to gain institutional buy-in for the removal of impediments to investments in the energy sector, these same KIs considered it not as indicating EIA underperformance, but rather as a reality and an issue of BiH economic governance. In the future, EIA will focus on the development and completion of case studies that present a clear illustration of broad range of identified impediments and will additionally contribute to the better understanding of current problems for investors in the energy sector at all levels. Finding 26: The six (6) KIs from the private sector were not optimistic and saw no improvements for investors. They stated that during the last five years, except for Stanari TPP, there was no direct green field investment in the electricity production sector. Complex procedures and the lack of efficiency, transparency, and communication between institutions has resulted in a new trend in which investors aim to avoid as many obstacles as possible. Investors have learned from their common experience and discovered that the least painful way through the process is to acquire completed energy plants or take over projects with completed approvals for construction. This has become a business for some, because a notable number of small energy projects have gathered the needed permits for building but have been put on hold as the investor looks to sell the project and make a profit without going through the construction and operation phases of the plant. This trend has had a negative influence on energy sector development; there is a lack of major energy projects, and even the small ones take a long time to be completed. For this reason, six of the private sector Kls were not optimistic about the future of energy sector development in BiH. They stated that in the last five years only the Stanari TPP was a project that had direct green field investment in the electricity production sector. Stakeholders see the lack of results in the overall objective of increasing direct investments in the energy sector as a logical result of the overall national political situation, which has a negative influence on the USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 24 generation of new investments in all business sectors. They also view as negative the overall complexity of the procedure for the construction of electricity production facilities. Finding 27: According to the M&E plan, the EIA results in the Component 1, „Address Impediments to Investment in the Energy Sector in BiH”, are measured by the value of mobilized investments. In the technical assistance on preparation and implementation of the National Emission Reduction Plan (NERP), EIA reported USD 377 million in mobilized investments for FY 2016. BiH, as a Contracting Party of the Energy Community (EnC) Treaty, has the obligation to reduce emissions from TPPs, according to Directive 2001/80/EC on the emission of pollutants into the air from large combustion plants. EIA assisted BiH institutions and TPP operators to develop the National Emission Reduction Plan (NERP), which was submitted to the EnC Secretariat. The plan consists of 12 emission reduction plans for different large combustion plants (LCPs), that were expected to be completed by December 31, 2015. This was the first time that BiH fulfilled an obligation to the energy community on time. On October 13, 2016, the EnC published Opinion No. 2/16 on compliance with the NERP. Only the BiH NERP was found to be compliant with both the Policy Guidelines and Directive 2001/80/EC.5 EIA has been working intensely to ensure that governments and power utilities (the investors) commit to implement the measures and carry out investments until 2027. The FBiH Government and the RS Government have already made commitments to invest in the existing LCP power plants, USD 216 million and USD 161 million, respectively. In regard to the investments through NERP obligations in RS, 127 million USD is currently being realized. Elektroprivreda RS has fully completed one of the measures from the NERP—reconstruction of particle/dust filters at the Ugljevik TPP—at a cost of about 12 million USD, which the utility financed itself. In addition, the company secured around USD 115 million (12,633 billion yen) from the Japan International Cooperation Agency (JICA) for implementation of NERP interventions and started the construction of the Flue Gas Desulfurization (FGD) plant in November 2017. So far, the utility has “spent” about USD 24 million of the 115 million USD on on-site construction and equipment orders. EP BiH contracted with companies that have developed the preliminary design to implement the NERP measures and is now in the process of developing an investment plan for the implementation of NERP measures (desulphurization). According to unofficial information received from EP BiH, the management of the company enacted the Decision on the Investment Plan, and it is now waiting for the Supervisory Board to verify it. Once the Decision becomes official, EIA will be able to use the plan as an official source. Finding 28: Eleven (11) of eighteen (18) institutional stakeholders confirmed that opening the electrical energy market on January 1, 2016 and implementing the obligations from the Third Energy Package represented an important step forward. Although the overall result for C2 (“retail market functional”) has been achieved, EIA continues to work on implementation of this component. Two important results have to be achieved: 5 USAID Energy Investment Activity, Annual Summary Report 2016–2017. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 25 • Result 2.1, Distribution System Operator (DSO) as an independent market facilitator is functional. The DSO has not been implemented either in RS or in FBiH; • Result 2.2, Rules for the balancing settlement mechanism developed and implemented. This is one of the preconditions for the integration of renewable energy. The new system for balancing the electricity market and procuring ancillary services began functioning as of January 1, 2016. The task of the new system was to decrease imbalances in the internal electricity market and with neighboring countries, and to conduct procurement of ancillary services based on market principles. EIA worked closely with the ISO and SERC to prepare the new system for kick-off in January. In the first three months of implementation, the system proved to be functional and effective. Although stakeholders were generally satisfied with the new system, they pointed out problems that became evident in the course of implementation. EIA, in cooperation with stakeholders, identified that the DSO as an independent market facilitator has yet to be functional and that rules for balancing the settlement mechanism need to be developed and implemented to achieve the long-term functionality and effectiveness of the system. The stakeholders and EIA agreed to continue working on removing deficiencies and improving the system in order to ensure better system stability while maintaining the liquidity of the balancing market. Regarding achievement of the independence of the DSO, EIA developed a set of documents including “DSO Guidelines,“ “Feedback Report on the DSO Guidelines,” “Guidelines for Electronic Data Interchange,” and “Guidelines for Revision of the Market Rules,” which then led to the development and submission of a draft version of “Guidelines for the Development of the Business Processes for the DSO as an Independent Market Facilitator” to the three regulatory commissions, three power utilities, operators for renewable resources, the Transmission Company, and the Independent System Operator. The draft document lists all the roles and relevant business processes that should be included in the different pieces of legislation in order for the DSOs to function as the neutral market facilitators in BiH. The overall purpose of this effort is to enable smooth functioning of the BiH retail electricity market. In solving the other identified problem—the lack of rules for the balancing settlement mechanism—EIA has made progress by providing assistance to the Operator for Renewable Energy Sources in FBiH to move forward with the development and issuance of the Rule for Allocation of the Costs for the Balancing of Electricity from Renewable Energy Sources. The EIA interventions of this intervention included:6 (i) Several meetings and communication with the RES Operator staff on this matter; (ii) A proposal of solutions for the pending obstacles; (iii) Development of a letter/memorandum addressing the most problematic obstacle (the nonexistence of legal provisions that clearly define who is responsible for developing the schedules for the generators using RES) and its submission to the RES Operator; (iv) Development of draft Procedures for the Development and Submission of Daily Schedules; and 6 USAID Energy Investment Activity, Annual Summary Report 2016–2017. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 26 (v) Development of a draft proposal for the upgrade of the RES Operator to a balance responsible party. This intervention aims to assist the RES Operator in moving forward with the definition and organization of RES balancing to enable the integration of RES into the electricity system in BiH. Finding 29: Eleven (11) of eighteen (18) institutional stakeholders consider the main positive effect of the opening of the electricity market is to be changes in business practices in all three-state owned electrical power authorities toward market conditions in relation to their clients. Eleven out of 18 KIs (institutional stakeholders) confirmed that opening of the electrical energy market in 2016 and implementation of the obligations from the Third Energy Package are important steps in the EU accession process of BiH. Three of those KIs considered that the achievement of very clear instructions and established rules for functioning of the electricity market will contribute to faster development of the market itself in the next period. One KI considered that the first public procurement process and tender procedure for electricity supply for a publicly owned institution in 2018 (Sarajevo Airport) is the best confirmation that the opening of electrical energy market has been achieved. Not long after the opening of the electricity market, customers connected to the distribution network in Bosnia and Herzegovina started to change their suppliers. Independent (privately owned) suppliers have offered better prices to business customers when compared to the prices offered by the existing, state-owned power utilities. In response, the state-owned power utilities offered lower prices to almost one hundred other customers as incentives to stay with the utility. The 11 KIs mentioned above considered this the main positive effect of the electricity market opening, as EP BiH, EP HZHB, and EPRS started to change their monopoly supply habits, applying a flexible approach in the price negotiation process with their customers. Improvement of the quality of the electrical supply as part of supplier competitiveness is an additional positive effect. According to the 2017 State Electricity Regulatory Commission (DERK) annual report, at the end of the year 54 customers were no longer supplied by their traditional supplier, and thousands of customers, through negotiated contracts, benefited from improved conditions in the new business environment on the open energy retail market. The retail market is now functioning, which was considered impossible by the state-owned power utilities and governments at the start of the process. This market intervention is a major step forward in the functioning of the BiH retail electricity market, especially since more businesses are expected to change their supplier when they find lower prices and better conditions. This customer switching process is happening after years of USAID technical assistance to BiH in the field of electricity sector reform and electricity market opening. Finding 30: EIA did not report on any energy savings enabled as a result of US Government assistance since the targets for this indicator are set for years 2018 and 2019. According to the EIA’s annual reports, no targets were set for FY 2015, FY 2016, and FY 2017. Achieving energy savings and implementing energy efficiency measures was rated by nine stakeholders as the USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 27 component that has the highest potential for a sustainable contribution to the BiH economy because of the wide range of multiplicative effects and the least amount of resources required for implementation. EIA addressed energy savings through the activity’s component Energy Efficiency Obligation Scheme development. One of the first tasks, and a precondition for work on the development of EEO Schemes, was lobbying for the adoption of the Energy Efficiency (EE) Law in the Federation of BiH, which has been proven successful. The EE Law was blocked by the Federation Parliament’s House of Peoples after its introduction into the Federation Parliament in 2013. EIA coordinated the efforts of six donors and diplomatic missions in BiH to jointly give strong support for the adoption of the FBiH Energy Efficiency Law in a letter that was submitted to the FBiH House of Peoples, the FBiH Prime Minister, and the relevant ministers. In addition, EIA launched a media campaign and held a number of bilateral meetings with decision makers to lobby for adoption of the EE Law. The House of Peoples of the FBiH Parliament adopted the EE Law on February 2, 2017, and the law was later harmonized with the FBiH Parliament’s House of Representatives and published in the FBiH Official Gazette on March 24.7 Since then, EIA has been working on the development of detailed and precise documents for different institutions (FBiH Chamber of Commerce, Energy Regulatory Commissions, EEO Working Group members) and has proposed various interventions to address the energy savings needed for implementation of the EEO Scheme mechanism in BiH. Most notably, EIA has been involved in the development of the RS and FBiH Energy Efficiency Action Plans (EEAPs), and the development of the BiH National Energy Efficiency Action Plan and the FBiH Development Strategy for the Building Materials Industry for the period 2016–2025. At the same time, EIA has been working on the developing the Energy Efficiency Obligation documents: the EEO Savings Target Methodology, the EEO Collection Fee Calculation Methodology, the Regulation/Decree on Implementation of the EEO, and Energy Efficiency Investment Benefits and How to Achieve Them. Progress has been made in adopting each document needed for the EEO Scheme Model implementation, but it is insufficient to allow the energy savings resulting from USAID assistance to be measured since the political situation and the complexity of the governmental structure slow down decision-making processes. The energy efficiency obligation schemes are therefore still in a stage of development and discussion between EIA experts and working group members. It is hard to determine when the EEO Schemes will result in energy savings, but the potential, which is obvious, and the attitude of Kls towards this intervention indicate the possibility of more efficient decision-making and implementation of EEO Schemes in the near future. Finding 31: The National Energy Efficiency Action Plan (NEEAP), adopted by the BiH Council of Ministers and both entity EEAPs, has been adopted by entity governments. The plan includes the energy efficiency obligation schemes developed by EIA's EEO Scheme Working Group. Through the technical assistance interventions, EIA prepared proposals, proposed interventions, and provided expertise that contributed to the acceptance of the Energy Efficiency Obligation Scheme (EEO) as part of the adopted National Energy Efficiency Action Plan (NEEAP), and 7 USAID Energy Investment Activity - Annual Summary Reports 2016–2017, 2015–2016, 2014–2015. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 28 both entity EEAPs, and the documents include references to the materials developed by EIA's EEO Scheme Working Group. EEO has been incorporated into NEEAP through four measures:8  A program for energy efficiency obligation schemes in FBiH;  A program for energy efficiency obligation schemes in FBiH through heating energy distributors;  A program for energy efficiency obligation schemes in RS; and  A program for energy efficiency obligation schemes in RS through heating energy distributors. The NEEAP sets the financing target required for the implementation of the EEO Program through the electricity and heating distributors in the FBiH at 181,082,708 KM, and in the RS at 86,498,133 KM, which amounts at the state level to a total of 267,580,841 KM (USD 163,159,049). It is estimated that investment into these measures will achieve energy savings of 900,000 gigajoules for FBiH and 400,000 gigajoules for RS, which amounts to a total of 1,300,000 gigajoules for all of Bosnia and Herzegovina. Finding 32: The Action Plan to implement the Third Liberalization Package has been adopted, and EIA cross-cutting result has been achieved. Although BiH is a signatory of the Energy Community (EnC) Treaty, and according to it BiH is obliged to comply with the laws and requirements incorporated into this treaty, some institutions are not ready to accept this. The complex political situation makes it difficult to pass transposition legislation and, in particular, to harmonize it across the state, entities, cantons, and other government layers and institutions. For this reason, there are four ongoing infringement cases against BiH brought by the EnC related to failure to adopt required legislation. As a result, EIA provided technical assistance to the Ministry of Foreign Trade and Economic Relations (MoFTER) and hosted country partners to advance implementation of the Third Liberalization Package in relation to EIA Components 1–3. Recently, EIA has provided assistance to MoFTER in implementing Regulation 347, as adapted, by the EnC and participated in a meeting on the implementation of EnC Regulation 347 and the procedure for selection of a Projects of Energy Community Interest (PECI), which was organized by the EnC Secretariat for the PECI MoFTER Working Group in Sarajevo. Prior to this meeting, EIA and MoFTER met with the EnC Secretariat representatives separately to discuss the key elements of a draft decision on the implementation of EnC Regulation 347 that needed to be adopted by the BiH Council of Ministers. In addition, in the past year EIA developed the Road Map for Implementation of the West Balkan 6 Initiative (WB6) in Bosnia and Herzegovina. The Road Map lists all the activities and tasks ahead of BiH as defined by the WB6 and the institutions that are in charge of that particular intervention, along with the necessary steps to be undertaken by stakeholders. The Road Map was submitted to MoFTER, which is in charge of coordinating the WB6 activities and is responsible for reporting to the EnC. On March 7, 8 Energy Efficiency Action Plan in Bosnia and Herzegovina for the period 2016–2018. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 29 2017, the Council of Ministers acknowledged the Information on the WB 6 Initiative and charged MoFTER to define, together with other partners, the timeframe for the implementation of the Action Plan and to actively work on its implementation. The Action Plan contains about 40 activities that should be jointly undertaken by the competent institutions in BiH and the entities to enhance the capacities of the institutional structures necessary for the functioning of the relevant electricity market in accordance with the EnC Treaty and relevant acquis of the European Union.9 EIA has continued to assist MoFTER in managing and implementing the Road Map. In this regard, EIA has helped MoFTER to communicate with stakeholders and the EnCS in order to identify the status of implementation of certain activities in the Road Map and to initiate the assignment of responsible parties to work on them. After these parties commented on the Road Map, EIA assisted MoFTER in developing a revised version and finalizing the action plan for its implementation.10 Five (5) out of ten (10) KIs considered the EIA’s contribution to the fulfillment of the obligations from the Energy Community Treaty as a positive accomplishment. At the same time, however, they mentioned that many obligations are still pending and their implementation has faced significant delays. Four KIs underlined the fact that only EIA is able to overcome all politically divided stakeholders and to collect all of them together to open discussions related to Energy Community Treaty obligations. Two KIs said that NEAP, although it was developed with the support of other donors, “never would have been adopted” without the proactive and integrative role of USAID EIA. E. CONCLUSIONS (RQ2 – EIA) Conclusion 9: The scope of EIA is broad and includes investments, retail markets, energy efficiency, and renewable energy, but stakeholders perceived that the main focus of the activity is ensuring investment in the energy sector. The majority of them did not see the scope of activity outside their own area of work. The scope of EIA interventions, which includes three main components and two cross-cutting components, is very broad and includes investments, retail markets, energy efficiency, and renewable energy under the joint name Energy Investment Activity. The fact that stakeholders often perceive that the focus of the interventions is only investment in the energy sector is related to a lack of horizontal communication among stakeholders. The interviews with stakeholder representatives reviled differences in their perceptions of the activity. Eighteen (18) KIs representing institutional stakeholders, who are professionally involved in the energy sector interventions, did recognize the specifics of each component of EIA. The majority of the other KIs perceived EIA as an activity mainly focused on investments in the energy sector. 9 USAID Energy Investment Activity, Annual Summary Report 2016 -2017. 10 USAID Energy Investment Activity, Twelfth Quarterly Progress Report. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 30 Within the latter group, most of the KIs acknowledged a lack of horizontal communication between members of different EIA working groups from the same institution. Because of the different scope of interventions and different departments, they do not have even basic information about the interventions and performance in the components of EIA that are outside of their expertise. Conclusion 10: An inadequate regulatory framework renders the business environment in the electricity sector ill-suited to investments. The permitting, licensing, and other authorization procedures in Bosnia and Herzegovina are very complex, and are not harmonized. Unfortunately, they are also inefficient and non-transparent and the process from application to approval usually stretches over a long period of time. Inefficiency, lack of transparency, and complex permitting procedures for the construction of energy facilities are slowing down the development of the energy sector and are the main reason for a lack of interest in the BiH energy sector on the part of local and international investors. Also, the negative political environment makes BiH less appealing to investors and creates the impression that even the government-owned company is unable to pass through this process without major obstacles. This renders the business environment in the energy sector investment-unfriendly and is the cause of the decline in the number of investments in BiH electricity production facilities. Conclusion 11: Despite tremendous efforts by EIA to identify all the impediments to investments, prepare comprehensive sets of recommendations for their removal and the necessary policy documents, and build institutional capacities during the last period, there has been no major breakthrough in improvement of the investment climate. Recently, there has been significant work done to improve the permitting procedure. This work was initiated by EIA and is being conducted in cooperation with GIZ, which is performing some interventions through its own project “Promotion of Renewable Energy Sources in BiH.” This activity is supported by all government authorities and authority representatives that are directly involved in the permitting procedure. They are participating in the process with the intention to improve the efficiency of the permitting procedure to make the process easier for investors and to attract more investment into the BiH energy sector. But despite that, due to a new trend of buying finished or half-finished projects and the negative political environment, there is a dearth of major energy projects, and even the small ones are taking a long time to be completed. Conclusion 12: The energy retail market in BiH is functional; however EIA still needs to achieve functionality in Result 2.1: Distribution System Operator (DSO) as an independent market facilitator. Completion of this element is one of the preconditions for the development of other elements of an open electricity retail market. Stakeholder representatives who are informed about the electricity market recognize the negative impact of the delay in the overall process of electricity market development, and consider that the basic preconditions for further development of electricity market in BiH. Conclusion 13: Unless the DSO is functional, it will be hard to adopt and implement the integration of renewable energy sources into the energy market (Result 2.4.) and implement the debalance allocation mechanism. The new system for balancing and procurement USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 31 of ancillary services began functioning as of January 1, 2016. The task of the new system was to decrease imbalances in the internal electricity market and with neighboring countries, and to conduct procurement of ancillary services based on market principles. EIA worked closely with the ISO and SERC to prepare the new system for its kick-off in January. In the first three months of implementation, the system proved to be functional and effective. While stakeholders were generally satisfied with the new system, they identified problems that became evident in the course of actual implementation. EIA, in cooperation with stakeholders, pointed out that the Distribution System Operator (DSO) as an independent market facilitator is yet to be functional and that rules for the balancing settlement mechanism need to be developed and implemented to achieve long-term functionality and system effectiveness. Conclusion 14: Stakeholders believe that the “Guide for Investors” is a good basis for institutions to start to improve the investment environment in the energy sector. Stakeholders expect that EIA can and should use the “power of USAID and the US Embassy” to exert pressure on politicians and institutions to move forward with implementation of the proposed interventions. The lack of readiness of institutions to eliminate impediments to investments is a main reason for the low level of investments in the energy sector although unexplored possibilities of the potential of BiH’s natural resources are evident. A significant number of KIs believe that “stronger” pressure on representatives of institutions and political structures by US Government agencies is necessary and firmly believe that it can produce positive results. EIA has completed preparation of the “Guide for Investors,” and it will be reported as a deliverable in the current quarter. Presentation to the relevant ministries is planned in due course, and official promotion of the document will be organized in a special event for that purpose. F. RECOMMENDATIONS (RQ2 – EIA) Recommendation 5: Stakeholders and beneficiaries strongly value the continuity of USAID interventions in the electric power sector. The reforms and improvements achieved in all aspects of the energy sector could not have been implemented without USAID support. To ensure the sustainability of reforms. USAID should continue those interventions. This recommendation is based on the extremely high reputation among stakeholders and beneficiaries from all over BiH for USAID’s immeasurable contribution to the institutional development and reforms in the BiH energy sector during last two decades. At the same time, the energy sector is one of the most important economic sectors and the most promising for the future economic development of the country. The 2012–2016 CDCS for the US Mission in BiH identified economic growth and the strengthening of economic aspects of governance relevant to the business environment, as well as private sector development, as key foreign policy priorities for the US Government. As stated in the USAID CDCS for BiH, the country needs to improve its competitiveness and productivity, attract investment, and make economic policy reforms. Economic growth provides the material basis for progress in all other dimensions of development and long-term stability. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 32 In the fourth year of EIA implementation, it is evident that the positive effects of the technical assistance provided are maximized and harmonized with the real needs of the energy sector reform process, and are fully in accordance with the needs of beneficiaries and stakeholders. These effects were confirmed by all KIs in the interviews conducted during the evaluation. EIA has developed very good relations with other donors and has harmonized its own interventions with them to ensure that all interventions are complementary and to maximize the quality of assistance provided to BiH stakeholders and partners in the energy sector reforms. This recommendation serves only to emphasize the needs and the rationale for continuing the USAID interventions in the reform of the BiH energy sector. The scope of the interventions and the amount of support are the subject of the USAID strategic focus, but continuity of the technical assistance in energy sector has extremely high value for BiH and significantly contributes to USAID’s high reputation among all energy sector stakeholders. Recommendation 6: USAID should be flexible and realistic in setting targets for its interventions because energy sector reform is a long-term and complex process, especially given the constitutional complexity of BiH organization. Definitions of the objectives and expected results should be adjusted to the complexity of the energy sector. This recommendation is based on the evident complexity of the EIA components and the demanding nature of the required results, which were recognized during the evaluation. Although the critical assumptions and development hypotheses defined in the results framework include two major elements, energy system reform is a long-term process, and BiH is a country with a complex political organization. Also, each of the EIA’s components could be implemented as a standalone activity due to the fact that even the people responsible for them belong to different departments with different beneficiaries and stakeholders and have very little or no horizontal communication among themselves. A wide portfolio of intervention components minimizes the possibilities for potential synergy effects in the interventions and generates a high number of involved representatives, making coordination of sub￾interventions very demanding and complex. In combination with setting strong requirements for the results, in complex environment like the BiH energy sector a large number of interventions may generate even more demanding and hardly achievable indicators. The intention of this recommendation is only to point out the reality of the dynamics of the reforms in the energy sector and to suggest setting a relevant and appropriate range of interventions to achieve energy sector reform in BiH. Recommendation 7: USAID should consider more detailed planning of interventions during the preparation stage of the Activity. Advance discussions with beneficiaries regarding their responsibilities for the implementation of interventions may be helpful. This recommendation is based on a common observation made by twenty-one (21) out of twenty-eight (28) KIs interviewed who were asked about obstacles to the implementation of measures agreed upon at a technical level and resistance to adopting optimal technical solutions. Despite the clear USAID procedural steps for the preparation of interventions, including a detailed examination of the development hypothesis, critical assumptions assessment and other preparation USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 33 activities implementation of agreed-upon interventions in the reform of energy sector in BiH is very challenging. Postponing decision-making, prolongation of the adoption of the laws or regulations, or unexpected complications occur unpredictably. Political interests are the key reason why the obstacles are often in direct conflict with economic interests and needed reforms. An illustrative example is the sanctions adopted, after repeated efforts, by the Secretariat of the Energy Community against the Ministerial Council of BiH because of the delay in energy sector reforms related to the gas sector, lack of unbundling of public utilities, and deregulation of energy generation prices in RS. Despite the sanctions, these issues are still not resolved. This recommendation attempts to address this important issue this by proposing additional steps in USAID’s process of preparing for the interventions. Advance discussion with potential beneficiaries about their interest in the intervention, followed by an officially signed commitment document, might be made a precondition for the involvement of BiH institutions in the USAID activity. Lack of a clear commitment could eliminate a particular institution from receiving USAID support in the interventions. Recommendation 8: USAID should assess support for the establishment of an electricity stock exchange in BiH for further development of the electrical energy market. Some experts consider such an institution to be an urgent requirement. With three existing electric power companies and a number of registered energy suppliers, BiH has the necessary preconditions for the development of a stock exchange for electric energy and its integration into an evolving regional stock exchange network. The initial intention of this recommendation is to emphasize need for further support of reforms in the BiH energy sector that will lead to further development of energy market and catching up with development trends of regional energy market. The European Target Model is a market model that represents the main source of guidance for the European single energy market design. Under its framework, its principles imply that wholesale energy markets should be integrated at the daya-head stage via implicit auctions of available cross-border transmission capacity, through a market coupling process. Intraday markets should also progressively be integrated to provide greater opportunity for renewal generation investors to adjust their positions close to real time. In the end, also balancing markets, and more generally ancillary service markets, will be integrated so that consumers benefit from lower balancing costs and improved security of supply across the European Union. The integration of the day-ahead electricity markets in Europe should lead to a single multi-area market that will integrate (couple) the local/national markets. To do this BiH must start on the path toward the creation of a Power Exchange base on a day-ahead market platform as a preliminary step to its integration with neighboring markets. This would be fully consistent with the purpose and overall target of Western Balkan 6 (WB6). The WB6 Programme Steering Committee for the day-ahead market integration interventions intends to develop a road map for optimal and feasible day-ahead market integration among the WB6 and with EU USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 34 member states. It is supposed to lead to the coupling the WB6 region into the Multi-Regional Coupling (MRC). If BiH is included in these processes it will imply that BiH, by continuing further energy sector reforms and entering the WB6 Initiative, is giving a clear commitment to establish the EU Target Model. This is the main intention of this recommendation. Recommendation 9: USAID should consider taking a wider role in building competence and expertise within the energy sector. Specifically, interventions in the education sector at the university level should contribute to the education of a younger generation of engineers ready for the transition to a market-oriented energy sector in BiH. This recommendation is not related to any of the findings and conclusions of this evaluation, but is based on a view of the wider picture by the interviewed stakeholders and a consultant. Looking at the transition that the energy sector is going through and the obligations that await Bosnia and Herzegovina in the future, it is clear that one generation of experts and institutional representatives will not be able to complete all the needed reforms and that succeeding generations will have to take over at some point in the process. Current educational programs at the universities deal with this subject briefly, which gives rise to the question, how and who will take over the work so that BiH continues to keep up with the progress made by other countries. To ensure that the BiH energy sector transition doesn’t slow down or fail, it is of the utmost importance to design specific curricula, or integrate this topic into other courses at the BiH universities, to prepare the next generations to assume control. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 35 FISCAL SECTOR REFORM ACTIVITY (FSRA) G. FINDINGS (RQ2 - FSRA) Finding 33: Eight (8) out of twenty-five (25) KIs were acquainted with TA aimed at the Fiscal Council of BiH. In the interviews conducted, eight respondents (beneficiaries and non-beneficiaries) were able to provide some information or had heard about the intervention aimed at the Fiscal Council. Three of those KIs stated that they believe the actions aimed at the Fiscal Council were either irrelevant or not useful because the council is a highly politicized institution. One KI confirmed that an additional study has been written and that the situation in this intervention is not the responsibility of USAID or the implementer. Finding 34: Only three (3) KIs out of the eight (8) who provided some information about the actions aimed at the Fiscal Council of BiH could identify specific results related to improved reporting and coordination of the Fiscal Council. Three KIs knew about the inclusion of “t-1” budget execution data in the fiscal tables for determining the Global Fiscal Framework. All three KIs considered this activity as relevant, but again, due to lack of political will, insufficient for the overall planned objective. Finding 35: There has not been any harmonization of reporting of public finance statistics between the entities. Such harmonization is important in order to facilitate more transparent, accountable, and consistent reporting to the EU. Even though three KIs recognized the significance of this issue, none of them could provide any information regarding the implementation of consistent reporting in line with EU standards. One KI indicated that this activity has been abandoned. In fact, some institutions and donors are independently (of the FSRA) trying to adjust and adopt EU standards of fiscal reporting. Statistics offices in BiH have been recognized as relevant institutions unrelated to FSRA activity. One KI, unrelated to FSRA, had attempted to receive training related to EU fiscal reporting in line with ESA 95, but the request was declined. Another KI indicated that this activity is planned to be implemented with the statistics offices and not under FSRA. Finding 36: The intervention related to microsimulation modelling aimed to strengthen analytical capacity of the entity ministries of finance. This is considered successful in RS, and Brcko District while in FBiH, further assistance is required. The improvement in analytical “[B]ut I don’t see any progress in that segment, nothing is done concretely, with the exception of one study related to that system. [O]ne study was done on the reporting system and how the reporting system is functioning at the level of the canton, Federation [FBiH], thus, the state level. Therefore, we were told what we already know. The goal was to improve coordination through the Fiscal Council. I immediately told them that it is not possible. Therefore, the goal was set at a very optimistic level, without prior consultations. On the Fiscal Council, exclusively due to political reasons, it is difficult to make concrete improvements.” “[W]e didn’t achieve, even by a long shot, what we wanted, but accountability for that is not on the people that worked on the project on behalf of USAID. Accountability is on somebody that desired to gain through that one step, another five steps, diverted the project in wrong direction, and encountered opposition from Republika Srpska, thus not ensuring the goal that we all desired; but people working on the project are not responsible for that.”. “Fiscal Council is a joke – they meet only for IMF Programmes three times a year. Fiscal Council is structured the wrong way. We told USAID not to work on this but to spend more on things that are attractive and doable.” USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 36 capacities in the RS Ministry of Finance has reached a more advanced stage than in FBiH. However, only one person in the RS Ministry of Finance is trained and fully capable of independently using the developed microsimulation models, which makes the Ministry’s institutional capacity limited and vulnerable. Further actions are planned by the end of the activity, but there have been delays in the collecting the data related to the inclusion of social benefits in the model (social security contributions (SSC) have been included). In RS, four STATA software licenses have been purchased. In the future, the beneficiaries hope that more staff will be able to use the model independently, and an additional manual has been developed. In the FBiH Ministry of Finance, the staff received training but are unable to use the microsimulation independently. The model does not yet include SSC, but the KIs stated that they expect to include SSC as planned by the end of the activity. A microsimulation model for corporate income tax has been developed in RS but faces delays in FBiH. Actions related to the macroeconometric modelling of revenue projections in both entities is in progress. Three KIs who are direct beneficiaries, together with one KI non-beneficiary, were all very satisfied with the professionalism and knowledge of the technical staff and the technical support that they have received under this intervention. In addition to software purchases, they have received training. Further actions are planned by the end of the activity under this objective. Finding 37: Other beneficiaries (cantonal ministries of finance) expressed their interest in being included in the intervention related to strengthening analytical capacity. This finding especially relates to the education and training of the cantonal ministry of finance staff; two out of the three relevant KIs expressed interest in and even contacted the Federal Ministry of Finance about being included in the trainings, but the request was denied. According to the federal law on the budget, the ministry of finance is responsible, inter alia, for revenue projections. The ministry is planning to extend the analytical capacity of the cantons in the future. During the interviews, KIs from the cantonal ministries of finance were especially interested in macro￾econometric revenue forecasts modelling because most of the revenues from the corporate income tax and all revenues from the personal income tax are either cantonal or shared cantonal–municipal revenues, respectively. Currently, the cantonal ministries receive revenue projections from the federal ministry of finance. Finding 38: Four (4) beneficiaries confirmed that in all ten (10) cantons, BMIS software has been installed, but only one beneficiary has begun using the software in a way that includes all budget users. The results show that nine KIs are acquainted with the BMIS software purchases. Four KIs confirmed that in all 10 cantons, BMIS software has been installed. However, three direct beneficiaries stated that only one beneficiary in the cantons has begun using the software and implementing budget instructions, whereas the others have installed the software, but have not yet “We were not included, we fought, but we were not included.” (Reply to the question on participation in macroeconometric revenue forecasts) “We tried to have one of our members coopted in the working commission or group, the one that does that micro￾simulation, but we had no agreement…” We have asked that the members of the Federal [FBiH] Ministry of Finance to have training on that micro-simulation, to make those income tax simulations … [P]erhaps, in future we should contemplate involvement of cantons in that process in some form, because we usually get the law served to us on a platter, and the law relates to us most of all.” USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 37 began implementing it among budget users. All four KIs confirmed that the implementer covered the cost of software maintenance during 2018. After completion of the data collection process in April 2018, the evaluation team received information that some other cantons had started using the software, but the team was not able to confirm that information. There is sufficient time before the end of the activity to fully implement this FSRA intervention in all cantons. Finding 39: The implementation of BMIS in the cantons in FBiH has been perceived as a positive outcome, but further expansion would be beneficial. The implementation of BMIS in cantons in FBiH has been perceived positively. Even before the software installation, cantonal budgets were planned in numerous Excel files (different in each canton, depending on the number of budget users). As part of this process, a large number of tables have been manually retyped in the federal ministry of finance, which is time consuming, and the probability of making mistakes is high. Thus, little time is left for other more demanding activities such as analyses or projections. Not all relevant KIs could confirm that they were included in the implementation of BMIS from the beginning of the project (two out of four). Furthermore, the KIs expressed concern related to the cost￾benefit analysis of the intervention and the speed of implementation, the small number of BMIS-related trainings, and the duration of the trainings. Finding 40: According to the KIs, beneficiaries would appreciate further assistance related to BMIS implementation. As previously noted, beneficiaries expressed interest in further assistance related to BMIS implementation, especially in terms of training and education for the cantonal staff as well as budget users. However, such assistance is beyond the scope of the implementer at this stage. Finding 41: The implementation of the treasury system (TS) in local communities11 in FBiH began late in the activity (during 2016 or 2017). Out of six relevant KIs, five stated that they began cooperating with the project during 2016 and 2017 (one KI did not provide an answer). The cooperation under this activity was planned jointly with the Public Administration Reform Coordinator’s Office (PARCO), with whom a MoU was signed only in February 2017. That may have caused the delay in the implementation of the treasury system in local communities. The cooperation began with completing the survey organized jointly by the implementer and the Association of Municipalities and Cities in FBiH, to determine the level of implementation of the treasury system in each local community. The implementer was responsible for implementation in the identified local communities that had partially implemented the treasury system, whereas PARCO was responsible for those that had not implemented any aspect of the treasury system. PARCO was unable to implement this, so FSRA will provide support to any remaining local communities that do not have a TS. 11 The term “local community” refers to municipalities and cities in FBiH. “[B]y default, entities talk to donors, cantons talk less. But we have delays in project implementation; those delays happen because we learn about projects late, literally we learn about them late. Sometimes projects are implemented up to two years before somebody comes to the canton. By default, this is what is happening, and we are not really informed.” USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 38 Finding 42: Six interviewed KIs representing local communities in FBiH have installed TS software. All six local communities have purchased ITINERIS software (TS software) and are paying for its maintenance. In interviews, these beneficiaries stated that they received training and education from the implementer (organized jointly with the software provider). The implementer provided a website where beneficiaries can ask questions and exchange information and related procedures, rulebooks, instructions, etc. Three of the KIs reported that the website platform was a success but had been discontinued in the last six months. These beneficiaries expressed a need for the website to be reactivated. In these interviews, the KIs explained that local communities in FBiH mostly communicate regionally (neighboring municipalities within each canton, due to a very similar assignment of public expenditures) so it appears that the platform for data exchange could be organized regionally. Finding 43: Understaffed municipalities and lack of political will were the most frequently mentioned implementation barriers for a fully operational TS in local communities in FBiH. The two most frequently cited obstacles to full implementation of TS in local communities were understaffed municipalities (three KIs out of six) and a lack of political will to implement TS (two KIs). Both problems have been recognized as internal issues unrelated to FSRA. However, both problems have not been recognized in the available documents. Hence, even though it is beyond the scope of the implementer, none of the six local communities have admitted any additional budget users into TS. Finding 44: Six KIs confirmed that they received training and education, but they are expecting further technical assistance by the end of the activity. All relevant six KIs stated that they have received some education, namely, that they have participated in seminars and trainings. The cooperation with the implementer was assessed as very professional and positive, but in two cases, cooperation had ended (the website was discontinued). However, the KIs are expecting further training by the end of the activity, especially since the necessary TS software has already been purchased outside of FSRA. Finding 45: The cooperation and technical assistance related to the implementation of TS in health care institutions in RS has been assessed as positive. Two relevant KIs reported that they have received technical and material support as planned in the intervention, and commented on the high degree of involvement and professionalism on the part of the implementer’s staff. The implementation began successfully with the FSRA purchase and installation of the necessary hardware and software. Necessary acts have been developed and budgetary beneficiaries have received some training. Finding 46: One key beneficiary was not included in the design process for introducing TS in health care institutions in RS. The RS Ministry of Health and Social Protection (one of the two main beneficiaries) was not included in the intervention design process. The MoU was signed in May 2015, but the ministry was informed about the intervention only in September 2015. Cooperation began with the creation of a working group for the implementation of TS in RS health care institutions. “But I really don’t know why you stopped with those answers, you had that link, didn’t you, where you could post a question about that switching onto risky business, and there we could address so much of our inquiries. We have plenty of doubts, a bunch of us, all municipalities that I am in contact with, and we all have doubts, but replying stopped there.” USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 39 Obstacles to communication between the two ministries (the Ministry of Health and Social Protection and the Ministry of Finance of RS) were identified that might cause delays and implementation issues. This information was confirmed by both beneficiaries. Finding 47: Both relevant KIs believe they were unprepared for the implementation of the treasury system in the health care institutions in RS. As the intervention progressed, a few implementation difficulties occurred (unrelated to FSRA) from the beneficiaries’ perspective. Lack of reliable data related to the arrears of health care institutions was the major obstacle identified in this intervention. The beneficiaries attributed these obstacles to their insufficient preparation for undertaking the demanding process of including health care institutions in the RS treasury system. The complexity of the issue has also been recognized by other donors, such as the World Bank where both the World Bank and FSRA conduct complementary activities. Finding 48: Only two health care institutions in RS are included in TS. Other health care institutions are not a part of TS because of their arrears and because of their resistance to TS. Both relevant KIs have confirmed that the application of TS in health care institutions in RS is currently in the pilot stage at two health care institutions. These two institutions do not face any accumulated unpaid liabilities or arrears, and that is why they were selected to be included in the pilot phase of the introduction of TS. The KIs stated that other health care institutions might not have been included in the TS system because of their resistance to TS implementation. One KI stated that such resistance might be related to political reasons because, in the past, health care institutions have been an ideal place for political employment. Finding 49: Two expected results related to this intervention have been achieved: (i) improved legislation and procedures and (ii) selection of the most viable solution for the public debt management system. Three KIs were directly involved in the intervention’s design process. Two expected results have been confirmed as fulfilled from the implementer’s side: technical support regarding the development of normative acts (such as a code of ethics, according to one KI) and the selection of the most viable solution for the public debt management system. One relevant KI stated “Our insisting on the treasury system, I must admit it might be a little harsh to say, it was a premature step. But it would have been much better and more correct if we did the health sector reform first. We had it set in some four phases. The first was an analysis of outstanding obligations, the second was the introduction of the treasury system, the third was standardizing the network of health institutions, and the fourth was an analysis of new avenues for financing. This analysis for financing is something that is in front of us in the future, but I think that it would be a better course of action if we first standardized health institutions, meaning that we could say that some clinic in some small municipality should not be there, [some] hospital at some place should not be there, and then we start with this job, and that would be in a spirit of reform along the way later. Why are we introducing a treasury system into something that will be abolished tomorrow? I must admit that I was one of those that insisted for these deadlines to be shortened at the beginning, and I must say that I was not aware of all the problems that appeared later on. I was not, especially, aware of the financial implications of the problem that appeared later on, although I claimed that that was a key reform that we must finish. I believe that in the very moment when the project was designed we did not see what we see now; it was designed some three years ago in May 2015. Now when we talk in May 2018, maybe in May 2021 I will know that in May 2018 I did not see all. I will say it, in the moment when we were talking about it, we from the Ministry that insisted on this did not foresee problems of such magnitude that came later on.” USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 40 that the implementer fulfilled all expected results in terms of technical help with the creation of required documents and that communication and implementation were effective and efficient. Finding 50: The public debt management system was not installed, and the intervention was cancelled in October 2017. The final expected result—installed operational public debt management software in both entities—has not been fulfilled. In the interviews conducted, it was confirmed that this activity was cancelled in October 2017. Four of the five KIs interviewed, highlighted the fact that cancelation was not the responsibility of the implementer, which fulfilled its commitments, but rather issues related to the choice of partner (PARCO). KIs also stated that the implementer attempted to address this issue by reaching out to other donors. Two KIs expressed the hope that by the end of the project period, some solutions to this issue will have been found. Finding 51: Both tax administrations (FBiH and RS) have attained most of the expected results in line with the defined objectives: e-archiving/e-filing and reduction of costs in RS, an improved ADRIS system, and 23 databases integrated into a single one in FBiH. Only those KIs directly involved in this intervention, together with the staff that was included in the direct design of FSRA activities, were able to comment on this issue. In the Tax Administration of RS, all expected results in terms of e￾filing and e-archiving have been achieved, with significant estimated cost reductions for all interested parties. E￾filing and e-archiving are estimated to have saved 2.3 million BAM for all included parties (Tax Administration, taxpayers). Similar results have been achieved in FBiH with new e-services and improved inspection efficiency. The system has been made more efficient with 23 databases integrated into one and an increase in social contribution revenues has been registered, which is to some extent a result of this activity. Finding 52: The FBiH Tax Administration expects further technical assistance in order to build capacity for better tracking of gaming industry revenues. By the end of the activity, the FBiH Tax Administration expects further technical assistance so that ongoing interventions can be finalized. Unlike RS, in FBiH there is still a lack of quantitative data to precisely confirm expected results. Results related to ongoing activities for stop-filers are expected in October 2018. Finding 53: Results related to the introduction of a single registry of business accounts in FBiH have been achieved. Even though most KIs were not informed about this intervention (only five out of 15 KIs were aware of it, excluding the direct beneficiary and the implementer), this was a high-visibility intervention since those five KIs were not directly included in the intervention (they had to open their own main account). The two directly involved KIs confirmed that the expected results have been achieved. Further interventions are planned before the end of the activity, such as linking the FBiH tax administration system with the registry. However, this intervention requires legal changes to the law on tax administration. “[T]he estimate was done for 2 million and 300 thousand BAM last year, electronic submitting of 1002 forms, participants, taxpayers, tax administration. The yearly savings for taxpayers are 750,000; for tax administration 500,000; related to the electronic archive 170,000; related to the electronic mailbox we have some framework of 130,000 for taxpayers; and 750,000 for tax administration on a yearly basis.” USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 41 Finding 54: KIs believe that this intervention has enhanced the transparency of the business operations of legal entities in FBiH. Two KIs confirmed that, unlike other interventions, this intervention directly contributes to greater transparency. Finding 55: Brcko District does not have a single registry of business accounts, which undermines the results achieved in RS and FBiH. In the interviews conducted, KIs said that the single registry of business accounts in Brcko District does not exist. Sixteen banks have branch offices in Brcko District, which may indicate shadow operations of businesses there. If three registries were established and could exchange data, then this risk would be eliminated. Finding 56: To enable the FBiH Tax Administration to block the main account of a business, the law on tax administration will have to be amended. The evaluation team was informed that the FBiH Tax Administration has access to a single registry of business accounts. However, in order to conduct activities related to forced debt collection, the law on tax administration would have to be amended. At the present time, Tax Administration officials have no legal authority to block a business’s main account. However, the Tax Administration can apply to the courts to block the account. Finding 57: The publicly available registries of parafiscal fees in both entities have been perceived as positive outcomes, but further interventions related to the elimination and reduction of parafiscal fees are required. For this result, all KIs except one said that they were informed about this activity. This high rate is also a result of media coverage of this topic. Four KIs stated that further elimination and/or reduction of parafiscal fees should be mandatory in both entities. That would make the registries of parafiscal fees meaningful. Different subnational levels of government, specifically in FBiH (according to two KIs) have, independently of the project, already started activities related to reduction/elimination of parafiscal fees in their jurisdictions. Finding 58: One parafiscal fee that was abolished in RS is the result of IMF requirements. The evaluation team was informed on two occasions that the parafiscal fee that was eliminated in RS is not the consequence of project-related interventions. The fee was eliminated due to a request by the International Monetary Fund. Finding 59: The registry of parafiscal fees is more advanced in RS than in FBiH. In the interviews conducted, the evaluation team learned that legislation and other work related to the registry of parafiscal fees has reached a more advanced stage in RS compared with FBiH. Respondents explained that this is due to a complex federal organization. Also, research shows that RS began gathering data for the design of the registry before the project started. Finding 60: Non-beneficiaries emphasized how important it is that the business community have information on how the collected parafiscal revenues have been spent. Two non￾beneficiaries who are in direct contact with the business community stated that the business community is willing to pay parafiscal fees if, in return, they are provided with information on how the collected revenues have been spent. This directly contributes to greater transparency. Among non-beneficiaries, registries of parafiscal fees are very popular because three of the eight non-beneficiaries are conducting individual activities in this segment. These relate to the development of additional registries of parafiscal USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 42 fees (either regional registries applying a bottom-up approach, from municipalities to the entity/state level) or industry specific ones (for a specific industry activity, for example, extraordinary transport). One representative of a non-beneficiary institution indicated that they are (as part of other USAID activities) working towards the reduction of parafiscal fees (approximately 20 parafiscal fees), especially at the local level. Another non-beneficiary stated that they wanted to be included in this activity. H. CONCLUSIONS (RQ2 – FSRA) Conclusion 15: There have been limited achievements regarding improved coordination of the Fiscal Council of BiH and consistent reporting to the EU. The findings indicate that, primarily due to lack of political will, there have been limited achievements for two expected results: (i) improved coordination of the Fiscal Council of BiH and (ii) consistent reporting in line with EU reporting standards. Some respondents stated that they are conducting individual activities related to improvements in reporting consistent with the EU standards and are planning to include statistics offices rather than ministries of finance. Conclusion 16: Although the improved analytical capacity of the RS Ministry of Finance is considered a success, the evaluation team could not confirm that the analytical capacity of the FBiH Ministry of Finance have been improved. At the time of writing, the evaluation team could confirm that RS Ministry of Finance staff have been able to independently use the developed analytical tools. On the other hand, the team was informed that FBiH Ministry of Finance staff cannot use the developed analytical tools independently, but that they are expecting to be able to do so by the end of the activity. Conclusion 17: The analytical capacity of the RS ministry of finance has improved, but further interventions are expected. By the end of the activity, it is anticipated that further interventions will contribute to the expected results. These relate to the independent use of microsimulation models, improvements in macro-econometric modelling, etc. Furthermore, beneficiaries expressed interest in further strengthening their analytical capacities related to macroeconomic analysis and in receiving additional technical assistance related to the planned elimination and reduction of parafiscal fees (in RS). Conclusion 18: Microsimulation modeling training will not produce satisfactory results with public institution staff who lack an adequate prior knowledge. Prior to implementing this type of intervention, USAID/BiH should consider asking ministries to form analytical units and staff them with people who have the requisite prior knowledge. Conclusion 19: Most of the results regarding adoption of the BMIS system are expected to be achieved by the end of the activity. Even though only one canton has begun using the BMIS software, others have received software and trainings and are willing to use the system. However, they are still expecting further technical assistance in the form of more education and trainings. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 43 Conclusion 20: The implementation of TS in local communities in FBiH requires further action through the end of the activity. Even though cooperation with the implementer has been assessed as very positive, the implementation began late during the activity implementation period (2016 and 2017) and has ceased in the last six months. Hence, the beneficiaries recommended several ways in which they hope the implementation of the treasury system will fully completed: (i) relaunching the website in order to exchange experiences and documents with other local communities, and (ii) having the implementer write a letter to re-start the actions related to the introduction of TS in local communities. Conclusion 21: The introduction of TS to the RS health sector was premature given the current state of the health sector in RS. However, some progress has been made. Both KIs agreed that the introduction of TS to the RS health sector was premature as demonstrated by several obstacles the beneficiaries experienced. These relate to a lack of high quality data on health care institutions’ arrears and the resistance of health care institutions to join TS. Conclusion 22: The intervention to strengthen the public debt management system in BiH was cancelled in October 2017, but important lessons were learned. All relevant KIs expressed concern related to the choice of partner in this intervention and have stated that the implementer was not responsible for the cancelation. This intervention is of great importance to achieve better and more consistent reporting in BiH (in both entities and Brcko District), but in the future it will require more careful planning. Conclusion 23: Expected quantitative results related to the tax administrations in BiH were achieved in RS, whereas those in FBiH are expected to be achieved by October 2018. The presented quantitative data show that the expected results related to the Tax Administration in RS have been achieved. In FBiH a comparable achievement is expected by October 2018. Overall, cooperation with the implementer and technical support provided has been assessed as positive. Conclusion 24: Implementation of a single registry of business accounts in FBiH was assessed as necessary, and the implementation was conducted very quickly. Further actions are planned before the end of the activity. Without this intervention, the Law on Internal Payments Operations in FBiH would be ineffective. Also, this intervention directly contributed to reduction of the vast number of bank accounts. It also directly contributed to greater transparency. Further actions are required for the effects of this intervention to be fully realized. These actions relate to the introduction of a single registry of business accounts in Brcko District and changes in the law in FBiH so that the Tax Administration would have a legal basis to block the main account of a business when necessary. Conclusion 25: The high-visibility intervention related to establishing a register of parafiscal fees directly contributes towards the Project 2.2 purpose. The evaluation team concluded that both beneficiaries and non-beneficiaries were acquainted with this intervention and perceived it to be very important as a transparency-enhancing mechanism. Non-beneficiaries, using different methodologies, have begun activities related to reducing parafiscal fees at the local level. Although the establishment of a registry is a positive action, further efforts are required by the end of activity so that parafiscal fees will be eliminated or reduced. One parafiscal fee that was reduced in RS, USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 44 however, is not a consequence of this activity. The registry of parafiscal fees has reached a more advanced stage in RS than in FBiH, where a set of recommendations has been put forward for its improvement. I. RECOMMENDATIONS (RQ2 – FSRA) Recommendation 10: Abundant caution is needed when defining politically sensitive objectives and expected results. Most of the relevant beneficiaries interviewed stated that interventions directed at the Fiscal Council of BiH are highly politicized. The expected result has not been achieved, and has been perceived as being set at a level that was too ambitious. A further question remains about the extent to which the beneficiaries were properly selected for the expected result. The approach that has been implemented through “bottom-up” analytical capacity building is a better way of fulfilling the aims of improved intergovernmental fiscal relations at the level of BiH because the advisory group of the Fiscal Council of BiH consists of representatives from both entities. Recommendation 11: Possible additional interventions related to the reporting of public finance statistics, in line with the EU standards, should include other beneficiaries, such as the statistics offices of BiH. Detailed data regarding public finance statistics are generally available at the Ministry of Finance. However, national statistics offices are usually the ones that provide data in line with the system of national accounts such as ESA 2010, which defines the rules for consistent EU reporting. This applies for IMF GFS statistics. For this reason, ministries of finance should provide sound and reliable public finance statistics, but statistics offices in BiH should be the ones that prepare reports in line with EU/IMF requirements. Recommendation 12: Further technical support related to macroeconomic analysis and macroeconometric modelling would be beneficial for building stronger analytical capacities in all ministries of finance, which should result in more reliable projections. Direct beneficiaries expressed interest in the continuation of technical assistance related to macroeconometric analysis, especially in RS, since most results have been successfully implemented within FSRA interventions. However, some additional recommendations and suggestions were made by respondents who were not included in this intervention. As related to education and training of the cantonal ministry of finance staff; two out of three relevant KIIs expressed interest and even contacted the Federal Ministry of Finance about including their staff in the trainings, but the request was not approved. The Federal Ministry of Finance plans to include the cantons in this project in the future. Cantonal ministries of finance are especially interested in such interventions since most of the revenues from corporate income tax and all revenues from personal income tax are either cantonal or shared cantonal–municipal revenues, respectively. Currently, cantonal ministries receive revenue projections from the Federal Ministry of Finance, which is in line with the Federal Law on Budgets. With macro-econometric model, they would be able to track and project their revenues more accurately and could make suggestions and recommendations for possible legal changes in personal income tax (PIT) and corporate income tax (CIT). USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 45 Recommendation 13: More training and education related to the implementation of BMIS (program budgeting, budget beneficiaries, etc.) would be beneficial by the end of the activity. Beneficiaries suggested that additional training and education (up to two days) related to the implementation of BMIS be provided because past trainings were considered too short. Additional training related to program budgeting and education, and inclusion of budgetary beneficiaries in the BMIS application, has also been recommended as a future intervention. Trainings and education should be carefully tailored to the beneficiaries that express interest and have been active in the advancement of the implemented interventions. Recommendation 14: In the future, BMIS should be connected with the treasury system. Furthermore, technical support related to improvements in the treasury system has been identified as a need in some cantons in FBiH, especially in those with many budgetary users. The connection of BMIS with a reliable treasury system is a positive path toward more accurate and reliable program budgeting since it would rely on actual long-term data. Recommendation 15: Consider introducing local communities to BMIS. In the future, local communities could be connected to BMIS (by connecting them with the cantonal BMIS software), which would provide consistent vertical reporting. However, this should be considered a long-term activity because local communities in FBiH must first implement the treasury system among all budgetary users. Implementing the treasury system is a precondition to this activity because with a reliable treasury system, cantons and local communities could build a reliable database for future fiscal reporting. Budget management activities and the process of consolidating financial data at the level of cantons and FBiH would be significantly improved. However, this activity depends on political will and readiness to adopt the treasury system. Recommendation 16: Because the intervention aimed at introducing the treasury system in local communities in FBiH began late during the activity, it should be continued, but with a few important assumptions. Since this intervention was suspended, by the end of the activity, additional training and communication of best practices of local communities that have already fully applied the treasury system to those in the process of implementation would be beneficial. Those local communities with enough staff and the local political will to implement the treasury system require additional help, which could be regionally organized since neighbouring local communities (or those in the same canton) share similar issues related to implementation of the treasury system. Additional assistance in the creation of rulebooks and instruction manuals for successful implementation of the treasury system in local communities in FBiH would be beneficial. During this intervention, a website was developed by the implementer where local communities could exchange information and experiences. This activity ceased and FSRA should restart the activities related to the portal/web exchange of best practices, documents, etc., within local communities. Recommendation 17: The initiative related to adoption of the TS system in health care institutions in RS will require better planning by beneficiaries and a willingness to adopt such initiatives. Due to the complexity of the activity, beneficiaries entered the process unprepared and obstacles that occurred during the project were not recognized by the implementer or by the USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 46 beneficiary as potential risks. These include the amount and size of health care institutions’ arrears and the fact that not all key beneficiaries were included in the project design phase. Further assistance is required in the form of education and training and meetings or workshops to inform health care institutions about TS. This could reduce these institutions’ resistance to TS implementation. However, this is solely a beneficiary responsibility. The implementation of TS in health care institutions in RS is necessary for a variety of reasons, including enhancing transparency and reducing the potential for health care institutions to be used as instruments for political employment. An additional risk relates to the possibility that, over time, inefficient local health care institutions could continue to accumulate arrears and jeopardize local budgets. Another potential risk is that the implementation of the treasury system in health care institutions might highlight inefficient and therefore nonessential health care providers, which might cause local dissatisfaction and possibly lead to a reduction of health services being provided in some local communities. Recommendation 18: The intervention related to public debt management should be carefully tailored in the next project planning phase. The choice of a co-financing partner for the software procurement should be carefully tailored based on the willingness of all interested parties to participate in this activity. Since the evaluation team has been informed that the Ministry of Finance and Treasury of BiH has already introduced public debt management software, this intervention should be more focused on the entity levels, especially since the entity levels are direct beneficiaries of public debt. Fully functional public debt management directly contributes to more reliable and consistent fiscal reporting, which both entities have recognized the need for. For example, the Federal Ministry of Finance has recently adopted strategic documents that confirm their willingness to participate in this process. Due to BiH’s complex organization, implementation of this intervention could be considered at lower levels of government (cantons, local communities) to improve the public debt reporting mechanism. Recommendation 19: Interventions aimed at both tax administrations yielded positive results and significant cost reductions. The continuous technical assistance that USAID provides to the tax administrations of both entities is highly valued and appreciated and, most important, yields positive results in terms of cost reductions achieved by e-archiving and e-filing. In the interviews, KIs often perceived the tax administrations as institutions that only provide data for ministries of finance without any feedback. Also, the expertise of the tax administrations has not been used in terms of tax￾related legal changes. The evaluation team found that the tax administrations are mostly understaffed and that in the activity related to tax-related legal changes, the implementation has been poorly planned and there is a lack of funds available for the implementation of the laws (which is the tax administrations’ responsibility). In addition, the tax administrations have generally not been consulted regarding additional technical and staff needs. These institutions require technical assistance in these areas ensure positive results and contribute to the ease of doing business in BiH. Recommendation 20: Interventions related to a single registry of business accounts in BiH should be continued. The intervention in FBiH yielded the expected results and was conducted very USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 47 quickly. The intervention was also perceived as a transparency-enhancing mechanism. The potential benefits from the continuation of this intervention can be summarized as follows:  Interventions related to linking Brcko District business accounts into a single registry would contribute to greater transparency since this is not currently the case;  Connecting the three registries and the exchange of data between them would further enhance transparency, decrease the possibility of shadow operations, and better enforce debt collection. Additionally, it would be easier to acquire data on business accounts in BiH;  Possible further activities related to the introduction of a registry of public companies in FBiH (similar to the RS case);  Legal changes in FBiH in the law on tax administration are needed so that the data flow between Financial Intelligence Agency (FIA), the Central Bank of Bosnia and Herzegovina, and Tax Administration would be fully implemented. Recommendation 21: USAID should continue interventions related to registries of parafiscal fees and work toward the elimination and reduction of parafiscal fees in both entities and Brcko District. The importance of this intervention can be seen in the responses of non￾beneficiaries, which highlight the fact that the business community in BiH requires information on how the collected parafiscal revenues have been spent, thus directly increasing transparency. The creation of a register of parafiscal fees in both entities and Brcko District has been perceived as very beneficial, so the additional interventions related to elimination and reduction of parafiscal fees should be continued. Additional actions are needed in FBiH, by the end of the FSRA activity, to simplify the FBiH registry (making the register more user-friendly) and include the exact amounts of fees and charges that need to be paid in FBiH. The register of parafiscal fees in RS is more advanced than that in FBiH. Therefore, in RS, additional support in analyzing the effects that parafiscal fees have on the economy (individuals, business, etc.) would be helpful and could help eliminate unnecessary fees and charges. The positive RS experience and the methodology developed in RS could then be shared and serve as an example for FBiH. All four registries (BiH-level, the two entities, and Brcko District) should eventually eliminate overlapping fees and charges, but this requires strong political will at all levels. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 48 Evaluation Question 3: To what extent have the activities implemented under this project been coordinated and how have internal and external activity/project management practices supported progress toward the Project 2.2 purpose? What are the lessons learned for possible future interventions? J. FINDINGS (RQ3) Finding 62: The designs of EIA and FSRA have little in common, and interaction between EIA and FSRA activities has been limited. KIs confirmed that there were some attempts to establish cooperation between EIA and FSRA on topics related to VAT and nuisance taxes at the municipal level. There was also an attempt to establish cooperation with the FSRA activity related to VAT and nuisance taxes at the local level, but this was not successful. Finding 63: The only USAID/BiH venue providing an opportunity for coordination was the regular monthly project meeting. However, that practice was abandoned with the change in the Mission’s management. Finding 64: The Mission’s existing coordination mechanisms do not provide adequate opportunity for greater cooperation and coordination between individual USAID/BiH activities. According to KIs, cooperation between individual USAID/BiH activities is predominantly informal and based on personal relationships between COPs. The Mission’s meetings with the implementing partners are not an adequate platform for developing stronger cooperation and coordination between activities. Finding 65: EIA and FSRA have better and stronger cooperation with activities implemented under other projects than with each other under Project 2.2. A large part of EIA’s and FSRA’s work is related to the introduction of changes and improvements to the legal and regulatory framework. Both activities are faced with the challenge of moving proposed changes through the process of formal adoption by relevant institutions. To ensure better access to decision makers, both activities seek the assistance of the USAID/BiH’s activities working under the Democracy and Governance Office (DEMO). There has been close cooperation between Project 2.2 and Project 2.1, and good cooperation with Fostering Agricultural Markets Activity II (FARMA II) on biogas interventions. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 49 Finding 66: Overall, external cooperation and coordination is good, with a few exceptions. Eleven (11) respondents out of 32 confirmed that there was no overlapping of donors, and they assessed coordination between donors as good. However, some respondents highlighted the fact that they do face overlapping of donor activities under FSRA (in four cases, mostly the IMF, in one case the World Bank). Examples of good cooperation and coordination are cooperation with GIZ and UNDP. Collaboration between PARCO and FSRA on the procurement of public debt management software failed, which jeopardized implementation of the entire FSRA component. In order to mitigate the issue FSRA turned to the European Commission for help with the procurement of public debt software, but the request was denied. Except for a few cases, 12 there is no overlapping with other donors interventions. Finding 66: Both activities established better cooperation with FBiH and the state-level institutions than with RS. RS refused to participate in certain components of both activities. In the FSRA case, RS stopped all activities related to the C1 component and Result 2: Amended or eliminated ineffective regulations procedures of the Fiscal Council, which will result in more efficient work of the Fiscal Council. The absence of RS support to this particular intervention also prevented successful implementation of Result 3: The accounting standard harmonized, implemented and adopted for the purpose of Fiscal Council reporting, Analytical capacity of FC strengthened…. Further, there were certain political issues that prevented full introduction of TS into the RS health care sector. In the EIA case, RS stopped all activities related to the introduction of obligation schemes. While the RS Ministry of Spatial Planning works only with GIZ, the RS Ministry of Energy, Industry and Mining stated it has good cooperation with EIA and GIZ. Also, a majority of EIA’s policy and analytical work (such as case studies) was done for FBiH; very few studies were done for RS. K. CONCLUSIONS (RQ3) Conclusion 26: The evaluation team could not confirm that internal and external management practices at the project level made a significant contribution or supported progress towards the Project 2.2 purpose. Internal coordination mechanisms under Project 2.2 were nonexistent, and activities’ attempts to start cooperation were not successful. Conclusion 27: At the activity level, external coordination and cooperation with local stakeholders was perceived as good. However, there were challenges in acquiring RS 12 Complementary activities with the World Bank in regard to the introduction of the health sector in RS into TS. “USAID’s project (EIA) did not have, let’s say open doors and connection with RS, and they were working only in FBiH. So we brought RS on board for this activity. And then we agreed with them that we will lead for RS and they can lead for FBiH and then we can join and try to define interventions.” “There is a bit of resistance—strong resistance in some RS institutions—when it comes to obligation schemes.” —GIZ USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 50 Government support and cooperation on certain topics, such as strengthening the Fiscal Council and introducing interventions related to obligation schemes. Conclusion 28: The two activities cooperated with other international organizations to create synergies and to overcome obstacles they faced when cooperating with local institutions. For example, cooperation between GIZ and EIA enabled EIA to overcome a lack of RS Government interest in cooperating with the activity. However, a majority of interviewed RS institutions recognized GIZ as the lead agency in the energy sector. L. RECOMMENDATIONS (RQ3) Recommendation 22: USAID/BiH should consider the establishment of a formal and well￾structured coordination mechanism that would enable meaningful cooperation between individual activities within Project 2.2 and across projects. Recommendation 23: USAID/BiH should consider formalization and institutionalization of the project-level management function and establishment of a project-level coordination mechanism. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 51 ANNEX I: EVALUATION PURPOSE, DESIGN AND LIMITATIONS I. EVALUATION PURPOSE The purpose of the whole-of-project performance evaluation of the USAID/BiH Project 2.2, Improved Economic Aspects of Governance Relevant to Business Activity, is to: i. examine entire project progress toward stated project purpose and associated constituent activities progress toward project sub-purposes and project outcomes/outputs, ii. describe project and constituent activities coordination and implementation and key challenges and lessons learned, and iii. provide recommendations for possible future USAID/BiH interventions aimed at improving economic aspects of governance relevant to business activity. II. EVALUATION QUESTIONS In order to assess the entire Project 2.2, along with its constituent activities, this evaluation answers the following three research questions: RQ1. To what extent were the programmatic and contextual assumptions identified in the project theory of change sufficient to achieve the purpose of Project 2.2? What are lessons learned for future USAID/BiH planning at project level? RQ2. To what extent have the activities implemented under this project achieved their expected results, and to what extent have they been able to address and collectively contribute to the purpose of Project 2.2? How have constituent activities been implemented and what were the main challenges and lessons learned from stakeholder perspectives for any potential future interventions? RQ3. To what extent have the activities implemented under this project been coordinated and how have internal and external activity/project management practices supported progress toward the purpose of Project 2.2? What are lessons learned for possible future interventions? The evaluation questions are broadly based on the guidance from the Automated Directives System (ADS) 201 Additional Help for Whole of Project Evaluation. The first evaluation question studied the strategic planning process of the Mission at the project level by examining strengths and weaknesses of the project designed theory of change. The evaluation team analyzed how the Mission staff identified and used programmatic and contextual assumptions in the project theory of change and in decisions on activities to be designed and/or implemented under this project. Furthermore, the team analyzed the extent to which these assumptions have been validated in actual contextual conditions in the last three years and whether the expected results of the Project 2.2 USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 52 are still valid, relevant, and appropriate. The evaluation team also aimed to identify any potential gaps in the project’s design in order to help inform mid-course corrections or guide future project designs. To answer the first research question, the evaluation team conducted a review of CDCS/project/PAD and constituent activities’ planning and implementation documentation and databases (including M&E documentation), as well as conducted a review of secondary documentation relevant to EIA and FSRA, such as documentation from the European Commission, the Energy Secretariat, and the World Bank, as well as relevant documentation from relevant BiH government/public institutions (e.g. Ministries). The team also conducted key informant interviews (KIIs) with USAID/BiH staff involved in project-level planning and design (EDO Director, Project 2.2 Manager, and Program Office staff) and with the USAID/BiH CORs of EIA and FSRA. Furthermore, KIIs with EIA and FSRA implementers and stakeholders/beneficiaries also helped address this research question, as well as KIIs with other donors/international organizations relevant for the work in areas in which EIA and FSRA operate. For a broader perspective of Project 2.2, i.e. needed improvements of economic aspects of governance relevant to business activity, the team also reviewed secondary documentation from relevant international organizations and BiH government/public institutions and interviewed relevant donors/international organizations, relevant BiH government/public sector institutions, and relevant business associations and NGOs. We conducted a mini online survey of a sample of enterprises to examine their obstacles and priorities related to economic governance of business activity. We were able to obtain the email addresses of around 4,000 enterprises from TRON database to reach out to potential survey respondents. The second research question examined the project performance and the results that were achieved. We examined the contribution from all constituent parts of the project to provide insights into the extent of the achievement of each activity towards their expected activity results. Since this is a whole￾of-project evaluation, the evaluation team did not fully evaluate each of the constituent activities. However, as there are only two constituent activities under this project, the team examined the implementation of each activity with respect to their key expected results and the related challenges and lessons learned. This included an examination of stakeholder perceptions on whether the activities implemented under this project have been necessary and sufficient to achieve the project’s expected results and on any potential gaps where further assistance would be of strategic importance. Within the two constituent activities, this also included examination of stakeholder perceptions on types of assistance that are most valuable and any potential implementation challenges. To answer the second research question, the evaluation team conducted a review of CDCS/project/PAD and constituent activities’ implementation documentation and databases (including M&E documentation), as well as a review of secondary documentation and data. The team also conducted KIIs with USAID/BiH staff involved in project-level planning and design (EDO Director, Project 2.2 Manager, and Program Office staff) and with the USAID/BiH CORs of EIA and FSRA. The main source for addressing the part of question dealing with activities’ achievements and lessons are KIIs with EIA and FSRA implementers and stakeholders / beneficiaries, KIIs with other donors / international organizations relevant for the work in areas in which EIA and FSRA operate. For a broader perspective of what has been achieved in terms of expected Project 2.2 results, i.e. improvements of economic aspects of governance relevant to business activity in the past three years, the team also interviewed USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 53 relevant donors / international organizations, relevant BiH government/public sector institutions, relevant business associations and NGOs, and implementers and CORs. We also conducted a mini online survey of a sample of enterprises to examine their views on what has been achieved related to economic governance of business activity in the past three years. The third research question examined the strategic coordination process of the Mission at the project level by examining the interaction among the constituent activities as they contribute to the Project Purpose. The evaluation team examined actual processes employed by USAID/BiH for project-level coordination and whether they helped or impeded the achievement of the purpose of the project. We also analyzed coordination challenges and benefits. Furthermore, the evaluation team examined in what ways and how well have implementing partners of the constituent activities worked together and/or collaborated to take advantage of potential synergies and/or information sharing, and in what ways and how well have activities collaborated and coordinated with BiH government/public sector stakeholders and related interventions by other international donors/organizations. Moreover, we also examined the extent to which management practices both at project and activity level have supported progress toward the purpose of the project. To answer the third research question, the evaluation team conducted a review of CDCS/project/PAD and constituent activities’ planning and implementation documentation and databases (including M&E documentation) and conducted KIIs with USAID/BiH staff involved in project-level planning and design (EDO Director, Project 2.2 Manager, and Program Office staff) and with the USAID/BiH CORs of EIA and FSRA. Furthermore, KIIs with EIA and FSRA implementers also informed this question, as well as KIIs with other donors/international organizations, BiH government/public sector institutions, and business associations and NGOs relevant for the work of EIA and FSRA and Project 2.2 more broadly. III. DATA COLLECTION AND ANALYSES The research design employed mixed methods in order for the gathered information to be triangulated in addressing each of the evaluation questions using different data sources: 1. USAID/BiH CDCS documentation and PMP documentation related to Project 2.2 (PAD, Project LogFrame and project indicators and PIRS documentations); 2. Secondary documentation relevant to EIA and FSRA, such as documentation from the European Commission, the Energy Secretariat, the World Bank, as well as relevant documentation from relevant BiH government/public institutions (e.g. Ministries); 3. Secondary documentation from relevant international organizations and BiH government/public institutions related to broader project of achievements and gaps in terms of economic aspects of governance relevant to business activity; 4. EIA documentation/database review, including award documentation, work plans, quarterly and annual reports, as well as 47 key deliverables identified by EIA; USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 54 5. FSRA documentation/database review, including award documentation, work plans, quarterly and annual reports, as well as 26 key deliverables identified by EIA; 6. Sixty-three semi-structured key informant interviews; 7. Mini online survey of a sample of enterprises in BiH Overall, around 63 KIIs were conducted with the eight broad types of stakeholders: 1. USAID/BiH staff involved in project-level planning and design (EDO Director and Deputy Director, Project 2.2 Manager, and Program Office staff) – 1 interview 2. USAID/BiH CORs of EIA and FSRA - 2 interviews 3. EIA implementers – 1 interview 4. FSRA implementers – 1 interview 5. EIA stakeholders/beneficiaries – 22 interviews (out of 118 stakeholders/beneficiaries sent by EIA to the evaluation team, coming from 44 different institutions/organizations) 6. FSRA stakeholders/beneficiaries – 22 interviews (out of 121 stakeholders/beneficiaries sent by FSRA to the evaluation team, coming from 78 different institutions) 7. Other relevant BiH government/public sector institutions; and relevant business companies, associations and NGOs – 11 interviews 8. Other donors/international organizations – 3 interviews Exhibit 11. List of KIIs per Type of Key Informant Type of Key Informants Number of Interviews EIA Stakeholders/Beneficiaries 22 FSRA Stakeholders/Beneficiaries 22 Non-Beneficiaries – Relevant business associations, private companies and NGOs 11 Implementers 2 Donors 3 Other International Organizations 3 TOTAL 63 The data from above-listed sources was triangulated to address the same questions/sub-questions from multiple perspectives whenever possible. Comparing and contrasting data helped the team to gain a more complete understanding of the issues and provide more confidence in the findings. The evaluation report conforms to USAID guidance on report structure. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 55 The semi-structured KIIs were consolidated for a thematic analysis for each evaluation question. We applied coding categories while reviewing the interview transcripts. The qualitative analysis of interview transcripts was conducted, using a process in which we consolidated multiple responses related to a similar theme that are mentioned by different categories of respondents, and we analyzed them for general findings. In this manner, we were able to determine the common themes. Data collection instruments are presented in Error! Not a valid bookmark self-reference., and the Evaluation Matrix is outlined in Exhibit 12. Exhibit 12. Evaluation Matrix EVALUATION QUESTIONS DATA SOURCES AND DATA COLLECTION METHODS RESEARCH DESIGN 1. To what extent were the programmatic and contextual assumptions identified in the project theory of change sufficient to achieve the purpose of Project 2.2? What are lessons learned for future USAID/BiH planning at project level? Document review of Project/PAD and constituent activities’ planning and implementation documentation and databases (including M&E documentation) and secondary documentation from international organizations/donors and BiH authorities. Key informant interviews with USAID/BiH staff; USAID/BiH implementers; EIA and FSRA stakeholders/beneficiaries; other donors/international organizations; relevant BiH government/public sector institutions; and relevant business associations and NGOs. Mini online survey of a sample of enterprises in BiH. Mixed methods 2. To what extent have the activities implemented under this project achieved their expected results, and to what extent have they been able to address and collectively contribute to the purpose of Project 2.2? How have constituent activities been implemented and what were the main challenges and lessons learned from stakeholders’ perspective for any potential future interventions? Document review of Project/PAD and constituent activities’ implementation documentation and databases (including M&E documentation) and secondary documentation and data from international organizations/donors and BiH authorities. Key informant interviews with USAID/BiH staff; USAID/BiH implementers; EIA and FSRA stakeholders/beneficiaries; other donors/international organizations; relevant BiH government/public sector institutions; and relevant business associations and NGOs. NOTE: Depending on KII scheduling, some KIIs were replaced by focus group discussions (FGD) to ensure higher coverage of informants. Mini online survey of a sample of enterprises in BiH. Mixed methods 3. To what extent have the activities implemented under this project been coordinated and how have internal and external project management practices supported progress toward the purpose of Project 2.2? What are lessons learned for possible future interventions? Document review of Project/PAD and constituent activities’ planning and implementation documentation and databases (including M&E documentation). Key informant interviews with USAID/BiH staff; EIA and FSRA implementers; other donors/international organizations; and relevant BiH government/public sector institutions and business associations and NGOs. Mixed methods USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 56 The evaluation team prepared a map of FSRA and EIA beneficiaries based on FSRA and EIA databases (see Exhibit 13). The map was used to ensure that selected interviewees were geographically represented. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 57 Exhibit 13. Map of EIA and FSRA Beneficiaries USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 58 IV. EVALUATION LIMITATIONS Main limitations of the evaluation include: 1. Response bias. Because majority of the key informants had direct interaction with the activities, they may have overstated the positive effects of the intervention and understated its negative effects. We mitigated this response bias to the extent possible by drawing on multiple sources of information, by guaranteeing the interviewees their confidentiality, and by carefully designing and conducting data collection to request specific examples from the KIIs to describe their responses. We also ensured broad coverage of the stakeholders of both activities in the KIIs and included external stakeholders/non-beneficiaries in the KIIs. 2. Recall bias. As other donor interventions/international organizations have been supporting the government/public institutions in the same or similar areas of work as EIA and FSRA, it was challenging to isolate EIA and FSRA’s contribution from the contribution of other donors. 3. Selection bias. As the number of stakeholders and beneficiaries of EIA and FSRA activities is large, it was not feasible to interview all of them. To mitigate this, we interviewed as many as possible and made sure to include the main stakeholders and beneficiaries of the activities. We also covered all activities components by interviewing at least several key informants for each of those components. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 59 V. BACKGROUND INFORMATION ABOUT CONSTITUENT ACTIVITIES USAID/BiH implemented two activities contributing to the purpose of the project purpose within this CDCS period, as of September 2014: Energy Investment Activity (EIA) and Fiscal Sector Reform Activity (FSRA). The third activity under this Project – USAID Financial Reform Agenda Activity (FINRA) - commenced in October 2017 and is thus not part of this evaluation. Exhibit 14. Timing and Cost of the Project 2.2 Activities Activity Title Start Date End Date Total Cost Energy Investment Activity (EIA) September 2014 September 2019 $5,470,000 Fiscal Sector Reform Activity (FSRA) December 2014 September 2019 $5,000,000 Energy Investment Activity (EIA) is $5.47 million activity with duration from September, 2014 to September, 2019, implemented by Advanced Engineering Associates International. The purpose of EIA, as specified in the award, is to increase investments in the energy sector in BiH and advance the BiH accession process related to energy requirements by working on improving coordination, management, and transparency at all levels of BiH’s regulatory framework, a simplified energy investment environment, and targeted technical assistance to integrate BiH’s energy sector into regional and EU markets. EIA was envisioned to work on transparent and investor-friendly legal and regulatory environment (including shortening development timeframes, modifying conflicting regulations, and improving transparency of the investment process), to meet BiH’s future energy needs, maximize its export potential and make the sector attractive for private investors. The award specifies four expected objectives and results to support the accomplishment of the activity:  Objective 1: Address impediments to increased investment in the energy sector in BiH. The core focus within this objective is to identify and address obstacles in legislation and regulation that impede investment in the energy sector, with the focus on investment in renewable-fueled power plants. To create an investor-friendly environment for building new power plants, regulatory and administrative inefficiencies that deter transparent investments need to be eliminated. Overall expected result for Objective 1 is increased value of new investments in the energy sector.  Objective 2: Address Retail Market Deficiencies in BiH, and contribute to single economic space in BiH. Within this objective, EIA works on improving the functionality of the energy market, as electricity customers in BiH pay different tariffs, depending on the distribution area in which USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 60 they are located. The differences among customer tariff categories are quite large, which represents an economic loss. Overall expected result for Objective 2 is Functional energy retail market in BiH, where customers can choose their suppliers regardless of their location.  Objective 3: Achieve energy savings in BiH, using regulatory incentives. The EU Energy Community Treaty directive requires countries to set up energy efficiency schemes. These schemes require that all energy distributors and all retail energy sale companies achieve annual energy savings. The regulatory commission needs to develop rules, methodologies and measures to implement this directive by including energy efficiency goals in their decisions. Overall expected result for Objective 3 is Saved energy as a result of EE incentives, demand measures and introduced improvements.  Crosscutting- Objective 4: Advance EU accession requirements in the energy sector for BiH. Assisting Ministry of Foreign Trade and Economic Relations (MoFTER) and other relevant institutions to transpose and implement Energy Community Treaty decisions related specifically to the above three components. Overall cross cutting result is Adopted Action Plan to implement the Third Liberalization Package (in relation to the above components). Fiscal Sector Reform Activity (FSRA) is $5.0 million activity with duration from December, 2014 to September, 2019, implemented by FINIT Consulting, a local contractor. As specified in the award, FSRA supports efforts to facilitate gains in fiscal space and public investment, which in turn should result in increase of private domestic and foreign investments in BiH. The focus area of the activity is fiscal discipline, which entails comprehensive set of initiatives tackling coordination between the different levels of the BiH governments, and additional work with governmental institutions such as the tax administrations. The context within which FSRA was designed, as laid out in the award, included weaknesses of BiH’s public finances and fiscal sector being uncovered by the global economic crisis, with weak economic governance, influenced by irreconcilable political interests, instead of financing economic growth￾enhancing activities. The award notes that improving BiH’s public finances remains one of the key priorities identified by the USAID Mission, as well as the IMF and the EU delegations. Weak overall public sector coordination on economic and fiscal policy in underlined as weak and hampering reforms, while large and inefficient public sector with multiple overlapping competences across different government levels is underlined as a considerable risk. Moreover, the need to improve fiscal reporting to enhance the analytical quality of fiscal policy design is highlighted as necessary to address the poor management and structure of public spending. The main four activity tasks and associated expected results specified in award are: 1. Assistance for improved intergovernmental fiscal coordination: USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 61 2. Strengthening of the Treasury Systems 3. Strengthening of the Public Debt Management 4. Increased compliance and improved business environment: 4.1. Reducing tax evasion 4.2. Business environment 4.3. Elimination of nuisance taxes USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 62 EIA LOGFRAME INDICATOR NAME OVERALL ACTIVITY BASELINE Year 1 Year 2 Year 3 LIFE OF ACTIVITY Date Value Target Actual Target Target Number of laws, policies, regulations, or standards to enhance energy sector governance formally proposed, adopted, or implemented as supported by USG assistance 2016, Although the activity started in 2014, the baseline is 2016, as this is revised M&E Plan. 3 3 4 3 3 12 Amount of investment mobilized (in USD) for energy projects as supported by USG assistance 377,000,000 0 0 75,692,967 506,740,113 959,433,081 Number of energy sector stakeholder’s trained with USG assistance 250 150 677 100 50 550 Number of seminars and presentations organized and presented to stakeholders 70 20 20 15 10 115 Number of sets of Simplified Guidelines for Investors developed 0 2 2 1 1 4 Number of legislative acts and administrative procedures for a transparent and investor-friendly energy sector environment developed and submitted to the relevant institutions 0 3 3 3 4 10 Number of proposed interventions developed to address the obstacles for investments 0 5 7 5 5 15 Number of documents subject to analysis and peer review in order to map the investment obstacles in the existing legislation based on case studies 0 2 2 3 4 9 Percentage of consumers knowing that they can switch supplier 0 0 0 40 70 70 Number of Guidelines for amendment of the existing and institution of the missing roles and procedures in the electricity retail market developed and submitted to relevant institutions 0 2 3 1 0 3 Number of proposed interventions developed to address the problems in electricity market functioning 0 5 5 2 0 7 Number of documents subject to analysis and peer review in order to map the obstacles for full retail electricity market functioning and providing recommendations to overcome the obstacles 0 4 4 4 4 12 Energy savings enabled as a result of USG assistance 0 0 0 0 0.6 0.6 Number of adopted Emission Reduction Plans for nine large combustion plants in BiH and number of interventions and documents to address relevant amendments to regulations 0 1 1 4 1 6 Number of regulatory incentives that address energy savings developed and submitted to the relevant institutions 0 4 4 2 0 6 Number of proposed interventions to address energy savings 0 5 5 5 0 10 USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 63 FSRA LOGFRAME INDICATOR NAME OVERALL ACTIVITY BASELINE Year 1 Year 2 Year 3 Year 4 Year 5 LIFE OF ACTIVITY Date Value Target Actual Target Actual Target Actual Target Target 1.-1 Difference in coverage in consolidate execution of general government budget in GFF vs. IMF data 2015 2.37 0 0 0 1.1.-1 Number of adopted recommendations related to fiscal reporting harmonization 2015 0 0 0 0 0 5 0 8 1 14 1.1.1-1 Number of recommendations related to fiscal reporting harmonization that are submitted by FAR 2015 0 0 0 14 14 14 0 14 14 14 1.2-1 Percentage of FBiH municipalities with modern IT treasury information system 2015 24.05 27.84 74.68 63.29 100 100 100 1.2-2 Percentage of RS health sector institutions included in treasury information system in RS 2015 0 26.32 0 100 100 100 1.2-3 Percentage of FBiH cantons included in FBiH budget system 2015 0 100 100 100 100 100 1.2.1-1 Number of institutions newly included in the Treasury Information System (TIS) and number of institutions receiving TIS upgrades 2015 0 0 0 3 60 28 76 136 1.3-1 Progress introduction of IT system for debt management in State and Entity Finance Ministries, on a scale from 1-3 2015 0 1 1 1 2 1 3 3 3 1.3.1.-1 Number of documents related to debt management software processes produced 2015 0 0 2 2 2 0 0 0 2.1-1 Change in revenues from direct tax revenue collected from games of chance 2015 0 0 0 0 20,000,000 0 20,000,000 20,000,000 60,000,000 2.1-2 Total amount of forced collection of company debt 2015 119,716,965 0 0 0 120,914,134 123,308,474 125,702,813 369,925,421 USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 64 INDICATOR NAME OVERALL ACTIVITY BASELINE Year 1 Year 2 Year 3 Year 4 Year 5 LIFE OF ACTIVITY 2.1.1-1 Number of audit cases initiated by project intervention by connecting gaming and audit software 2015 0 0 0 0 27 0 27 27 81 2.1.1-2 Number of companies in the registry that has connected bank accounts 2015 0 0 0 0 30,002 98,897 30,002 30,002 30,002 2.2-2 Amount of cost savings for businesses due to simplified payroll procedures 2015 0 0 0 0 0 0 3,973,498 13,231,839 17,205,337 2.2.-3 Amount of cost savings arising from elimination/reduction of fees 2015 0 0 0 0 0 11,668,666 0 87,085,994 87,085,994 2.2.1-1 Number of minimum required monthly payment slips for payroll in FBiH 2015 9 5 1 1 2.2.1-2 Number of registers of para￾fiscal fees established in BiH 2015 0 0 3 0 3 0 3 3 3 2.2.1-3 Number of para-fiscal fees eliminated/reduced 2015 0 0 0 0 12 1 9 3 24 2.3-1 Amount of cost savings resulting from adoption of E￾solutions by Tax Administrations and Ministries of Finance 2015 0 0 2,336,799 2,401,303 2,411,168 2,550,999 2,460,612 2,481,612 9,690,191 2.3.1-1 Time savings for Tax Administration and Ministries of Finance for providing e-services 2015 0 0 252.21 34.97 281.06 92.54 295.49 295.49 1,124.25 2.3.1-2 Number of E-Documents in E￾Archive 2015 0 0 450 296,1 1,200,000 727 2,100,000 3,000,000 6,750,000 1.1.1-2 Capability to independently use models (microsimulation and econometric models) 2015 0 200 171 170 50.28 140 140 650 USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 65 ANNEX II: MAIN DATA COLLECTION INSTRUMENTS As presented in the Evaluation Design and Methodology section, data for this evaluation will be collected through: 1. USAID/BiH CDCS documentation and PMP documentation related to Project 2.2 (PAD, Project LogFrame and project indicators and PIRS documentations) 2. Secondary documentation relevant to EIA and FSRA, such as documentation from European Commission, Energy Secretariat, World Bank, as well as relevant documentation from relevant BiH government/public institutions (e.g. Ministries) 3. Secondary documentation from relevant international organizations and BiH government/public institutions related to broader project of achievements and gaps in terms of economic aspects of governance relevant to business activity 4. EIA documentation/database review, including award documentation, work plans, quarterly and annual reports, as well as 47 key deliverables identified by EIA 5. FSRA documentation/database review, including award documentation, work plans, quarterly and annual reports, as well as 26 key deliverables identified by EIA 6. Fifty-eight semi-structured key informant interviews (depending on scheduling and interviewees’ availability, FGD may also be conducted) 7. Mini online survey of a sample of enterprises in BiH Overall, around 58 key informant interviews (KIIs) are planned for the eight broad types of stakeholders: 1. USAID/BiH staff involved in project-level planning and design (EDO Director and Deputy Director, Project 2.2 Manager, and Program Office staff) – 2 interviews 2. USAID/BiH CORs of EIA and FSRA - 2 interviews 3. EIA implementers – 2 interviews 4. FSRA implementers – 2 interviews 5. EIA stakeholders/beneficiaries – 20 interviews (out of 118 stakeholders/beneficiaries sent by EIA to the evaluation team, coming from 44 different institutions/organizations) 6. FSRA stakeholders/beneficiaries – 20 interviews (out of 121 stakeholders/beneficiaries sent by 7. FSRA to the evaluation team, coming from 78 different institutions) USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 66 8. Other relevant BiH government/public sector institutions; and relevant business associations and NGOs – 6 interviews 9. Other donors/international organizations – 4 interviews The data from above-listed sources will be triangulated to address the same questions/sub-questions from multiple perspectives whenever possible. Comparing and contrasting data will help us to gain a more complete understanding of the issues and provide more confidence in the findings. The evaluation report will conform to USAID guidance on report structure. The semi-structured KIIs will be consolidated for a thematic analysis for each evaluation question, coding categories that will be applied when reviewing the interview transcripts. The qualitative analysis of interview transcripts will be conducted, using a process in which we consolidate multiple responses related to a similar theme that are mentioned by different categories of respondents, and analyze them for general findings. In this manner, we will be able to determine the common themes. We here present detailed semi-structured interview guides for the three main stakeholder groups with which most interviews will take place – USAID/BiH staff, EIA stakeholders/beneficiaries, and FSRA stakeholders/beneficiaries. Interviews with the remaining stakeholder groups will be based on the guides presented here, but adjusted for the specificities of each stakeholder group’s relation to the project/activity. We also present the mini online survey instrument for a survey of a sample of enterprises in BiH. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 67 ANNEX II.I: DOCUMENTS/DATABASES REVIEWED i. USAID/BiH CDCS 2012-2016 ii. Project 2.2-related documentation iii. EIA Contract (AID-168-C-14-00002) iv. EIA Contract Modification v. EIA Year 1 Annual Report vi. EIA Year II Annual Report vii. EIA Year III Annual Report viii. EIA – Year 1 First Quarterly Report ix. EIA – Year 1 Second Quarterly Report x. EIA – Year 1 Third Quarterly Report xi. EIA – Year 1 Fourth Quarterly Report xii. EIA – Year II First Quarterly Report xiii. EIA – Year II Second Quarterly Report xiv. EIA – Year II Third Quarterly Report xv. EIA – Year II Fourth Quarterly Report xvi. EIA – Year III First Quarterly Report xvii. EIA – Year III Second Quarterly Report xviii. EIA – Year III Third Quarterly Report xix. EIA – Year III Fourth Quarterly Report xx. EIA Revised Monitoring and Evaluation Plan (March 2017) xxi. EIA Monitoring and Evaluation Plan (February 2015) xxii. EIA Year 1 Work Plan USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 68 xxiii. EIA Year II Work Plan xxiv. EIA Year III Work Plan xxv. EIA Year IV Work Plan xxvi. EIA List of beneficiaries per component xxvii. EIA Set of different key deliverables per components xxviii. FSRA Contract (AID-168-C-14-00001) xxix. FSRA Contract Modifications xxx. FSRA Year 1 Annual Report xxxi. FSRA Year II Annual Report xxxii. FSRA Year III Annual Report xxxiii. FSRA – Year 1 First Quarterly Report xxxiv. FSRA – Year 1 Second Quarterly Report xxxv. FSRA – Year 1 Third Quarterly Report xxxvi. FSRA – Year 1 Fourth Quarterly Report xxxvii. FSRA – Year II First Quarterly Report xxxviii. FSRA – Year II Second Quarterly Report xxxix. FSRA – Year II Third Quarterly Report xl. FSRA – Year II Fourth Quarterly Report xli. FSRA – Year III First Quarterly Report xlii. FSRA – Year III Second Quarterly Report xliii. FSRA – Year III Third Quarterly Report xliv. FSRA – Year III Fourth Quarterly Report xlv. FSRA – Year IV First Quarterly Report xlvi. FSRA Monitoring and Evaluation Plan (April 2015) USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 69 xlvii. FSRA Revised Monitoring and Evaluation Plan (November 2016) xlviii. FSRA Year 1 Work Plan xlix. FSRA Year II Work Plan l. FSRA Year III Work Plan li. FSRA Year IV Work Plan lii. FSRA List of beneficiaries per component liii. FSRA Set of different key deliverables per components liv. Memorandum of Cooperation between the Public Administration Reform Coordinator’s Office and the Fiscal Sector Reform Activity Project USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 70 ANNEX II.II: SEMI-STRUCTURED KEY INFORMANT INTERVIEW GUIDE FOR PROJECT-RELATED ISSUES The following sets of questions will be used as guidance for KIIs (and FGDs if taking place). These interview guide sets have been designed for each of the three main groups of KIs – i) USAID staff, ii) EIA and FSRA related to project-level questions, and iii) EIA and FSRA beneficiaries/stakeholders. Interviews with the remaining stakeholder groups will based on the guides presented here, but adjusted for the specificities of each stakeholder group’s relation to the project/activity. These questions serve to ensure that all relevant areas of inquiry are pursued but do not necessarily represent the exact sequence of interviews and discussions. More detailed probes will be used to ensure the correct and full understanding of information gathered from interviewees for each question and sub￾question. Interviewees will be asked to provide examples for all relevant questions. As an introduction, interviewers will give background information to interviewees on definitions, difference, and rationale between USAID project and activities; the main information and expected results of project 2.2, as well as the main information, expected results, and implementation mechanisms employed and main achievements reported so far by the two constituent activities. STRATEGIC PLANNING PROCESS AND COORDINATION AT PROJECT LEVEL (USAID STAFF) 1. Can you please describe the process of USAID/BiH 2012-2016 CDCS formulation and subsequent formulation of PADs? a. How was the project theory of change and associated the programmatic and contextual assumptions identified? b. Who was involved in the process? What were the roles of Technical Offices, Program Offices, Mission management, and USAID/W, and what is your assessment of their capacities? c. To what extent were the BiH government stakeholders in BiH consulted in preparation of CDCS, PADs, and constituent activities and how specifically? Were potential G2G activities considered to be included and why or why not? d. To what extent were other international organizations and donors consulted in preparation of CDCS, PADs, and constituent activities and how specifically? 2. Can you please describe your understanding of differences and synergies between project 2.1 and 2.2, in particular as it pertains to work of the activities under 2.1 with the government stakeholders? 3. How useful do you believe the USAID/BiH 2012-2016 CDCS was in terms of defining expected results and in terms of effectiveness in achieving those results and guiding USAID’s development efforts in BiH? USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 71 a. How effective was it overall and what specific aspects were most useful? What were the obstacles or challenges? b. What are overall lessons learned for USAID/BiH future strategic planning at project level? c. What will be the main challenge in developing the next CDCS in your opinion? 4. How useful do you believe the PADs overall, and specifically 2.1 PAD, were in terms of defining expected results and in terms of effectiveness in achieving those results and guiding USAID’s development efforts in BiH? a. How effective was it overall and what specific aspects were most useful? What were the obstacles or challenges? b. What are overall lessons learned for USAID/BiH future strategic planning at project level? c. What will be the main challenge in developing the next CDCS in your opinion? 5. How useful do you believe the PADs overall, and specifically 2.1 PAD, were in terms of defining expected results and in terms of effectiveness in achieving those results and guiding USAID’s development efforts in BiH? a. How effective was it overall and what specific aspects were most useful? What were the obstacles or challenges? b. What will be the main challenges in developing the next PADs in your opinion? 6. How satisfied are you with the performance of the 2.2 activities? a. To what extent have the 2.2 IPs addressed and accomplished the expected results at the project purpose level? How do you measure and validate performance? How were the project LogFrame associated with PADs developed, how did they evolved over time, and to what extent have they informed decision-making? b. What internal/external constraints exist that affect the performance of 2.2 activities? c. Did the cooperation with the government counterparts and partners influence the achievement of desired results in your opinion and if yes, how? Has USAID/BiH been in direct contact with the relevant government counterparts and how? d. What have you been able to do to enable projects to become better at achieving their development goals? 7. What does USAID do to align objectives and to encourage and enable collaboration among activities that contribute to this project? USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 72 8. What does USAID do to align objectives and to encourage and enable collaboration of this project and its activities with those of other donors and international organizations? 9. What does USAID do to align objectives and to encourage and enable collaboration of this project and its activities with local government stakeholders? STRATEGIC PLANNING PROCESS AND COORDINATION AT PROJECT LEVEL FROM THE PERSPECTIVE OF CONSTITUENT ACTIVITY IMPLEMENTERS (EIA AND FSRA RELATED TO PROJECT-LEVEL QUESTIONS) 1. Please tell us in brief what do you know about USAID/BiH’s CDCS and project-level planning and are you familiar with the project theory of change and LogFrame and associated assumptions? a. Do you think that the project theory of change and the associated programmatic and contextual assumptions are relevant and appropriate? 2. What do you know about USAID/BiH’s coordination of work among USAID/BiH activities, in particular within project 2.2, how has your activity participated, and how frequently? 3. What are values of being an activity of project 2.2 from your perspective? 4. Please describe your interactions with other USAID/BiH activities, in particular within project 2.2. a. How do you get informed about other activities? b. With which activities have you coordinated/cooperated, how frequently, and in what ways? c. With which activity do you coordinate/cooperate the most and in what ways? 5. What are the most important systemic changes or improvements achieved by project 2.2 that contribute directly to the economic growth objectives? 6. Can you please describe how your activity contributes to project purpose 2.2 in terms of expected results and in terms of achievement to date? a. To what extent has your activity addressed and accomplished the expected results at the project purpose level? b. What internal/external constraints exist that affect the performance of your activity and overall 2.2 activities? c. Did the cooperation with the government counterparts and partners influence the achievement of desired results in your opinion and if yes, how? USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 73 7. How do you coordinate your activities with relevant BiH Government stakeholders? 8. How do you coordinate your activities with relevant intervention of other donors and international organizations? IMPLEMENTATION AND ACHIEVEMENT OF THE EXPECTED RESULTS FROM THE PERSPECTIVE OF THE ACTIVITY BENEFICIARIES – EIA 1. How did you start your collaboration with EIA and in what EIA interventions have you participated or collaborated with EIA? a. How did you first hear about EIA? b. Have you discussed this intervention in the time in which it was designed with USAID and if yes, in what way? c. Please describe: what kind of collaboration did you have with EIA, or what kind of assistance did your institution/organization/company receive? Which component(s) do these activities belong to (background information on each to be provided to interviewees by the interviewer)? 2. Overall, how would you assess your collaboration with EIA and EIA’s work and achievements in terms of your expectations and in terms of expected results of EIA (background information on each to be provided to interviewees by the interviewer)? 3. Were there any implementation challenges, and if yes how were they addressed? 4. Would you say that interventions implemented by EIA are relevant in terms of being the main priorities when it comes to energy sector reforms in BiH? a. If yes, why do you think so? And within different interventions implemented by EIA, which are of highest priority in your opinion? b. If no, what kind of other interventions should be prioritized? 5. In your opinion, to what extent has EIA so far addressed their main expected results, what were the key achievements, success factors, challenges, and lessons learned for the following main areas of work (background information on each to be provided to interviewees by the interviewer)?: a. Impediments to investments in BiH energy sector b. Retail market deficiencies c. Achieving energy savings USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 74 6. What are EIA’s activities related to EU accession, including implementation of Energy Community Treaty agreement and Third Liberalization Package, how have these activities been implemented, and how effective and relevant they were in your opinion? 7. In your opinion, to what extent has EIA contributed to integration of BiH’s energy sector into regional and EU energy markets through these interventions? a. Which interventions, components or sub-components are the most important when it comes to increasing the energy sector’s sustainable contribution to economy? b. Were all these interventions necessary to achieve this result? c. Were these interventions sufficient to achieve this result? 8. In your opinion, to what extent has EIA contributed to the energy sector’s sustainable contribution to economy through these interventions? a. Which interventions, components or sub-components are the most important when it comes to increasing the energy sector’s sustainable contribution to economy? b. Were all these interventions necessary to achieve this result? c. Were these interventions sufficient to achieve this result? 9. In your opinion, to what extent has EIA contributed to a broader goal of improved economic aspects of governance relevant to business activity in BiH through these interventions? 10. Overall, what would be your recommendations for potential future interventions in energy sector and more broadly in economic aspects of governance relevant to business activity in BiH? IMPLEMENTATION AND ACHIEVEMENT OF THE EXPECTED RESULTS FROM THE PERSPECTIVE OF THE ACTIVITY BENEFICIARIES – FSRA 1. How did you start your collaboration with FSRA and in what FSRA interventions have you participated or collaborated with FSRA? a. How did you first hear about FSRA? b. Have you discussed this intervention in the time in which it was designed with USAID and if yes, in what way? c. Please describe: what kind of collaboration did you have with FSRA, or what kind of assistance did your institution/organization/company receive? Which component(s) do these activities belong to (background information on each to be provided to interviewees by the interviewer)? USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 75 2. Overall, how would you assess your collaboration with FSRA and FSRA’s work and achievements in terms of your expectations and in terms of expected results of FSRA (background information on each to be provided to interviewees by the interviewer)? 3. Were there any implementation challenges, and if yes how were they addressed? 4. Would you say that interventions implemented by FSRA are relevant in terms of being the main priorities when it comes to fiscal sector reforms in BiH? a. If yes, why do you think so? And within different interventions implemented by FSRA, which are of highest priority in your opinion? b. If no, what kind of other interventions should be prioritized? 5. In your opinion, to what extent has FSRA so far addressed their main expected results, what were the key achievements, success factors, challenges, and lessons learned for the following main areas of work (background information on each to be provided to interviewees by the interviewer): a. Fiscal coordination between different government levels in BiH b. Improvements in public finance management information systems, including treasury system in FBiH municipalities and in RS health sector, BMIS in FBiH cantons, and public debt management at all levels c. Increased compliance and improved business environment: i. Reducing tax evasion ii. Creating a business-friendly environment in BiH iii. Elimination of nuisance taxes 6. In your opinion, to what extent has FSRA contributed to improved overall fiscal coordination in BiH through improvement of fiscal reporting and its harmonization across government levels and through Finance Ministries’ modeling capacities through these interventions? a. Which interventions are the most important in this aspect? b. Were all these interventions necessary to achieve this result? c. Were these interventions sufficient to achieve this result? 7. In your opinion, to what extent has FSRA contributed to increased coverage of the government /public sector units in treasury, budget planning, and debt management information systems through these interventions? USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 76 a. Which interventions are the most important in this aspect? b. Were all these interventions necessary to achieve this result? c. Were these interventions sufficient to achieve this result? 8. In your opinion, to what extent has FSRA contributed to increased tax compliance (through work on revenues collected from games of chance and through connecting gaming and audit software and connecting company bank accounts), improved business environment (through planned cost savings due to simplified payroll procedures and elimination/reduction of parafiscal fees), and introduced and enhanced e-government and e￾communication into Ministry of Finance and Tax Administrations through these interventions? a. Which interventions are the most important in this aspect? b. Were all these interventions necessary to achieve this result? c. Were these interventions sufficient to achieve this result? 9. In your opinion, to what extent has FSRA contributed to a broader goal of improved economic aspects of governance relevant to business activity in BiH through these interventions? 10. Overall, what would be your recommendations for potential future interventions in fiscal sector and more broadly in economic aspects of governance relevant to business activity in BiH? USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 77 ANNEX II.III: MINI ONLINE SURVEY QUESTIONNAIRE FOR ENTERPRISES 1. How important for BiH in your opinion is improvement of economic governance relevant to business activity for achieving a competitive, market-oriented economy providing better economic opportunities for its citizens? a. Strongly agree b. Agree c. Somewhat agree d. Neither agree nor disagree e. Somewhat disagree f. Disagree g. Strongly disagree Please explain briefly why you selected the chosen answer______________________ 2. In your opinion, what are the three priority needs to improve economic governance relevant to business activity in BiH (please rank in priority and be specific as possible): a. PRIORITY 1: _____________ b. PRIORITY 2: _____________ c. PRIORITY 3: _____________ 3. To what extent do you believe priorities should be the following, on a scale from 1 to 5, where 1 is not a priority at all and 5 is top priority? a. Fiscal sector reforms b. Adoption of employment, social, and disability policies related to economic growth c. Financial sector providing stability and serving private sector needs d. Energy reform policies e. Anticorruption – improved control over public resources/funds f. “Green development” g. E-Governance Reform USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 78 h. Other – please list____________________ 4. To what extent do you believe there have been improvements in fiscal sector last three years in the following, on a scale from 1 to 5, where 1 is not at all and 5 is extensive improvements? a. Overall increased tax compliance and improved business environment b. Reducing tax evasion c. Creating a business-friendly environment in BiH d. Elimination of nuisance taxes 5. To what extent do you believe there have been improvements in energy sector in last three years in the following, on a scale from 1 to 5, where 1 is not at all and 5 is extensive improvements? a. Energy sector´s sustainable contribution to economy increased b. Addressing impediments to investments in BiH energy sector c. Addressing retail market deficiencies d. Achieving energy savings 6. On a scale from 1 to 7, where 1 is ‘extremely poor' and 7 is ‘excellent’, how would you rate economic conditions in this country today? 7. Right now, do you think that economic conditions in this country are getting better or getting worse? a. Better b. Worse 8. In which of the following sectors should BiH authorities invest its resources as a priority? NOTE FIVE ANSWERS, RANKED. a. Energy sector b. Transport infrastructure c. Social infrastructure (such as schools and hospitals) d. Tourism USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 79 e. Agriculture f. Science and technology g. Industrial development h. Small and medium enterprise development i. Other, please list______________________________________________ 9. Are you familiar with USAID/BiH’s interventions assisting private sector or economic governance relate to business activity in BiH? a. Yes b. No 10. (If YES on previous questions) Which USAID/BiH’s interventions are you familiar with? _________________________________________________________ 11. Have you been a beneficiary/participant in any of the listed USAID/BiH interventions – if yes, please list each and for each explain in which way you participated and how you assess its usefulness? ________________________________________________________ 12. Overall, what would be your recommendations for potential future interventions in economic aspects of governance relevant to business activity in BiH? ____________________________________________________________ USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 80 ANNEX III: KEY INFORMANT INTERVIEWEES # Project 2.2 Activity Date Location 1 EIA March 27 Sarajevo 2 EIA March 30 Sarajevo 3 FSRA March 30 Sarajevo 4 EIA & FSRA April 2 Sarajevo 5 FSRA April 2 Sarajevo 6 FSRA April 2 Sarajevo 7 EIA April 3 Sarajevo 8 EIA April 3 Sarajevo 9 EIA April 3 Sarajevo 10 FSRA April 3 Sarajevo 11 FSRA April 3 Sarajevo 12 FSRA April 3 Sarajevo 13 EIA April 4 Sarajevo 14 FSRA April 4 Sarajevo 15 FSRA April 5 Sarajevo 16 FSRA April 5 Sarajevo 18 FSRA April 5 Sarajevo 19 EIA April 6 Sarajevo 19 FSRA April 9 Tuzla 20 FSRA April 9 Gradacac 21 FSRA April 9 Odzak 22 EIA April 10 Banja Luka 23 EIA April 10 Banja Luka 24 EIA April 10 Banja Luka 25 FSRA April 10 Banja Luka 26 FSRA April 10 Banja Luka 27 FSRA April 10 Banja Luka 28 FSRA April 10 Banja Luka 29 EIA & FSRA April 11 Sarajevo 30 EIA & FSRA April 11 Sarajevo 31 EIA April 11 Sarajevo 32 EIA & FSRA April 11 Sarajevo 33 FSRA April 11 Sarajevo 34 EIA April 16 Mostar 35 EIA April 16 Mostar 36 EIA April 16 Mostar 37 FSRA April 16 Konjic 38 FSRA April 16 Jablanica 39 FSRA April 16 Ljubuski 40 EIA April 17 Trebinje 41 EIA April 17 Trebinje 42 EIA April 17 Trebinje 43 FSRA April 17 Tomislavgrad 44 FSRA April 17 Livno 45 FSRA & EIA April 18 Sarajevo 46 EIA April 18 Sarajevo 47 EIA April 18 Sarajevo 48 EIA & FSRA April 19 Sarajevo 49 EIA & FSRA April 19 Sarajevo 50 EIA & FSRA April 19 Sarajevo 51 EIA April 19 Sarajevo 52 EIA April 20 Sarajevo USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 81 # Project 2.2 Activity Date Location 53 EIA April 23 Tuzla 54 EIA April 23 Tuzla 55 EIA & FSRA April 24 Sarajevo 56 EIA April 24 Sarajevo 57 FSRA April 25 Sarajevo 58 EIA April 25 Neum 59 EIA April 25 Neum 60 EIA April 25 Neum 61 EIA April 25 Neum 62 EIA April 25 Neum 63 EIA April 25 Neum USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 82 ANNEX IV: THE EVALUATION TEAM RESPONSES TO COMMENTS FROM THE IMPLEMENTING PARTNER August 8, 2018 To: Elma Bukvic Jusic Development Assistance Specialist / MEASURE-BiH COR USAID/BiH Subject: THE EVALUATION TEAM’S RESPONSES TO COMMENTS FROM THE FSRA ACTIVITY IMPLEMENTING PARTNER Dear Ms. Bukvic-Jusic, Below, we have provided the evaluation team’s responses to comments received from the FSRA Activity implementing partner on the WOPE evaluation report. RESPONSES TO THE IMPLEMENTING PARTNER’S COMMENTS IP’s Comment #1: Finding #34: The issues highlighted here prompted FSRA to discuss amending contractual activities with USAID. In November 2015 FAR submitted proposed changes which were agreed by USAID during 2016. For example, in Section C9, A1, the contractual language was amended from “draft amendments to the Law on Fiscal Council and relevant corresponding entity level legislation…” to “draft amendments to relevant entity and Brcko District legislation and rule books…”. The results part was also amended to focus on working on the entity /district level. We would suggest that you confirm that the evaluator’s work has been based on the correct version of contractual activities and not the original text. We raised this at our “kick off” meeting with the evaluation team on 30 March. We also noted at this meeting that our approach to this is specifically low key and where possible under the radar. The t-1 development is a significant pre-requisite for further work. While work had not started during the time of the evaluation field work, we are now focusing on a draft of the Fiscal Council Working Group’s rule book which, if adopted, would realize many results without state level legislative USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 83 change. It is correct to note that achieving fiscal harmonization is not easy, however we at FSRA consider our approach is unique; all prior interventions were perhaps more directly aimed at the Fiscal Council and yielded no notable results. The focus here has been at the operative and working level. The Evaluation Team Response #1: Regarding finding #34, we are aware that the SoW changed and was expanded to working with entities’ legislation and rulebooks (point two in the contract modification states “…Explore and evaluate the feasibility of amending the BiH Law on the Fiscal Council to reinforce this process and evaluate other essential changes and the feasibility of the proposed amendments...) indicating that the objective related to improvements of reporting at FC remained. This has been noted in the expanded results section since under result 2, where it states that “…Evaluated the possibility of amending the BiH level Law on Fiscal Council and relevant rule books to secure ineffective regulations/procedures of the FC are either amended or eliminated, which will result in more efficient work of the FC. Drafted amendments to Entity legislation and rule books for to improve and harmonize fiscal reporting…” Also, please note that this was a performance evaluation. This means that the evaluation team evaluated stated and achieved results (in submitted FAR documents). Under this result, the submitted FAR documents and the FAR M&E plan report on the three measurements defined in the PIRS (Difference in coverage in consolidate execution of general government budget in GFF vs. IMF data, in percentage, Number of adopted recommendations related to fiscal reporting harmonization, Number of recommendations related to fiscal reporting harmonization that are submitted by FAR). We could only confirm that the result under “t-1” has been accomplished. IP’s Comment #2: Finding #36: It is likely that the field work was being conducted while training was in progress. Microsimulation training has now been completed and testing of staff reveals they are now capable of using this independently. The model does include SCC (Social Security Contributions). Macroeconomic modelling has since been delivered to the FBiH - again a timing of fieldwork issue. It should be noted that these activities are also conducted in the Brcko District with very good results. The Evaluation Team Response #2: USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 84 Thank you for this input. Please note that the scope of the WOPE evaluation included all results, activities, and beneficiaries through the end of April 2018 (the last interview was conducted on April 27, 2018) However, finding #36 states that these interventions are work in progress. The evaluation team amended finding #36 by adding information on the activities conducted in Brcko District. IP’s Comment #3: Finding #38: Per contract mod dated 1/30/2018, this is reflected in the SoW. The activities part was amended to include FBiH Cantons and the results part amended to “Analytical capacity of the FC, FMOF (including possibility of cantons’ MOFs…..” The Evaluation Team Response #3: The evaluation team did not have access to that specific contract modification. Upon receiving and reviewing the contract modification, the evaluation team addressed IP’s comment by deleting finding #38. IP’s Comment #4: Finding #39: Usage has considerably expanded during 2018. We have conducted outreach visits to cantons that required additional assistance to utilize BMIS. As of 30 June, 5 cantons were fully utilizing the system with three getting starting under budget process “Instruction 2”. The two remaining cantons requiring further assistance (Western Herzegovina and Cantons 10) were visited by FSRA staff including the COP. Good engagement has been observed since then. The Evaluation Team Response #4: Thank you for this input. Please note that the scope of the WOPE evaluation included all results, activities, and beneficiaries through the end of April 2018 (the last interview was conducted on April 27, 2018) USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 85 IP’s Comment #5: Finding #40: FSRA briefed Cantonal PMs about this during 2016 at a FBiH Coordination Board meeting chaired by PM Novalic. The Evaluation Team Response #5: Thank you for this input. Please note that documentation from the meeting was not provided and it was not mentioned during data collection period. IP’s Comment #6: Finding #41: Follow up training has and will take place during 2018. A new model for salary calculation/projection is currently being rolled out. The Evaluation Team Response #6: Thank you for this input. The evaluation team has been informed that the new model for salary calculation/projection is being planned. Please note that the scope of the WOPE evaluation included all results, activities, and beneficiaries through the end of April 2018 (the last interview was conducted on April 27, 2018). IP’s Comment #7: Finding #42: Tendering process for this were started in summer 2016 and awarded early in the fall 2016. PARCO had been expected to finance TS for those municipalities which previously were understood not to have any former TS installation at all. PARCO failed to deal with this, so any remaining municipalities that are in need of TS will be dealt with by FSRA. The Evaluation Team Response #7: USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 86 Thank you for the useful comment and the observation. The evaluation team expanded the finding with a clearer explanation and added: “PARCO were unable to implement this, so any remaining local communities that are in need of TS and have not yet implemented any aspect of the treasury system will be dealt with by FSRA”. IP’s Comment #8: Finding #44: Risks were identified in several quarterly reports and annual reports. Similar to BMIS above, we continue to conduct outreach with municipalities to identify practical problems which delay full implementation and in June held a successful event in Bihac where representatives from 8 out of a total of 9 municipalities attended. The Evaluation Team Response #8: Political risk was noted in FAR quarterly/annual reports as political/resource availability risk and it states: ..”There could be obstructions or lack of cooperation from some counterparts as a new treasury system will shed transparency to the municipalities’ budgets and spending so some municipalities may not be eager to implement it. This is more likely with regard to few remaining municipalities with which we have not yet concluded MOUs and action plans.” The evaluation team did not find a lack of cooperation from local communities as an obstacle or potential risk under this activity. The evaluation team identified a lack of local political will to fully implement TS as a political risk. Under this finding, the evaluation team identified a lack of human resources and understaffed municipalities as an additional risk, which is not defined in FAR documents under this section. Please note that the scope of the WOPE evaluation included all results, activities, and beneficiaries through the end of April 2018 (the last interview was conducted on April 27, 2018). IP’s Comment #9: Finding #45: See previous comment. The Evaluation Team Response #9: USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 87 Please note that the scope of the WOPE evaluation included all results, activities, and beneficiaries through the end of April 2018 (the last interview was conducted on April 27, 2018). IP’s Comment #10: Finding #47: This is an internal RS government communication issue. The RS government considered the MOU carefully and specifically did not want to include fiscal harmonization in it (which is included in the FBiH and Brcko MOUs). So maybe a memo failed to be sent but it appears strange that there was no awareness. In any case, this is three years ago. The Evaluation Team Response #10: Thank you for your comment and explanation. The evaluation team already included your comment under this finding. (…Obstacles to communication between the two ministries (the Ministry of Health and Social Protection and the Ministry of Finance of RS) were identified that might cause delays and implementation issues. This information was confirmed by both beneficiaries…) IP’s Comment #11: Finding #48: The earlier WB intervention referred to here was discontinued by the recently arrived WB Country Director. The WB is now designing a holistic approach for RS health sector reform. FSRA met with the WB health expert in July and had a conference call with him in April concerning how TS should and could be included within this holistic approach. The Evaluation Team Response #11: Thank you for the clarification and a useful comment. Please note that the scope of the WOPE evaluation included all results, activities, and beneficiaries through the end of April 2018 (the last interview was conducted on April 27, 2018). USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 88 IP’s Comment #12: Finding #49: The RS health care sector is the only major area of public expenditure in the RS that is outside the RS TS system. TS is part of the fight against corruption as it removes a great deal of “discretion” when payments are made. While recognizing that there are many problems in the RS health sector, mixed and non-coherent noises from the now defunct WB health sector intervention could be seen to have provided useful cover and alternate reasons for the delay in further TS rollout. The Evaluation Team Response #12: Thank you for your comment and information. IP’s Comment #13: Finding #51: Page 19 of the FSRA contract notes that at the time, the preferred debt software option, UNCTAD’s DMFAS would be available at no cost to BiH. When the contract was awarded and FSRA was able to commence activities (3 month delay due to an award appeal process), the change in BiH’s status to a “middle-high income country” as determined by the WB meant that this software was no longer free and would cost just under US$400,000. This was not anticipated for within the ceiling of US$4.97 million. FSRA reached out to PARCO, its management board and other donors to try to overcome this. The Evaluation Team Response #13: Thank you for your comment and additional explanation. The evaluation team was informed about this issue and hence included it in the finding (“KIs also stated that the implementer attempted to tackle this issue by reaching to other donors to try to overcome this issue”). IP’s Comment #14: Finding #56: Agreed. The Director of the FBiH Financial Information Agency (FIA) has offered assistance to Brcko re software and other practical help. USAID has also indicated approval that the single register software developed for FIA can be donated to Brcko to help resolve this issue. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 89 The Evaluation Team Response #14: Thank you for the update and the comment. This comment has already been included in the finding. IP’s Comment #15: Finding #57: Correct, but the TA can apply to the courts to block the account. FIA’s Director has been working with the courts to make them aware of this to allow as speedy as possible action when an application is made. The Evaluation Team Response #15: Thank you for the explanation. The evaluation team amended finding #57 by adding information that Tax Administration can apply to the courts to block the account. IP’s Comment #16: Finding #58: There is a common error of terminology here. There are registers of fees and charges that include those fees and charges that are “parafiscal” in nature (it is not the right place to go into detailed definitions, however a short definition is a fee or charge is something that you receive a service for (e.g. passport fee, charge to exploit natural resources etc. while a parafiscal fee is called a fee but it is in fact a general taxation for no specific service but called a fee in order to “justify” it – e.g. the FBiH general water fee which in fact is a 0.5% tax on all net employment costs in the FBiH. The Evaluation Team Response #16: Related to this issue, please note the following: the Reform Agenda signed by the Council of Ministers for the period 2015-2018 and undersigned by all governments in both entities states that all levels of USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 90 government will create and make publicly available registers of parafiscal fees and charges 13(original text: …svi nivoi vlasti će sačiniti (i objaviti) sveobuhvatan popis parafiskalnih nameta u cilju osiguravanja njihove transparentnosti i smanjenja u skladu s podjelom nadležnosti, Vijeće Ministara, 2015, p.3-4). Unfortunately, the Reform Agenda (same term was used in SoW) related to the category of non-tax revenues as parafiscal fees and charges which caused several methodological errors. In theory and in practice of neighboring countries (Croatia, Serbia), created registers are rightfully called the Register of non-tax revenues (including fees, charges, parafiscal fees and charges and other non-tax revenues). The evaluation team also acknowledged the explanation provided in the email attached to this message and therefore is uncertain about the source of definitions provided either in the comment or in the text of the email. Regarding correct definition of fees, charges and especially parafiscal fees and charges (dažbine) please refer to for example, Lovčević, 1961, Raičević, 2005 or Popović, 2010. Parafiscal fees and charges have a few characteristics (see Lovčević, 1961) where the most important one is “included counter-service” provided to the payer. The payer shares some social or economic interest with the institution to which parafiscal fee or a charge is paid to (for example membership paid to Chamber of Commerce). In that sense, parafiscal fees and charges in theory and practice are closer to contributions than they are to taxes. In theory and in BiH practice at all levels of government, the most concerning term are charges due to their number and unclear classification. IP’s Comment #17: Finding #59: FSRA is in regular contact with the IMF. While the IMF made this one of their requirements, the FSRA team in Banja Luka did bring their attention to this. The Evaluation Team Response #17: Thank you for your comment. This comment is addressed in finding # 17. 13 Please note that Anglo-Saxon law does not recognize continental term for overall taxes, fees, charges and contributions unlike continental law, see Popović, 2010. Therefore, we will use the term 'fees and charges' which should correspond to our term 'dažbina' or 'namet'. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 91 IP’s Comment #18: Finding #60: In the RS, FBiH and Brcko, registers of fees and charges are completed (and always subject to updating and review as they are living documents). In the RS, the overall reform process is more advanced. The Law on the RS Tax System is in force and since June 2018, it is not legal to impose a fee unless it is included in the register of fees. In Brcko, a law similar to that in the RS is close to adoption. In the FBiH, a law has been drafted and FSRA is currently, at the request and support of the FBiH Minister of Finance, conducting a consultation process with Cantons on this. The significantly more complicated constitutional architecture of the FBiH means that this process takes longer. The Evaluation Team Response #18: Again, thank you for the useful explanation. However, we are unclear on the relationship between this comment and the finding, which relates to the fact that in RS the gathering of data for the design of the registry began before the project started. IP’s Comment #19: Finding #61: It is possible to search by industry type, municipality, canton etc. already on the FBiH register. This will be made web based later this year. Similar functionality is under discussion for the RS and BD also. The Evaluation Team Response #19: The constraint is noted. Please note that the scope of the WOPE evaluation included all results, activities, and beneficiaries through the end of April 2018 (the last interview was conducted on April 27, 2018). IP’s Comment #20: Conclusion #15: See comments above about contract mod and other FSRA work in this area. The Evaluation Team Response #20: Please see comments in the evaluation team response #1. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 92 IP’s Comment #21: Conclusion #16: Is it the intention to say that analytical capacity of the RS MF is a success, but so far, FBiH is not as successful? Please also note comment in Finding 36 above. The Evaluation Team Response #21: Please refer to evaluation team response #2. IP’s Comment #22: Conclusion #20: The FBiH MOU was not signed until July 2015 (delays in forming FBiH government after 2014 elections led to this) and given the wide range of activities, not everything could commence at the same time. Tendering processes were underway in 2016 and TS can only effectively start at the commencement of the fiscal year. As noted above, significant outreach has been conducted during 2018 and outstanding TS is due to PARCO’s inability to mobilize. Noted re the website and FSRA will investigate this. The Evaluation Team Response #22: Thank you for the update and explanation. Your comment was already included in the conclusion. IP’s Comment #23: Conclusion #21: Noted – but as indicated above, other factors apply. TS can play a major part in fighting corruption and this is not universally welcomed. The Evaluation Team Response #23: Thank you for your comment. This information was already included in both - finding and conclusion. IP’s Comment #24: Conclusion #25: Note comments above and clarification of terminology (fees v parafiscal fees). USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 93 The Evaluation Team Response #24: Please refer to explanation given under the evaluation team response #16. IP’s Comment #25: Conclusion #25: Again, as noted above, FSRA coordinates with the IMF. The Evaluation Team Response #25: Please see comments provided in response #17. IP’s Comment #26: Recommendation #12: Note comment re contract mod above. The intervention exceeds the originally planned activity but was amended with USAID agreement. The Evaluation Team Response #26: The evaluation team did not have access to that specific contract modification. Upon receiving and reviewing the contract modification, the evaluation team addressed IP’s comment by deleting that specific sentence from recommendation #12. IP’s Comment #27: Recommendation #13: Note comment above about further ongoing outreach. The Evaluation Team Response #27: Not clear what is IP’s comment #27. IP’s Comment #28: USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 94 Recommendation #16: FSRA did provide pro-forma rulebooks. The Evaluation Team Response #28: In the evaluation analysis conducted, the evaluation team were unable to confirm this statement. IP’s Comment #29: Recommendation #21: Noted. By the end of FSRA, it is possible that the FBiH will have adopted a framework regulatory law similar to that of the RS, the transitional period required to reach a situation whereby all fees and charges are imposed by law only (and not random and non-transparently decision based) and all are included within the register of fees will take some time. The Evaluation Team Response #29: Thank you for the update and explanation. Your comment was already included in the recommendation. USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 95 MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE-BiH) Fra Anđela Zvizdovića 1 UNITIC Tower B, Floor 21 71000 Sarajevo Bosnia and Herzegovina Phone: + (387) 033 941 676 contacts@measurebih.com www.measurbeih.com