This document is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government. MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE II) PERFORMANCE EVALUATION OF THE USAID FINANCIAL REFORM AGENDA ACTIVITY (FINRA) IN BOSNIA AND HERZEGOVINA September 2020 MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE II) PERFORMANCE EVALUATION OF THE USAID FINANCIAL REFORM AGENDA ACTIVITY (FINRA) IN BOSNIA AND HERZEGOVINA Final Report September 2020 Prepared under the USAID Bosnia and Herzegovina Monitoring and Evaluation Support Activity (MEASURE II), Contract Number AID-167-I-17-00004, Task Order Number 72016819F00001 Submitted to: USAID/Bosnia and Herzegovina, September 2020 Contractor: IMPAQ International, LLC CONTENTS ACKNOWLEDGEMENTS I LIST OF EXHIBITS II ACRONYMS III EXECUTIVE SUMMARY VI PURPOSE AND EVALUATION QUESTIONS VI EVALUATION METHODS AND LIMITATIONS VI BACKGROUND VI EVALUATION FINDINGS AND CONCLUSIONS VII EVALUATION QUESTION 1 VII EVALUATION QUESTION 2 VIII EVALUATION QUESTION 2A IX EVALUATION QUESTION 3 IX RECOMMENDATIONS X BACKGROUND INFORMATION ON THE ACTIVITY 1 COMPONENT 1: BANK SUPERVISION AND INSURANCE 1 COMPONENT 2: CENTRAL BANKING 2 COMPONENT 3: AUDIT OF PUBLIC FINANCIAL RESOURCES AND QUALITY OF GOVERNMENT SPENDING 2 FINRA GRANT FUND 2 ACTIVITY MONITORING 2 EVALUATION PURPOSE, QUESTIONS, METHODS, AND LIMITATIONS 4 EVALUATION PURPOSE 4 EVALUATION QUESTIONS 4 METHODOLOGY 5 LIMITATIONS 6 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 7 EVALUATION QUESTION 1 7 MEL PRACTICES 7 MEL REPORTING 8 CONCLUSIONS 11 EVALUATION QUESTION 2 12 ACTIVITY DESIGN 12 RELATIONSHIP WITH THE PARTNER GOVERNMENT ENTITIES 13 DONOR COORDINATION 14 HUMAN RESOURCES 15 GRANT FUND IMPLEMENTATION 15 IMPLEMENTATION CHALLENGES 18 ACTIVITY VALUE-ADDED 20 CONCLUSIONS 21 EVALUATION QUESTION 2A 22 CONCLUSIONS 23 EVALUATION QUESTION 3 24 CONCLUSIONS 26 RECOMMENDATIONS 27 MEL PRACTICES 27 ACTIVITY DESIGN 28 ACTIVITY IMPLEMENTATION 28 FUTURE ASSISTANCE TO THE BIH FINANCIAL SECTOR 29 ANNEXES 30 ANNEX I: STATEMENT OF WORK FOR PERFORMANCE EVALUATION OF THE USAID FINANCIAL REFORM AGENDA ACTIVITY (FINRA) IN BOSNIA AND HERZEGOVINA 30 ANNEX II: LIST OF REVIEWED DOCUMENTATION 40 ANNEX III: LIST OF KEY INFORMANT INTERVIEWS 42 ANNEX IV: KEY INFORMANT INTERVIEW GUIDES 44 INTERVIEW GUIDE FOR USAID/BIH 44 INTERVIEW GUIDE FOR THE IMPLEMENTING PARTNER 46 INTERVIEW GUIDE FOR BENEFICIARY GOVERNMENT INSTITUTIONS 49 INTERVIEW GUIDE FOR INTERNATIONAL ORGANIZATIONS 51 INTERVIEW GUIDE FOR FINRA CONSULTANTS 53 INTERVIEW GUIDE FOR FINRA NON-BENEFICIARIES 55 ANNEX V: SURVEY QUESTIONNAIRES 56 ONLINE SURVEY QUESTIONNAIRE FOR FINRA CONSULTANTS 56 RESULTS OF ONLINE SURVEY OF FINRA CONSULTANTS 59 ONLINE SURVEY QUESTIONNAIRE FOR CAPACITY BUILDING PARTICIPANTS 64 RESULTS OF ONLINE SURVEY OF CAPACITY BUILDING PARTICIPANTS 67 ANNEX VI: EVALUATION MATRIX 71 ANNEX VII: LIST OF RELEVANT KEY FINANCIAL STABILITY-RELATED DOCUMENTS 72 ANNEX VIII: RECOMMENDATIONS FROM DQA SESSION WITH FINRA 73 ANNEX IX: LIST OF COMPLIANCE REQUIREMENTS RELEVANT FOR FINRA 74 ANNEX X: RECOMMENDATIONS FROM IMF FSAP/FSSA 2015 76 ANNEX XI: FINANCIAL SECTOR PRIORITIES IDENTIFIED THROUGH KIIS AND ONLINE SURVEYS 78 ANNEX XII: DISCLOSURE OF CONFLICT OF INTEREST FOR EVALUATION TEAM MEMBERS 80 I | FINRA PERFORMANCE EVALUATION USAID.GOV ACKNOWLEDGEMENTS Under the USAID/Bosnia and Herzegovina (BiH) Monitoring and Evaluation Support Activity (MEASURE II), USAID/BiH commissioned IMPAQ International to conduct a performance evaluation of the USAID/BiH’s Financial Reform Agenda Activity (FINRA). FINRA, implemented by the Financial Markets International, Inc. (FMI), is a $5.99 million Activity which extends over a period of five year, September 2017 through September 2022. The evaluation team included Damir Odak, MEASURE II’s Subject Matter Expert (Evaluation Team Co-Lead), Anela Kadić Abaz, MEASURE II’s Deputy Chief of Party (DCoP) and Evaluation Expert (Evaluation Team Co-Lead), Jasmina Mangafić, PhD, MEASURE II’s Research Fellow with local financial sector expertise (Evaluation Team Member), Haris Mešinović, MEASURE II’s Senior Research Analyst (Evaluation Team Member), Amela Kurta, MEASURE II’s Analyst (Evaluation Team Member) and Amer Čekić, MEASURE II’s Analyst (Evaluation Team Member). The authors wish to thank all participants who contributed to the process of developing this evaluation report. At USAID/BiH, Elma Bukvic Jusic, MEASURE II Contracting Officer’s Representative (COR), was instrumental in designing the evaluation Scope of Work (SoW). The authors also wish to thank Amy Kracker Selzer, MEASURE II Project Director and Managing Director at IMPAQ International, Tanya Rochelle Hurst, Research Associate at IMPAQ International, Edis Brkic, Chief of Party (COP) at MEASURE II, Salminka Vizin, Senior Research Analyst and Component 2 Lead at MEASURE II, and Rebecca Herrington and Chelsie Kuhn, collaboration, learning and adapting (CLA) experts at MEASURE II sub-contractor Headlight Consulting Services, LLP, for providing technical guidance and advice throughout the process of preparing the report. We also thank the FINRA management team for their assistance with providing Activity documentation and databases. Finally, we would like to thank the representatives of international organizations and local governmental and non-governmental institutions for their collaboration during the data collection process. Any errors in this report are the sole responsibility of the authors. II | FINRA PERFORMANCE EVALUATION USAID.GOV LIST OF EXHIBITS EXHIBIT 1. ACTIVITY INDICATORS WITH ANNUAL AND LIFE OF ACTIVITY TARGETS............3 EXHIBIT 2. NUMBER OF KIIs and KIs..................................................................................................................5 EXHIBIT 3. ONLINE SURVEYS RESPONSE RATE............................................................................................5 EXHIBIT 4. OVERVIEW OF CRs ADDRESSED BY FINRA.............................................................................9 EXHIBIT 5. ACTIVITY INDICATORS WITH ANNUAL AND LIFE OF ACTIVITY TARGETS AND ACTUALS...................................................................................................................................................................10 EXHIBIT 6. PGE STAFF PERCEPTION ON FAVORABILITY OF FINRA’S ASSISTANCE....................13 EXHIBIT 7. FINRA GRANT FUNDS ALLOCATED TO DATE...................................................................16 EXHIBIT 8. PGEs REPRESENTATIVES’ PERCEPTION ABOUT IMPROVED CAPACITIES OF CBBH, DIA, FBA, AND BARS WITH FINRA’S ASSISTANCE....................................................................................24 III | FINRA PERFORMANCE EVALUATION USAID.GOV ACRONYMS APRMBIH Association of Private Risk Managers of BiH BARS Banking Agency of Republika Srpska BBA BiH Bank Association BiH Bosnia and Herzegovina BPMIS Budget Planning and Management C1 Component 1 C2 Component 2 C3 Component 3 CBBH Central Bank of Bosnia and Herzegovina CBS Coordination for Banking Supervision CDCS Country Development Cooperation Strategy CLA Collaboration, Learning and Adapting CO USAID/BiH’s Contracting Office COM Council of Ministers COP Chief of Party COR Contracting Officer’s Representative CR Compliance Requirement CRD EU Capital Requirements Directive CRR EU Capital Requirements Regulation DCOP Deputy Chief of Party DIA Deposit Insurance Agency DIF Deposit Insurance Fund DO Development Objective EBRD European Bank for Reconstruction and Development EC European Commission EE Evaluation Expert EFF Extended Fund Facility EPA USAID/BiH’s Energy Policy Activity EQs Evaluation Questions EU European Union FBA Federation of Bosnia and Herzegovina Banking Agency FBiH Federation of Bosnia and Herzegovina FINRA Financial Reform Agenda Activity FINREP Financial Reporting FMF FBiH Ministry of Finance IV | FINRA PERFORMANCE EVALUATION USAID.GOV FMI Financial Markets International, Inc. FO Field Office FSAP Financial Sector Assessment Program FSF Financial Stability Fund FSSA Financial Sector Stability Assessment FX Foreign exchange GDP Gross Domestic Product GSC Grant Selection Committee HO Home Office HQLA High Quality Liquid Assets ICT Information and Communication Technology IFIs International Financial Institutions IFRS International Financial Reporting Standards IMF International Monetary Fund IMPAQ IMPAQ International, LLC IP Implementing Partner IR Intermediate Result JACA USAID/BiH’s Justice Against Corruption Activity KI Key informant KII Key informant interview LCR Liquidity Coverage Ratio LoA Life of Activity LoE Level of Effort MEASURE II Monitoring and Evaluation Support Activity MEERS RS Ministry for Enterprise and Entrepreneurship MEL Monitoring, Evaluation and Learning MIB Macroeconomic imbalances MOF Ministry of Finance NFSR Net Stable Funding Ratio NPL Non-Performing Loan OM Office Manager PARE Partnership for Advancing Reforms in the Economy PFM Public Financial Management PGE Partner Government Entity PI Performance Indicators RA Reform Agenda V | FINRA PERFORMANCE EVALUATION USAID.GOV RA Research Analyst RF Research Fellow RFP Requests for Proposals RS Republika Srpska SAA EU Stabilization and Association Agreement SC Sarajevo Canton SCFS Standing Committee for Financial Stability SECO Swiss State Secretariat for Economic Affairs SIB Systemically Important Bank SME Subject Matter Expert SoW Scope of Work SRA Senior Research Analyst SREP Supervisory Review and Evaluation Process TT Tracking Table USAID/BiH United States Agency for International Development BiH Mission WB World Bank WBG World Bank Group VI | FINRA PERFORMANCE EVALUATION USAID.GOV EXECUTIVE SUMMARY PURPOSE AND EVALUATION QUESTIONS IMPAQ International, LLC (IMPAQ) has been contracted by the United States Agency for International Development in Bosnia and Herzegovina (USAID/BiH) under the USAID/BiH’s Monitoring and Evaluation Support Activity (MEASURE II) to conduct a performance evaluation of the USAID/BiH’s Financial Reform Agenda Activity (FINRA). The main purpose of the evaluation was to assess FINRA’s progress towards achievement of its contractual obligations to date, so it can inform the Mission about the degree and factors affecting FINRA’s performance and collect evidence on best approaches to facilitate programming in the financial sector. The evaluation also aims to promote collaboration, cultivate learning, expand the existing technical evidence base, and encourage program design adaptations both by USAID and by other evaluation stakeholders. Knowledge, to be generated by the evaluation, will support USAID/BiH’s evidence-based decision-making and therefore guide the Mission to support BiH on its Journey to Self-Reliance (J2SR). The evaluation addresses three questions pertaining to: (i) the achievement of contractual targets, (ii) stakeholders’ perception about the Activity design and implementation to date, including assistance provided to counter the COVID-19 crisis, and (iii) partner government entities (PGEs) readiness to continue their progress towards compliance with relevant international standards and requirements without further FINRA support. EVALUATION METHODS AND LIMITATIONS The evaluation design included a wide range of stakeholders to inform the findings, conclusions, and recommendations. These stakeholders include: the USAID/BiH; FINRA Implementing Partner (IP) and subcontractor; PGEs and other relevant government institutions; FINRA consultants; international organizations active in the BiH financial sector; and representatives of local organizations and banks that have not received any support from the Activity. The evaluation employed rigorous methods to capture high-quality data and produce credible findings, conclusions, and recommendations. The data was collected from relevant Activity documentation and secondary data sources, key informant interviews (KIIs), and three online surveys. The evaluation was affected by several limitations including different sources of bias – response bias, interview bias, online survey methodology bias, and environment bias (COVID-19). The evaluation team employed different approaches (such as data triangulation) to mitigate these biases and to ensure the development of timely and objective findings, conclusions, and recommendations. BACKGROUND FINRA, implemented by the Financial Markets International, Inc. (FMI), is a $5.99 million Activity that started on September 29, 2017. The Activity was envisaged as a three-year intervention with a possibility of a two-year extension. The Activity extension period was approved by a contract modification signed on September 13, 2019. The general purpose of FINRA is to enhance the stability of BiH financial sector. To achieve its purpose, the Activity is required to accomplish the following results defined by its contract: VII | FINRA PERFORMANCE EVALUATION USAID.GOV  Result One: BiH advances towards compliance with the European Union (EU) accession requirements, as set out in the Reform Agenda (RA), the EU Progress Report and the International Monetary Fund (IMF) program in the areas of strengthening financial stability, public audit and quality of government spending.  Result Two: Policies, laws and regulations from the IMF Extended Fund Facility (EFF), Financial Sector Assessment Program (FSAP), the EU RA, and those required by the EU Stabilization and Association Agreement (SAA) process are drafted for adoption and implementation in the financial, audit, and quality of government spending areas. At the outset, FINRA’s design incorporated three specific areas of intervention, (i) bank supervision and insurance, (ii) central banking, and (iii) audit of public financial resources and quality of government spending, which was envisaged as optional contingent on availability of budget resources and need for assistance. To achieve the desired objectives, the Activity is organized in three components. Component 1 (C1) which relates to bank safety and soundness and management of problem banks, includes the provision of assistance to the BiH Deposit Insurance Agency (DIA), FBiH Banking Agency (FBA), and the RS Banking Agency (BARS). Under Component 2 (C2), FINRA provides demand-driven assistance to the Central Bank of BiH (CBBH) to help align its work with the EU requirements and best practices long with the relevant IMF requirements. Component 3 (C3), an optional component, focuses on the provision of assistance to the BiH government institutions to meet the EU RA policy benchmarks in the area of public financial management. EVALUATION FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1: WHAT PROGRESS HAS BEEN ACCOMPLISHED IN REACHING CONTRACT TARGETS? FINRA monitors and reports on six indicators to measure progress towards achieving defined results. Two goal-level indicators measure its contributions to USAID/BiH’s Economic Growth Project goal of creating ‘a competitive, market-oriented economy providing better economic opportunities for all its citizens’. These goal-level indicators include: 1) nonperforming loans (NPLs) of commercial banks as a percent of their total gross loans, and 2) domestic credit to the private sector as a percent of gross domestic product (GDP), both of which are sourced from secondary data. The progress towards achieving the Activity purpose, ‘improved financial sector stability’, is measured by tracking movement on 20 IMF/EU Financial Stability Compliance Requirements (CRs) toward fulfilment with the Activity’s assistance. FINRA also tracks the number of standards addressing IMF/EU CRs and other areas of financial stability which have been (i) drafted with FINRA assistance and (ii) adopted by BiH authorities; number of institutional improvements documented by FINRA and proposed to counterpart government agencies; and number of government and financial institutions staff trained with the Activity support. In the design of its performance indicators, FINRA incorporated the recommendations from the performance evaluation of USAID/BiH’s Partnership for Advancing Reforms in the Economy (PARE) Activity, FINRA’s predecessor Activity implemented from April 2008 through September 2013. Specifically, FINRA utilized PARE evaluation recommendations when designing its purpose-level indicator to monitor the number of CRs significantly advanced towards fulfillment with its assistance, and an outcome/output indicator on number of government and financial institutions staff trained. FINRA also actively engaged its key partners, four PGEs and IMF, in assessing Activity progress towards meeting the targets for its purpose-level indicator. This approach bolsters the local ownership and sustainability of the Activity results, which in turn contributes to the country’s advancement towards self-reliance. VIII | FINRA PERFORMANCE EVALUATION USAID.GOV However, the Activity targets have been set with considerable caution because of difficulties experienced with the implementation of similar interventions through the PARE Activity. Although already adjusted once, after the 2019 DQA, the Life of Activity (LoA) targets for two of the four indicators that the Activity has direct impact on have already been surpassed, and the other two are also within reach. Therefore, the targets as currently defined do not fully reflect potential achievements the Activity should strive towards in the following years of implementation. Finally, The COVID-19 crisis, whose repercussions are beyond FINRA's control, is expected to negatively affect the country’s financial stability and undermine the ‘no external shocks’ assumption that underpins the broader environment stability necessary for achieving the Activity Goal. EVALUATION QUESTION 2: HOW IS FINRA ASSISTANCE PERCEIVED BY BENEFICIARIES AND OTHER STAKEHOLDERS FROM THE STANDPOINT OF ITS DESIGN AND IMPLEMENTATION? FINRA addressed key issues identified in the BiH financial sector. Specifically, during the Activity design process, USAID consulted the PGEs and the IMF to determine priorities in the financial stability sector. FINRA’s design also incorporated the recommendations of the key financial sector assessments conducted by the IMF and the World Bank Group (WBG) in 2015-2017, as well as the requirements of the EU RA. In addition to focusing on these recommendations, FINRA also accommodated specific sector priorities raised by the PGEs themselves that had not been identified through the key financial sector assessments used to inform the Activity design. Implementation framework for the first two Activity components and the C3 differs. The first two Activity components were developed and implemented in close collaboration with the beneficiary institutions. As the optional part of the Activity, subject to demand and availability of funding, C3 work started late and was mainly under the Mission‘s direct programmatic guidance. Predominantly due to its constricted timeframe and the onset of the COVID-19 pandemic, C3 yielded limited results to date. The work on this component will cease at the end of FINRA’s Base Period. FINRA established partner and trustworthy relationship with the PGEs and international organizations active in the BiH financial sector. FINRA’s interventions are well tailored to the needs and absorption capacity of the beneficiary institutions which is achieved through joint annual planning and permanent two-way communication. FINRA also works closely with other international organizations active in the financial sector to ensure complementarity of interventions and its success and contribution are explicitly recognized by its counterparts. One of FINRA’s key strengths is its extensive pool of experts. FINRA's consultants are selected in consultation with the beneficiary institutions. They are generally perceived as highly knowledgeable, flexible, and cooperative, and most of them speak the local language and are familiar with the complex local environment. FINRA’s Grant Fund budget is unlikely to be fully disbursed by the end of the Activity. The implementation of the Grant Fund has been assessed as effective and satisfactory. However, due to low demand for grant funds, to date FINRA has allocated only 42 percent of the $1 million Base Period Grant Fund budget. The Grant Fund was augmented by additional $0.5 million when FINRA was extended for two additional years, but with only a couple more grant applications in the pipeline it is unlikely that these funds will be fully utilized. IX | FINRA PERFORMANCE EVALUATION USAID.GOV FINRA's interventions faced number of politically based challenges, the major one being the delay in adopting the Law on Deposit Insurance. Nevertheless, FINRA’s flexible management and planning helped in preventing major implementation problems. On the other hand, FINRA’s implementation successes and close partnerships allowed it to generate added value for its beneficiaries in the country, most palpably through supporting organizational changes in the CBBH and the drive of the entity banking agencies to achieve the EU regulatory equivalence recognition, while generally improved communications between the beneficiary institutions represent a major additional benefit. EVALUATION QUESTION 2A: HOW HAS FINRA ASSISTED ITS BENEFICIARIES TO RESPOND TO THE POTENTIAL ECONOMIC FALLOUT CAUSED BY THE COVID-19 PANDEMIC AND HOW DO BENEFICIARIES PERCEIVE THIS ASSISTANCE? Although causing some delays, the COVID-19 will not prevent completion of any planned FINRA activities. Whenever possible, FINRA mitigates the challenges in implementation caused by the COVID-19 pandemic by switching entirely to online work mode. FINRA also supported PGEs in mitigation of the economic and financial impacts of COVID-19, mainly through advisory services on appropriateness of potential responsive measures or applicability of EU practices. FINRA’s advice was seen as crucial in enabling the CBBH to successfully manage public relations in the early months of the crisis and to withstand populist pressures which could have compromised the institution’s independence and the country's financial stability. EVALUATION QUESTION 3: TO WHAT EXTENT DO STAKEHOLDERS AND BENEFICIARIES PERCEIVE PGES ARE ABLE TO ADVANCE COMPLIANCE WITH THE INTERNATIONAL REGULATORY REQUIREMENTS IN BANKING SUPERVISION, DEPOSIT INSURANCE, AND CENTRAL BANKING COORDINATION ACTIVITIES WITHOUT USAID ASSISTANCE? PGEs perceive that their progress towards compliance with international standards and requirements would be much slower without FINRA. Therefore, although their capacities have improved with FINRA’s assistance, they still find the Activity’s support necessary to continue their progress in aligning with these requirements, especially with regards to continuously evolving EU regulatory framework. Moreover, USAID/BiH is recognized as having a long-lasting presence in the BiH financial sector, and its engagement and support are perceived as necessary to bolster further sector development. Despite substantial improvement in BiH banking regulation facilitated by FINRA, considerable and distinctive shortcomings in the BiH regulatory environment persist, especially with regard to regulators’ capacity to deal with a potential or developing crisis and most notably, in the case of an imported crisis of confidence. Such a crisis may occur if one of the international banking groups in BiH experiences a crisis in other markets. As a consequence, their BiH subsidiary might suffer a liquidity shock and would be unable to restore lost trust of its owner singlehandedly. While the likelihood of such an occurrence is low, the limited options available to BiH’s banking regulatory institutions, particularly the absence of the ‘lender of last resort’ arrangement, make this risk real, and its consequences potentially grave. In addition to the need to address the lack of a ‘lender of last resort’ arrangement, the following financial sector priorities were brought up most frequently by the KIs: (i) further improvement in harmonization with the EU financial regulatory standards; (ii) capacity building of local employees in X | FINRA PERFORMANCE EVALUATION USAID.GOV the financial sector; (iii) financial education of the general population (e.g., money transfer, money laundering); (iv) adoption of the Supervisory Review and Evaluation Process (SREP) methodologies; (v) establishment of a Restructuring Fund; and (vi) expanding access to finance and specifically access to non-banking finance. RECOMMENDATIONS The widespread approval and resoundingly positive assessment of FINRA by all stakeholders merits a comprehensive study to identify the relevant good practices and lessons learned for potential replication. USAID/BiH’s strategic donor coordination effort benefitted the Activity design process and was well complemented by FINRA’s approach to operational consultations with key stakeholders during the implementation stage. The combined effect was broadly perceived as crucial for the Activity’s success in responding to the main challenges in its work and may also be considered as a model for future USAID interventions. FINRA’s model of involving local stakeholders in the progress assessment process might help strengthen local ownership and sustainability of results for other USAID/BiH interventions as well. Such practice seems to be already acknowledged by the Mission since recently inititated USAID/BiH’s Activities were encouraged to pursue the same approach, including the Justice Against Corruption Activity (JACA), Energy Policy Activity (EPA), and E-governance Activity. Moreover, engaging stakeholders in assessing Activities’ progress has been included in the request for proposals (RfP) for the upcoming USAID/BiH’s Financial Reform Activity in BiH: Greater fiscal accountability and transparency in BiH. However, active stakeholder engagement could be further reinforced by ensuring that feedback on final assessment results is systematically shared with key stakeholders, which would represent an added boost to results. In addition to good practices that USAID/BiH should consider replicating in its ongoing or future interventions, the evaluation team also identified areas for improvement that the Mission should consider to address. As FINRA is at the point of meeting LoA targets for its performance indicators with two years of implementation remaining, USAID/BiH should consider possible options for broadening the Activity’s scope of work and/or revising its objectives and/or targets for the remaining implementation period to ensure the Activity’s full implementation potential is realized and maximum benefits extracted from the Activity’s sound design and the IP’s noteworthy implementation competence. While FINRA’s Grant Fund made it easier to meet some of its beneficiaries’ needs for technology upgrades, the demand has been lower than expected. Therefore, USAID should consider introducing an option of transferring part of the unallocated FINRA’s Grant Fund to technical assistance to help optimize the Activity’s overall benefits. In addition to providing continual support to the BiH banking regulation agencies/institutions, to preserve and further reinforce financial stability in BiH, USAID/BiH should consider identifying channels to start/reopen the discussion on the burning issues, including:  lack of the ‘lender of last resort’ function;  lack of a government backstop for the deposit insurance scheme or failing banks;  incomplete resolution framework; and  failure to define the list of strategically important banks. Finally, whenever possible, USAID should consider implications of the COVID-19 crisis on prospects, opportunities, and risks in implementation of its ongoing or future Activities and ensure they are flexible enough to adjust to work under any potential external shocks. 1 | FINRA PERFORMANCE EVALUATION USAID.GOV BACKGROUND INFORMATION ON THE ACTIVITY The Financial Reform Agenda Activity (FINRA) is a $5,999,136 USAID/BiH-funded Activity implemented by the Financial Markets International, Inc. (FMI). The Activity started in September 2017 and was envisaged to end in September 2020 with the possibility of a two-year extension. The extension option was approved under the contract amendment signed in September 2019 with September 30, 2022 set as the Activity’s end date. The general purpose of FINRA is to enhance the stability of BiH financial sector. The Activity contributes to USAID/BiH’s Country Development Cooperation Strategy (CDCS) 2012-2020 Development Objective (DO) 2: BiH is a competitive, market-oriented economy providing better economic opportunities for all its citizens, and specifically to the Intermediate Result (IR) 2.2: Improved economic aspects of governance relevant to business activity. The results that the Activity is required to accomplish are defined in its contract as follows:  Result One: BiH advances towards compliance with the European Union (EU) accession requirements, as set out in the Reform Agenda (RA), the EU Progress Report and the International Monetary Foundation (IMF) program in the areas of strengthening financial stability, public audit and quality of government spending.  Result Two: Policies, laws and regulations from the IMF Extended Fund Facility (EFF), Financial Sector Assessment Program (FSAP), the EU RA, and those required by the EU Stabilization and Association Agreement (SAA) process are drafted for adoption and implementation in the financial, audit and quality of government spending areas. To achieve these results, the Activity is organized in three components:  Component 1- banking supervision and insurance,  Component II - central banking,  Component III (optional): audit of public financial resources and quality of government spending. According to the Activity contract, FINRA is required to closely cooperate with key implementing PGEs which are the key financial sector regulators in BiH: the Central Bank of BiH (CBBH), BiH Deposit Insurance Agency (DIA), FBiH Banking Agency (FBA), and the RS Banking Agency (BARS). COMPONENT 1: BANK SUPERVISION AND INSURANCE Two functional areas of FINRA’s work, bank safety and soundness, and problem bank management are at the core of Component 1 (C1). The types of work envisaged under this component include capacity building, training and in-kind assistance for banking supervision regulators and all other relevant host country partners to implement the new banking regulation framework, in close consultation with other international organizations present in the sector, namely the IMF and the World Bank Group (WBG). FINRA is expected to assist the entity banking agencies in aligning banking supervision with Basel III and EU directives, including the Directive 2013/36 (CRD) and Regulation No. 575/2013 (CRR), and all their amendments. The Activity is also expected to assist the DIA with the implementation of the regulatory framework aligned with the IMF requirements. In addition to assisting the entity banking agencies and DIA to advance compliance with specific international requirements, FINRA is also envisaged to enhance the capacities of its PGEs to implement the new regulations/framework. 2 | FINRA PERFORMANCE EVALUATION USAID.GOV COMPONENT 2: CENTRAL BANKING As part of its Component 2 (C2), FINRA is expected to provide demand-driven assistance to CBBH in aligning its operations with the EU requirements and best practices and with the IMF’s FSAP and Financial System Stability Assessment (FSSA) requirements. The Activity is also entrusted with strengthening the capacity of the CBBH as a leading country financial institution in the areas where the CBBH could strengthen its organization and enhance its performance. COMPONENT 3: AUDIT OF PUBLIC FINANCIAL RESOURCES AND QUALITY OF GOVERNMENT SPENDING Component 3 (C3) was planned as an optional component, to be implemented if budget resources are available and when the demand for this type assistance is confirmed. According to the Activity contract and as part of this component, FINRA is expected to provide assistance to BiH and entity level governments in meeting the EU RA policy benchmarks related to the quality of government spending. This is a separately priced Contract Option (CLIN 0003) that the Activity, based on the Mission’s approval, began implementing in Year 2. FINRA GRANT FUND To complement the technical assistance provided throughout the Activity components, the contract envisaged establishment of a grant program to support the development of PGE’s information and communications technology (ICT) systems or other PGEs’ needs and priorities. The program budget was $1.0 million for the base period, with additional $500,000 approved for the two-year extension period. The contract envisaged the adoption of the Grants Management Manual as a guidebook for administering the grants under the contract (GUC). ACTIVITY MONITORING According to the Activity Monitoring, Evaluation, and Learning (MEL) Plan, FINRA tracks six indicators to measure its progress towards achieving defined Activity results. These include two secondary sourced high-level indicators that FINRA tracks but is not responsible for at the Activity goal level, one performance indicator it collects data for at the Activity purpose level, and three outcome/output performance indicators. Exhibit 1 presents all six indicators with accompanying annual and Life of Activity (LoA) targets. 3 | FINRA PERFORMANCE EVALUATION USAID.GOV EXHIBIT 1. ACTIVITY INDICATORS WITH ANNUAL AND LIFE OF ACTIVITY (LoA) Level of result Narrative Summary Indicators Targets Life of Activity FY 2018 FY 2019 FY 2020 FY 2021 FY 2022 (LoA) Targets Activity Goal A competitive, market-oriented economy providing better economic opportunities for all of its citizens Nonperforming loans (NPLs) of commercial banks as a percent of their total gross loans* n/a n/a n/a n/a n/a n/a Domestic credit to the private sector as a percent of GDP* n/a n/a n/a n/a n/a n/a Activity Purpose Improved financial sector stability Number of IMF/EU Financial Stability Compliance Requirements significantly advanced toward fulfillment with FINRA assistance** 5 10 10 10 10 18 Activity Outcome/ Output 1 Policies, laws, regulations, and implementing procedures (“Standards”) are drafted for adoption in the financial, audit and quality of government spending areas Number of Standards addressing IMF/EU Compliance Requirements and other areas of financial stability (i) drafted with FINRA assistance and (ii) adopted by BiH authorities** 10/8 20/15 20/15 10/10 5/5 65/53 Activity Outcome/ Output 2 Strengthened managerial organization and operational practices, procedures, and tools (“Institutional Improvements”) of Counterparts Number of Institutional Improvements documented by FINRA and proposed to Counterpart government agencies 4 20 10 10 5 49 Activity Outcome/ Output 3 Strengthened professional capacities of Counterpart agencies and financial institution staff Number of staff of Counterpart agencies and financial institutions trained with FINRA support 50 70 70 50 40 140 * Higher level indicators that are tracked, but not targeted ** Project indicator 4 | FINRA PERFORMANCE EVALUATION USAID.GOV EVALUATION PURPOSE, QUESTIONS, METHODS, AND LIMITATIONS EVALUATION PURPOSE The main purpose of this mid-term performance evaluation is to assess FINRA’s progress towards achievement of its contractual obligations to date. Specifically, the evaluation aims to inform the Mission about the scale and determinants of FINRA’s performance and collect evidence on best approaches to facilitate programming in the financial sector regulatory compliance. Evaluation findings, conclusions, and recommendations are expected to promote collaboration, cultivate learning, expand the existing technical evidence base, and encourage program design adaptations within the Mission, as well as among other evaluation stakeholders. Knowledge, to be generated by the evaluation, will support USAID/BiH’s evidence-based decision-making and therefore, help Mission support BiH on its journey to self-reliance (J2SR). In addition to the USAID/BiH, evaluation findings, conclusions, and recommendations will contribute to the knowledge and learning of several groups of stakeholders: FOR THE IMPLEMENTING PARTNER (IP), the evaluation’s findings will offer a valuable independent reflection on the approaches adopted and the degree of success in meeting the Activity’s goals and encourage improvements in their future efforts. THE GOVERNMENTS IN BiH AND THEIR RELEVANT AGENCIES will receive independent insight into their progress towards meeting the IMF’s and EU policy benchmarks related to the financial sector that are integral for the European integration process. FOR THE LEADING INTERNATIONAL ORGANIZATIONS, notably the WBG and IMF, whose priorities and conditions provided the core basis for the design of the Activity, the benefit of this evaluation will be in an external overview of the progress in the financial sector reforms that they are contributing to. EVALUATION QUESTIONS The FINRA performance evaluation answers the following evaluation questions (EQs): EQ1: What progress has been accomplished in reaching contract targets? EQ2: How is FINRA assistance perceived by beneficiaries and other stakeholders from the standpoint of its design and implementation? EQ2a: How has FINRA assisted its beneficiaries in responding to the potential economic fallout caused by the COVID-19 pandemic and how do beneficiaries perceive this assistance? EQ3: To what extent do stakeholders and beneficiaries perceive PGEs are able to advance compliance with the international regulatory requirements in banking supervision, deposit insurance, and central banking coordination activities without USAID assistance? 5 | FINRA PERFORMANCE EVALUATION USAID.GOV METHODOLOGY In order to produce relevant findings, conclusions and recommendations, the evaluation is based on the mixed method approach. The evaluation team triangulated data across four different sources to answer the evaluation questions and ensure systematic and efficient data collection. The following data sources and techniques were used:  Desk review of relevant documentation and data, including Activity documentation and databases, as well as documentation from relevant BiH government/public institutions and international organizations. For the full list of reviewed documentation, see Annex II.  Key informant interviews (KIIs) with six groups of stakeholders. Overall, the evaluation team conducted 34 KIIs with 44 key informants (KIs) using semi-structured interview guides. Exhibit 2 highlights the number of KIIs per stakeholder category and the number of KIs included. The full list of key informants is provided in the Annex III along with the semi￾structured interview guides in Error! Reference source not found.IV. In view of the o ngoing COVID-19 pandemic, interviews were arranged and conducted using suitable online tools. EXHIBIT 2. NUMBER OF KIIs and KIs Key informants Number of KIIs Number of KIs USAID/BiH 1 3 IP 1 2 PGEs 14 17 International organizations 5 6 FINRA consultants 10 11 Non-beneficiaries 3 5  Online surveys were administered to FINRA’s beneficiaries and key stakeholders to gather additional quantitative evidence to inform the evaluation questions. The surveys were designed and administered using the SurveyMonkey platform and distributed by email to PGEs’ staff members who participated in FINRA capacity building workshops/seminars and consultants who were engaged in implementing specific FINRA interventions. Contact information for the target audience was provided by the IP. Participation in the surveys was voluntary and anonymous. The sample size of the surveyed stakeholder groups and surveys’ response rates are described below in Exhibit 3. The survey questionnaires are provided in the Annex V. EXHIBIT 3. ONLINE SURVEYS RESPONSE RATE Target group Sample # of respondents Response rate PGE’s staff members who participated in FINRA capacity building workshops/seminars 31 24 77.4% FINRA consultants 102 36 35.3% At the suggestion and with the assistance of the Association of BiH Banks, the evaluation team conducted an additional short survey of banks to collect their inputs on priority areas in the banking sector that could be addressed in the upcoming period. Disseminated by the 6 | FINRA PERFORMANCE EVALUATION USAID.GOV Association, the survey had a very low response rate (nine percent), which can be explained by the short period of time available for collecting responses and its timing in late July which coincides with the summer holiday season. Whenever possible, the evaluation team triangulated data from different data sources and stakeholders to address the evaluation questions from multiple perspectives and to provide relevant and valid findings. The Complete Evaluation Methodology Matrix, which presents in detail the methodological approach used to answer each evaluation question, is provided in the Annex VI. LIMITATIONS Different sources of bias, including response bias, interview bias, online survey methodology bias, and environment bias (COVID-19), were the limiting factors for this performance evaluation. The evaluation team employed different approaches to mitigate these biases and ensure timely and objective findings, conclusions, and recommendations:  Response bias, including possible over- and understatement of positive effects by the Activity beneficiaries. The evaluation team triangulated data across multiple data sources to verify the credibility of results and mitigate the response bias.  Interviewer bias. Interviewers’ behavior and reactions may lead KIs to respond in a certain way. The evaluation team avoided asking leading questions and assured that respondents understood that their honest opinions were the best contribution to the evaluation. In addition, the evaluation team ensured that respondents knew that their contributions would be anonymized and that there will be no attribution of their responses.  Limitations related to the online survey methodology, including low response rates for online surveys. To mitigate this, the evaluation team requested the PGEs’ assistance in disseminating the survey to their staff members who participated in FINRA workshops, as well as FINRA assistance in disseminating the survey to the consultants who were engaged FINRA’s work. The evaluation team sent multiple reminders to potential survey respondents to improve response rates.  Environment limitations. o The effects of FINRA interventions on the financial sector’s performance and the economy as a whole may be difficult to separate from the influence of larger market and environmental forces (such as the economic crisis triggered by the COVID-19 pandemic). The evaluation team triangulated across multiple sources to validate the data and produce objective findings. o Possible constraints related to the impossibility of conducting in-person KIIs due to the COVID-19 pandemic and related safety measures: the evaluation team mitigated this limitation by opting for remote rather than in-person interviews. Depending on their preferences and available means of communication, the evaluation team used phone and/or online tools to conduct KIIs. 7 | FINRA PERFORMANCE EVALUATION USAID.GOV FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EVALUATION QUESTION 1: WHAT PROGRESS HAS BEEN ACCOMPLISHED IN REACHING CONTRACT TARGETS? MEL PRACTICES Finding 1. FINRA incorporated the recommendations from the performance evaluation of USAID/BiH’s Partnership for Advancing Reforms in the Economy (PARE) Activity, when defining its performance indicators. Performance evaluation of USAID/BiH’s PARE, FINRA’s predecessor Activity implemented from April 2008 through September 2013, offered a list of recommendations for USAID to consider in designing and implementing its future interventions. One of the principal recommendations provided in the evaluation report proposed the use of benchmarks for defining performance indicators for measuring progress towards adopting regulatory framework/documents. The evaluation team found that this approach had been applied in defining the Activity purpose-level indicator – “Number of IMF/EU Financial Stability Compliance Requirements (CRs) significantly advanced toward fulfilment with FINRA assistance”. Specifically, the Activity defined twenty CRs it aimed to address during its implementation as benchmarks of its contribution to the BiH’s financial stability. The selection of the CRs was informed by the key financial stability-related documents produced in the 2015-2017 period by the IMF, WBG, and the EU and reflect some of the major financial sector priorities identified through these documents. A complete list of the relevant reports is available in Annex VII. Similarly, the Activity accommodated the recommendations of the PARE performance evaluation when defining one of its outcome/output indicators – “Number of staff of Counterpart Agencies and financial institutions trained with FINRA support”. The PARE evaluation team suggested counting unique participants in capacity building events to avoid double-counting in case a given participant attends more than one training event delivered by the Activity. Pursuant to this recommendation, FINRA's number of staff of Counterpart Agencies and financial institutions trained with FINRA support presents the number of unique individuals trained within a reporting period. However, the evaluation team noted that the Activity does not disaggregate data on the indicator “Number of staff of Counterpart Agencies and financial institutions trained with FINRA support” by sex. Such practice contradicts the USAID’s requirement for disaggregation of performance data by sex for all performance indicators that capture data about people, as noted in the USAID Automated Directives System (ADS) Chapter 201 Program Cycle Operational Policy and Chapter 205 Integrating Gender Equality and Female Empowerment in USAID’s Program Cycle. The need for disaggregating person-level data by sex was not captured nor recommended by the data quality assessment (DQA) conducted in early 2019. However, the evaluation team noted that, although this information is not reported in the regular progress reports submitted to USAID nor through the BiHPERFORM system, FINRA keeps sex-disaggregated data for their trainees in a separate database and should be able to provide it to USAID retroactively. Finding 2. The Activity’s key partners are involved in assessing FINRA's progress towards meeting the targets for the Activity purpose-level indicator. The Activity contract defined the CBBH, DIA, and two entity banking agencies as key local institutions FINRA should work with (referred to as partner government entities (PGEs) in the Activity’s documents). The evaluation team found this approach to transcend the implementation of specific interventions, as the PGEs are also involved in assessing the progress towards the 20 CRs defined at the Activity purpose level. Moreover, FINRA involved the IMF, as a relevant international financial organization whose reports 8 | FINRA PERFORMANCE EVALUATION USAID.GOV informed the CRs, in the annual assessments of the FINRA’s progress towards achieving targets for the Activity’s purpose-level indicator. Upon receiving independent ratings for each CR from each partner institution/organization, which was confirmed in seven relevant KIIs, FINRA calculates the average scores. Those CRs whose average annual score equals or exceeds 2.5 are counted as having recorded significantly advanced progress. However, the final scores calculated based on individual partners’ inputs are not communicated back to the participating institutions/organizations. This was confirmed during three out of seven relevant KIIs. Engaging key stakeholders in assessing the Activity’s progress has already been recognized as good practice by USAID, since a number of new USAID-funded Activities were encouraged to take a similar approach, including the Justice Against Corruption Activity (JACA), Energy Policy Activity (EPA), and E-governance Activity. Moreover, according to the Request for Proposals (RfP) for the yet to be initiated USAID's Financial Reform Activity in BiH: Greater fiscal accountability and transparency in BiH, the selected IP is required to involve relevant stakeholders in assessing the progress achieved toward meeting the Activity purpose-level indicators. MEL REPORTING Finding 3. The COVID-19 pandemic will undermine the ‘no external shocks’ assumption that underpins the broader environment stability necessary for achieving the Activity Goal. According to the Activity MEL Plan, achieving FINRA’s purpose of improved financial sector stability would contribute to a competitive, market-oriented economy and provide better economic opportunities for all its citizens. The Activity postulated that, for it to contribute to the Activity Goal, the assumption of “no significant exogenous shocks/contagion from international financial disturbances, business cycle downturn, or significant domestic political instability” that can “interrupt the reform/improvement process” should hold. However, this assumption has been invalidated by the onset of the global economic crisis caused by the ongoing COVID-19 pandemic. The local currency BAM has depreciated by around three percent since the beginning of the year, which signals potential for capital outflows and renders international trade and investment decisions more difficult.1 In the first six months of 2020, the export decreased by 15.1 percent compared to the same period of 2019.2 Moreover, BiH gross domestic product (GDP) is expected to decrease by 6.6 percent in the 2020.3 Although non-performing loans (NPLs) were declining in the period before the crisis, the percentage as share of total gross loans was still high (8.8%).4 The resilience of the financial sector in BiH to the COVID-19 shock is currently assessed as high.5 However, as the crisis is ongoing, its consequences are still hard to predict. Finding 4. In early 2019, the Activity modified its MEL Plan to accommodate the DQA recommendations. Following the DQA6 conducted in February 2019, a list of recommendations was provided to improve FINRA’s performance monitoring. The Activity accommodated five out of the total of nine DQA recommendations (see Annex VIII) and the MEL Plan modifications included the following changes: (i) the Activity increased its annual and/or LoA targets for the purpose-level and outcome/output indicators, and (ii) the Activity eliminated the indicator on “Hours of training delivered to Counterpart agencies and financial institutions staff with FINRA support”. 1 OECD, Update on COVID-19 situation, July 2020 2 Agency for Statistics of BiH. 2020. BiH International Trade in Goods Statistics: January-June 2020 3 World Bank. 2020. Global Economic Prospects 4 World Bank. Bank NPLs to total gross loans (%): BiH 5 European Bank for Reconstruction and Development (EBRD). 2020. Resilience to the COVID-19 shocks in the EBRD regions 6 USAID, ADS Chapter 201 Program Cycle Operational Policy 9 | FINRA PERFORMANCE EVALUATION USAID.GOV Finding 5. The Activity already surpassed the LoA targets for two of its outcome/output indicators and is close to reaching the LoA target for its purpose-level and one remaining outcome/output indicator. By the end of its Year 2, FINRA’s actuals nearly reached the LoA targets for two of its tracked indicators and exceeded the targeted levels for the remaining two, in both cases by a very considerable margin. For its purpose-level indicator “Number of IMF/EU Financial Stability CRs significantly advanced toward fulfillment”, the Activity’s achievement was 16, just two CRs short of the target for the full five-year implementation period. Among the 20 CRs the Activity is envisaged to address during its implementation, only one (5%) is perceived to be completely fulfilled (“Develop banking agency organizational structures emanating from the new bank resolution mandate”).7 Fifteen CRs are assessed as significantly advanced with FINRA’s assistance (75%),8 while the remaining four CRs were not addressed by the beginning of Year 3 (20%). Exhibit 4 provides an overview of CRs addressed by FINRA across three areas of assistance, while the complete list of CRs accompanied by average scores attributed to their advancement in Year 1 and Year 2 is provided in the Annex IX. EXHIBIT 4. OVERVIEW OF CRs ADDRESSED BY FINRA Area CRs not addressed yet CRs addressed but not yet significantly advanced CRs addressed and significantly advanced Total CRs Average score of all CRs Bank safety and soundness 1 0 7 8 3.5 Problem bank management 2 0 5 7 4.4 Central banking 1 0 4 5 3.8 4 0 16 20 3.9 The actuals for the outcome/output indicators “Number of Institutional Improvements documented by FINRA and proposed to Counterpart Government Agencies” and “Number of unique staff of Counterpart Agencies and financial institutions trained with FINRA support” are substantially higher than projected (actuals 71 and 244 vs. the targets of 49 and 140, respectively). This occurred even after the targets had been increased considerably (from 22 and 100, respectively) following the 2019 DQA. The Activity also overperformed on the “Number of Standards addressing IMF/EU CRs and other areas of financial stability (i) drafted with FINRA assistance and (ii) adopted by BiH authorities” by exceeding the defined Year 1 and Year 2 annual targets. Although the number of Standards addressing IMF/EU CRs and other areas of financial stability drafted with FINRA assistance exceeded the LoA target (73 vs. the target of 65), the second component of this indicator (adopted by BiH authorities) is still below the LoA target (48 vs. 53).9 Exhibit 5 provides a detailed overview of Activity annual and LoA targets and results achieved by the end of Year 2. According to the Activity reports, and as confirmed during one KII, FINRA did not envisage such high demand for capacity building assistance. Accommodating such demand resulted in much higher number of staff trained with FINRA support than initially expected. Based on information from the same sources, the evaluation team also found that the targets for the remaining two outcome/output indicators were defined with considerable caution because of difficulties experienced with the implementation of similar interventions through the PARE Activity. The originally defined target levels were found to be too low during the 2019 DQA. Even though the Activity accommodated DQA recommendations related to increasing the targets for all three outcome/output indicators, already in Year 2 FINRA exceeded even the adjusted targets. 7 CRs with average score of 5 are considered as completely fulfilled. 8 CRs with average score of 2.5 or higher are considered as significantly advanced towards fulfilment. 9 Note that this is the data as of end of Year 2, and it is reasonable to expect that, even with the onset of the pandemic, some additional progress towards the targets was made in Year 3. 10 | FINRA PERFORMANCE EVALUATION USAID.GOV EXHIBIT 5. ACTIVITY INDICATORS WITH ANNUAL AND LIFE OF ACTIVITY (LoA) TARGETS AND ACTUALS Level of result Narrative Summary Indicators Targets (Actuals) Life of Activity (LoA) Targets Attained actuals at the end of Year 2 FY 2018 FY 2019 FY 2020 FY 2021 FY 2022 Activity Goal A competitive, market-oriented economy providing better economic opportunities for all of its citizens Nonperforming loans (NPLs) of commercial banks as a percent of their total gross loans* n/a (10.0%) n/a (8.9%) n/a n/a n/a n/a Domestic credit to the private sector as a percent of GDP* n/a (58.3%) n/a (58.6%) n/a n/a n/a n/a Activity Purpose Improved financial sector stability Number of IMF/EU Financial Stability Compliance Requirements significantly advanced toward fulfillment with FINRA assistance** 5 (5) 10 (16) 10 10 10 18 16 Activity Outcome/ Output 1 Policies, laws, regulations, and implementing procedures (“Standards”) are drafted for adoption in the financial, audit and quality of government spending areas Number of Standards addressing IMF/EU Compliance Requirements and other areas of financial stability (i) drafted with FINRA assistance and (ii) adopted by BiH authorities** 10/8 (37/30) 20/15 (36/18) 20/15 10/10 5/5 65/53 73/48 Activity Outcome/ Output 2 Strengthened managerial organization and operational practices, procedures, and tools (“Institutional Improvements”) of Counterparts Number of Institutional Improvements documented by FINRA and proposed to Counterpart government agencies 4 (27) 20 (44) 10 10 5 49 71 Activity Outcome/ Output 3 Strengthened professional capacities of Counterpart agencies and financial institution staff Number of staff of Counterpart agencies and financial institutions trained with FINRA support 50 (120) 70 (124) 70 50 40 140 244 LEGEND Target not exceeded Target met Target exceeded Data still not collected * Higher level indicators that are tracked, but not targeted ** Project indicator 11 | FINRA PERFORMANCE EVALUATION USAID.GOV Finding 6. There are discrepancies in FINRA’s performance among sources, including annual progress reports and the BiHPERFORM system. The evaluation team found discrepancies in reporting actuals for Year 1 for the following performance indicators:  “Domestic credit to the private sector as a percentage of GDP”, and  “Number of institutional improvements documented by FINRA and proposed to Counterpart Government Agencies” Specifically, the reported actual for the “Domestic credit to the private sector as a percentage of GDP” for Year 1 was 58.3 percent in the Activity reports and 54.1 percent in the BiHPERFORM system. In terms of the “Number of institutional improvements documented by FINRA and proposed to Counterpart Government Agencies”, the actual reported in Year 1 was 27 in the Activity reports and 14 in the BiHPERFORM system. The evaluation team also found that the Activity did not report amounts it spent between July 2019 and April 2020 nor did it upload the Year 3 Work Plan in the BiHPERFORM system. However, no BiHPERFORM issues or challenges were mentioned during the KIIs. CONCLUSIONS The design of the Activity’s MEL Plan embodies good collaboration, learning, and adapting (CLA) practices primarily through reliance on existing evidence to inform the design of its performance indicators. In terms of deepening its partnerships, by engaging its key beneficiaries in the assessment of progress towards achieving its results, FINRA bolsters local ownership and sustainability of the Activity results, which helps the country’s advancement towards self-reliance. Already in Year 2 (FY 2019), the Activity exceeded or was very close to exceeding the LoA targets for its performance indicators. The targets, as defined in the current MEL Plan, do not reflect the Activity’s potential achievements for the remaining years of implementation. The COVID-19 crisis, whose repercussions are beyond FINRA's control, is expected to negatively affect the country’s financial stability, which would create considerable risks and impediments for the attainment of the Activity Goal. Although FINRA is not responsible for the Activity Goal, the negative effects and ramifications of the COVID-19 are expected to be extremely large and to spill over to the financial sector as well. Therefore, regardless of FINRA’s promising progress towards achievement of its purpose, the pandemic and its accompanying economic and financial crisis are expected to inflict considerable damage to normal operation of the competitive, market-oriented economy and to its chances to provide better economic opportunities for all of its citizens. The Activity’s reporting through the BiHPERFORM system, which is of paramount importance for timely informing USAID/BiH’s decisions, has not been fully consistent with its periodic reporting requirements and leaves room for improvement. 12 | FINRA PERFORMANCE EVALUATION USAID.GOV EVALUATION QUESTION 2: HOW IS FINRA ASSISTANCE PERCEIVED BY BENEFICIARIES AND OTHER STAKEHOLDERS FROM THE STANDPOINT OF ITS DESIGN AND IMPLEMENTATION? ACTIVITY DESIGN Finding 7. During the Activity design process, USAID consulted beneficiary institutions, IMF, and the WBG to determine the priorities for financial sector stability. All four PGEs confirmed that USAID/BiH reached out to them during the Activity design process to discuss main priorities for safeguarding stability in the financial sector that could be integrated into the Activity design. The IMF and WBG also noted that they had been consulted by the Mission. Among the government officials the evaluation team surveyed, 75 percent perceive the Activity was designed well, while the remaining 25 percent did not provide an answer to the question. Finding 8. FINRA was designed to incorporate the recommendations of the key financial sector assessments conducted by the IMF and WBG, as well as the requirements of the EU RA. According to the Activity contract, FINRA is expected to support financial sector reforms outlined in the IMF EFF policy benchmarks and EU RA 2015-2018. Specifically, the Activity contract provides a list of 29 recommendations from the IMF FSAP (2015) and FSSA (2015) that the Activity was envisaged to address to enhance the stability and support development of the financial system in BiH. To operationalize this task, the Activity defined 20 CRs to address during its implementation. The CRs were sourced from the FSAP and FSSA, but also from the EU RA and EFF. Out of 29 FSAP/FSSA requirements listed in the Activity contract, 11 were already completed by the time the Activity initiated, five were not included in FINRA’s list of CRs, while the remaining 13 are incorporated in the final list of CRs. The Activity did not incorporate any FSAP/FSSA recommendations related to the insurance oversight and financial markets infrastructure, while the creation of a Financial Stability Fund (FSF), streamlining collateral execution procedures, and revising and expanding the insolvency regulation were also left out of the list of CRs. A complete overview of the FSAP/FSSA recommendations and their linkages to the FINRA’s CRs is provided in Annex X. During 21 out of 27 relevant KIIs, KIs confirmed that the Activity design accommodated financial sector priorities at the time. However, in addition to addressing priorities identified through relevant financial sector￾related reports published in 2015- 2017 period, FINRA also responded to specific needs raised as priorities by the beneficiary institutions themselves. Responding to specific requests expressed by the PGEs has been facilitated through both the Grant Fund and technical assistance. FINRA supported ICT systems design and upgrades through its Grant Fund based on specific grant applications received from the PGEs. FINRA also provided on-demand technical assistance to the PGEs upon request. For example, FINRA assisted the CBBH in further development of its research function, including broadening and strengthening the macroeconomic analysis. FINRA also assisted all PGEs with improving their public communications and responding to public pressure, especially during the COVID-19 crisis. “However, in parallel we discovered that we need help in other segments and areas of our work, specifically how our tasks and activities are conducted and how we perform in certain international activities that we participate in. We identified areas for improvement and FINRA was the first address to reach out to for assistance.” PGE 13 | FINRA PERFORMANCE EVALUATION USAID.GOV RELATIONSHIPS WITH PARTNER GOVERNMENT ENTITIES Finding 9. The design of FINRA’s interventions is tailored to the needs and absorption capacity of the beneficiary institutions. This was confirmed in 11 out 14 KIIs with the PGEs and by 81 percent of surveyed PGE staff members (see Exhibit 6). Although the Activity contract envisaged that FINRA would sign a Memorandum of Understanding (MoU) with each PGE to formalize their commitment to implementation of reforms and assistance, none of the PGEs confirmed that this happened. Rather, the Activity facilitated PGEs’ buy-in through their active engagement in designing each intervention. Specifically, FINRA develops the scopes of work for each of its interventions in close collaboration with the PGEs. In most cases, the Activity works with middle management on designing the scopes of work which are then communicated and approved by upper management. Application of such a ‘bottom-up’ approach was confirmed by five out of 10 FINRA consultants and three PGEs in total. As noted, both by the IP and the PGEs, planning of the interventions for the next FY has already initiated. Twelve of 14 PGEs observed that FINRA activities were flexible and could be adjusted throughout the implementation if needed. This was corroborated by 94 percent of surveyed PGE’s staff members (see Exhibit 6). The PGEs also take active part in the consultant selection process, either by proposing consultants or by participating in making final selection decisions. This was highlighted by the IP and mentioned during nine out of 14 KIIs with the PGEs. EXHIBIT 6. PGE STAFF PERCEPTION ON FAVORABILITY OF FINRA’S ASSISTANCE “In that regard, we did not initially sign a memorandum of cooperation. We don't have a contract. But even without that, it works without problems... We communicate every project task. We are provided with a project task proposal. We give our comments. When FINRA prepares the final version, we give our consent to the proposal and to the consultant CV. In case we choose a consultant, we also give some formal consent. If it is FINRA that finds a consultant, then suggestions are submitted to us to give an opinion on whether we agree for that consultant to be engaged.” PGE 14 | FINRA PERFORMANCE EVALUATION USAID.GOV Finding 10. The beneficiaries demonstrate a high level of trust in the competence of FINRA, which enables smooth implementation of FINRA activities. This was emphasized in 19 out of 26 KIIs during which trust of PGEs towards FINRA was discussed. Along with all PGEs that demonstrated strong willingness to work with FINRA, as well as confidence in and reliance on FINRA assistance, three of five international organizations made similarly positive statements. In two KIIs, the evaluation team learnt that FINRA’s strong relationship with the PGEs sometimes helps improve the collaboration between themselves, with FINRA serving as an informal facilitator. The evaluation team also found that one PGE effectively co-opted FINRA’s resources as their day-to-day advisors to top management, while other PGEs also stated that they feel free to reach out to FINRA consultants even informally for any advice or assistance. In nine of 14 KIIs with the PGEs, it was expressly noted that the Activity is highly cooperative and ready to provide ad-hoc assistance when needed. In addition, 94 percent of surveyed government institutions’ representatives perceive FINRA’s work as useful for strengthening their institutions and 97 percent think that practices and procedures employed by their institutions have improved due to FINRA assistance. DONOR COORDINATION Finding 11. FINRA works closely with other international organizations present in the financial sector to ensure complementarity of interventions. The evaluation team found that FINRA maintains high-level and continuous communication with other international organizations active in the BiH financial sector, either through the PGEs or directly. In 26 out of 27 KIIs in which complementarity of foreign assistance was discussed, it was confirmed that there were no overlaps between FINRA’s work and activities of other international organizations. This point was expressly made in all five KIIs with the international organizations and 11 out of 14 KIIs with the PGEs. Finding 12. FINRA is recognized by other international organizations as an important international player in the BiH financial sector. All international organizations interviewed during the data collection process expressed appreciation for the work that FINRA performed and acknowledged FINRA as a key ongoing international assistance program in the financial sector. One international KI even declared the intention to consult FINRA and ask for FINRA’s opinion on designing that organization’s future activities and on the right approach to the beneficiary institutions to ensure a smoother implementation of planned projects. “And in fact, I can tell you that there has been times when I have tried to push for certain reforms and there is more trust between let’s say Central Bank and FINRA than there is between the Central Bank and organization I work for. So, what I would do to try to get the reform, to move a reform will be to go through FINRA, I will approach FINRA and I will say: ‘can you see if you can make some progress in this area, because if I push too hard, I sense resistance from let’s say the Central Bank’.” International organization “It’s hard for me to say what I think FINRA should focus on in the coming years. It’s almost the other way around, I would turn to FINRA and say: ‘what do you think we should focus on in the coming years’.” International organization 15 | FINRA PERFORMANCE EVALUATION USAID.GOV HUMAN RESOURCES Finding 13. According to the perception of FINRA PGEs, the management and coordination of the Activity is successful. All but one interviewed PGE representatives perceive FINRA’s management successful, either in terms of administering and coordinating technical assistance or the Grant Fund implementation. Only one KI expressed some dissatisfaction with FINRA management because of the lack of feedback on comments and suggestions provided by the KI during implementation of one of FINRA activities. Following further analysis, the evaluation team determined that, rather than poor communication demonstrated by FINRA, the problem had been internal at the institution in question. Overall, 86 percent of surveyed government institutions’ representatives perceive FINRA management as successful, with 56 percent holding the same view concerning the Grant Fund administration. Thereby, it should be noted that 42 percent of surveyed government institutions’ representatives did not assess Grant Fund management as they had no experience with its implementation, while only three percent perceive Grant Fund management as unsatisfactory. Finding 14. FINRA's consultants are perceived as highly knowledgeable, flexible, and cooperative. In 23 KIIs, including in 12 of 14 KIIs with PGEs, during which the expertise of FINRA consultants was discussed, stated that they perceived that external experts engaged by the Activity are (i) highly knowledgeable in their own fields of expertise, (ii) well acquainted with the overall political and economy environment in the region, (iii) in most cases, speak local language, whereby the language barrier is avoided, and (iv) have experience in pursuing reform processes that BiH is currently undergoing, including the EU accession, and therefore are well-positioned to transfer knowledge and lessons learnt to ensure smooth improvement processes. The fact that the PGEs are actively involved in the consultant selection process was noted by the IP and corroborated during nine out of 14 KIIs with the PGEs. Separately, during five KIIs with PGEs, the KIs noted that they had previous experience in working with FINRA consultants, which facilitated the initial collaboration steps since they already knew and trusted these experts. In addition, in nine KIIs the PGE representatives stated that they had a need for ad-hoc assistance and that FINRA consultants proved themselves highly cooperative and willing to provide unanticipated advisory services, even on an informal basis. Lastly, all surveyed government institutions’ representatives perceive consultants’ expertise to be of a high order, while 97 percent found their teaching and training skills to be satisfactory as well. GRANT FUND IMPLEMENTATION Finding 15. Two FINRA grants have been fully implemented and dispersed, the implementation of one additional grant is ongoing, while the Activity expects to award two more in the upcoming period. The Activity contract envisaged FINRA awarding and administering grants funds for improvements to ICT systems and other necessary procurements. To respond to this requirement, the Activity developed a Grant Management Manual which describes the procedures for the solicitation, review, award, monitoring, and closing of GUCs. USAID/BiH’s Contracting Office (CO) approved the Grant Manual in March 2018. According to the Manual, designated eligible recipients of FINRA grants include the CBBH, DIA, FBA and BARS. In October 2018, FINRA delivered a two-day Grant Fund workshop for the eligible PGEs. The aim was to familiarize participants with the FINRA Grant Fund methodology and application procedure. FINRA also invited the four PGEs to designate contact persons from their institutions for the Grant Fund, but the institutions did not provide the names of designated contact persons. Although the Grant Manual envisaged the Activity publishing annual Requests for Applications (RFAs), the Activity kept an open call for applications since the Grant Fund workshop in 2018. In February 2020, the Manual 16 | FINRA PERFORMANCE EVALUATION USAID.GOV was revised, with the main changes being to expand grant eligibility to other BiH government institutions and enable awarding the directed grants based on the USAID/BiH technical direction and guidance. As of the time of writing, USAID/BiH CO approved execution of a total of five grants. Three were awarded to the DIA (2) and BARS (1) based on applications received. The remaining two are directed grants awarded to the RS Ministry of Economy and Entrepreneurship (MEERS) and FBiH Ministry of Finance (FMF). The directed grants were not subject of the FINRA Grant Selection Committee review process since they were approved by USAID/BiH directly. The total amount of five approved grants is $424,300. Four of the five grants are large grants according to the Grant Manual definition.10 As of late July 2020, three grant agreements were signed, with the total value of $320,320. Two of these grants were completely executed, while the design and procurement of the Data Warehouse and Business Intelligence System at BARS is ongoing. With regards to the grant awarded to the DIA for implementation of a public awareness survey, the disbursed amount was lower than the amount approved for award by CO as the competitive bidding resulted in a lower contract amount than originally estimated. In the case of the MEERS, a directed grant, the IP proceeded with the awarding of the grant prior to the receipt of formal USAID CO approval. The two remaining grant agreements have not yet been signed with the grantee institutions. The support to the DIA in designing the Deposit Insurance Fund (DIF) Model Software Platform, facilitated through a grant, has been postponed because of the COVID-19 crisis, while the execution of the grant for upgrading the Budget Planning and Management Information System (BPMIS) at the FMF is expected to start in September 2020. Exhibit 7 provides a detailed overview of grant fund allocations to date. EXHIBIT 7. FINRA GRANT FUNDS ALLOCATED TO DATE GRANTEE GRANT PURPOSE GRANT TIMELINE GRANT TYPE APPROVED GRANT AMOUNT DISBURSED GRANT AMOUNT Deposit Insurance Agency Public Awareness Survey Nov 2019 - Feb 2020 Small $33,000 $24,320 Ministry of Economy and Entrepreneurship of RS Registry of Subsidies to Private Sector Jan 2020 - July 2020 Large $188,000 $188,000 Banking Agency of Republika Srpska Data Warehouse and Business Intelligence System July 2020 - April 2021 Large $108,000 $108,000 Deposit Insurance Agency DIF Model Software Platform Jan 2020 - Large $54,000 n/a FBiH Ministry of Finance Budget Planning and Management Information System (BPMIS) Sept 2020 - Large $41,400 n/a TOTAL $424,400 $320,320 10 According to the Activity Grant Manual, a small grant is a grant of up to $35,000. 17 | FINRA PERFORMANCE EVALUATION USAID.GOV Finding 16. The Activity allocated 42.44 percent of the available Grants Fund of $1.0 million for the Base Period while the contract modification adds additional $500.000 for the two-year extension period. As of the time of writing, 42.44% percent of the available Grant Fund budget for the Base period has been allocated. The total value of signed grant agreements is 32.32 percent of the available amount, with 21.32 percent fully implemented. While the Grant Manual does not set a minimum share of PGEs’ co-funding, the percentage of cost-sharing was 15.7 percent on average. To facilitate the Grant Application Process, FINRA: (i) distributed a memorandum to the PGEs and invited them to identify contact persons for their respective institutions, (ii) held a two-day workshop to present the grants program, (iii) supported the tender documentation development process, and (iv) covered capacity building efforts to complement the software development, when necessary. In the contract modification signed in September 2019, an additional $500,000 was allocated to augment the Grant Fund. The Activity, however, anticipates awarding only three more grants in the remaining implementation period. The expected remaining grants include the designing and procuring an IT model for calculating deposit insurance premium with the total value of approximately $54,000 at the DIA, directed grant to FBiH Ministry of Finance for upgrading the BPMIS in the amount of $41,400, and a pending request for co-funding procurement of an IT system for reserves management at the CBBH. The Central Bank has already submitted a Concept Note for procurement of an IT solution for reserves management, but it has not been decided whether to build it in-house or purchase it from an outside vendor. To assist with this decision, FINRA provided technical assistance, including organizing study visits for a CBBH representative to neighboring central banks that use different reserve management models. However, as the CBBH remained unsure on how to proceed, they revised their initial application and requested FINRA’s support in preparing a gap analysis and functional specifications for the selected option. This support took the form of technical assistance rather than a grant. Of the eligible PGEs, only the FBA never applied for a grant. As explained during two KIIs, the FBA has been reorganizing its IT department, and once this reorganization is completed, they will define the priorities to be addressed with support of a FINRA grant. Since the IP anticipates that a large portion of the Grant Fund will not be disbursed, FINRA submitted a request to USAID/BiH to reallocate the unused Grant Fund budget to support the provision of technical assistance. The logic underlying this shift was borne out by the perception that technical assistance was more useful and in higher demand than the grant itself, expressed in six out of eight KIIs in which the Grant Fund was discussed. Finding 17. The Grant Fund management and implementation is perceived as professional and successful by the beneficiary institutions. The grant application, award and implementation itself is perceived as successful and satisfactory. The grantees did not make or cite any remarks. Out of eight KIs’ from the institutions that applied for the Grant Fund, six perceive FINRA’s support to the beneficiary institutions through the Grant Fund as effective and helpful. The Activity if found to react quickly to the grantees’ requests by organizing multiple meetings to find the fastest and most efficient solutions and providing their support throughout the entire process (e.g., preparation of tender documentation, participation in the public opening of bids, implementation, etc.). 18 | FINRA PERFORMANCE EVALUATION USAID.GOV IMPLEMENTATION CHALLENGES Finding 18. While the first two Activity components were developed and implemented in close collaboration with the beneficiary institutions, the C3 work was governed by the Mission’s programmatic guidance. The Activity C3, envisaged to support BiH government institutions in the area of audit of public financial resources and quality of government spending, was designed as optional subject to available budget resources and need for assistance. In April 2019, the Activity submitted an assessment report for its C3 which identified the needs for potential assistance to BiH governments in the public financial management (PFM) area. In the report, the Activity proposed 13 PFM priorities that FINRA can address while also elaborating the estimated level of effort (LoE) and risks for their completion. Subsequently, the Activity contract was modified in autumn 2019, whereby execution of the C3 was approved but also restricted to one year only. Specifically, C3 implementation was defined to start on September 13, 2019 and end on September 30, 2020. According to the contract modification, the total C3 funding corresponds, with some small discrepancies, to the estimated C3 budget submitted with the PFM assessment ($794,681 vs. $788,144). However, unlike the first two FINRA components, within which individual activities were developed and implemented in close collaboration with the beneficiary institutions (see Finding 9), the C3 work was governed by the Mission’s programmatic guidance. This is confirmed in the Activity documentation as well as during four out of five relevant KIIs. As of the time of writing, FINRA had been instructed by USAID/BiH to provide assistance to: (i) Sarajevo Canton Ministry of Finance in the area of program budgeting and performance planning, (ii) MEERS in establishing a registry of subsidies to the private sector, and (iii) FBiH Ministry of Finance in upgrading their BPMIS. While the assistance to Sarajevo Canton Ministry of Finance was facilitated as part of a technical assistance, USAID/BiH approved awarding grants to MEERS and FBiH Ministry of Finance to address their specific PFM-related needs. Finding 19. According to the Activity reports and KIs’ perception, C3 yielded limited results to date, predominantly due to its constricted timeframe and the onset of the COVID-19 pandemic. Among the C3 directed assistance, only implementation of the grant awarded to MEERS for establishing a registry of subsidies to the private sector has been completed. The remaining two activities under this component encountered various challenges in its implementation. Specifically, the COVID-19 crisis and corresponding imposed measures for combating the virus spreading caused delays in delivering series of capacity building workshops in budget planning to the Canton Sarajevo Ministry of Finance representatives and implementing activities envisaged under the $41,400 grant to the FBiH Ministry of Finance. This was confirmed by the Activity documentation as well as during four relevant KIIs in which implementation of C3 was discussed. Additional delays in program budgeting and performance planning related to the assistance to the Canton Sarajevo Ministry of Finance occurred due to political developments in the cantonal assembly. Specifically, the change of the ruling coalition in the Canton led to appointment of the new cantonal government and a new Minister of Finance. FINRA had to wait for these developments to settle down in order to obtain approval and continue to provide technical assistance to the Ministry. However, according to the Activity documents and one KII, the newly appointed Minister has been reserved about proceeding with FINRA assistance because of poor experience with another USAID￾funded Activity. The evaluation team reached out to the Ministry requesting for an interview but, despite several follow-ups, the Ministry never responded to the evaluation team’s calls nor e-mails. Finding 20. FINRA’s interventions in the area of central banking, deposit insurance, and audit of public financial resources/quality of government spending faced politically based challenges. The evaluation team found that FINRA faced politically based challenges in its implementation. Nineteen out of 26 KIIs in which political environment in BiH was discussed, noted that political climate represents a challenge to financial sector development. Additionally, during 10 19 | FINRA PERFORMANCE EVALUATION USAID.GOV KIIs it was observed that there has been no improvement in the political climate in the past few years. Surveyed government institutions’ representatives and FINRA consultants expressed similar views and found political climate unfavourable for improving the BiH financial sector stability (78 and 62 percent, respectively). In addition to delays in delivering capacity building workshops within the Activity C3 caused by restructuring of the Canton Sarajevo Assembly, implementation of a number of other activities was slowed down or adjusted because of political obstructions or developments. These include:  Delay in adoption of the Law on Deposit Insurance Although the Activity contract identified the Law on Deposit Insurance as essential for successful implementation of FINRA, it was adopted only in June 2020, after almost four years of delay and two and a half years after the launch of the Activity. None of the KIs found any meaningful political benefits for anybody from the delays nor any strong reasons for the Law to remain blocked in the Parliamentary procedure for so long. The long delay in the adoption of the Law on Deposits hindered FINRA’s work on strengthening the DIA’s participation in bank resolution processes. However, the Activity will now be able to proceed with building DIA’s capacities for participation in the bank resolution processes.  Improvements of organization of risk management function and practices at CBBH In early 2018, FINRA assessed risk management practices at the CBBH and provided a list of recommendations for their improvement. These recommendations included a proposal for integrating risk management as a management function, as well as introducing a position of Chief Risk Officer (CRO) and a Risk Management Committee (RMC) responsible to the Governing Board. To avoid conflicts of interest, FINRA proposed for the RMC to be chaired by one Vice Governor, but not the one whose portfolio includes financial risks. However, FINRA’s recommendations were not initially adopted. Only one KI shared his views on the reasoning behind delays in strengthening the risk management function at the Bank and ascribed those to political interests of Serb members on the CBBH Governing Board who perceived FINRA proposals as a transfer of powers between the state and entity levels, which they were opposed to. However, both Serb members of the Board were dismissed by the Serb member of the BiH Presidency in early 2019 and during the time that it took to appoint new Board members, the CBBH adopted FINRA’s recommendations, appointed the CRO and established the RMC. Finding 21. During the delay in adoption of the Law on Deposit Insurance, FINRA focused on supporting the DIA in its preparation for the Law's implementation, when it is eventually passed. KIs, in five out of six KIIs in which assistance to the DIA was discussed, noted that the absence of the Law on Deposit Insurance caused delays in FINRA’s activities designed to assist the Agency in implementing the Law. But in four out of six relevant KIIs, it was emphasized that FINRA was flexible enough to focus on interventions intended at supporting the Law adoption process and preparing the DIA for its implementation once it is finally passed. According to these KIIs and FINRA’s documentation, FINRA started building DIA capacities in bank resolution processes by including its staff in the workshops for the entity banking agencies covering same topics. FINRA also decided to support the DIA’s request for upgrading its current financial model to incorporate IFRS9, improve risk assessment, permit evaluation of the target Deposit Insurance Fund (DIF) size under different macroeconomic scenarios, and enable the introduction of risk-based deposit insurance premiums. Although the grant was approved by the USAID CO, its implementation has been delayed by the COVID-19 pandemic. The grant agreement with the DIA 20 | FINRA PERFORMANCE EVALUATION USAID.GOV has not been signed yet and the grant implementation has been postponed for later in this calendar year. ACTIVITY VALUE-ADDED Finding 22. The communication between the beneficiary institutions is perceived to have improved due to FINRA’s assistance. KIs during 16 out of 20 relevant KIIs recognized strengthened communication between the PGEs as the main unintended effect of the Activity implementation. While one of the CRs FINRA is envisaged to address focuses on improving cooperation, coordination and information exchange among PGEs to ensure effective oversight coordination, its current average score of 3.2 shows there is still room for improvement. Nevertheless, KIs in generally perceive that FINRA facilitated more frequent and effective communication between the PGEs. According to the IP, organization of joint capacity building events has been of paramount importance for the success of this approach (e.g., joint capacity building workshops in bank resolution organized for the entity banking agencies and DIA). Finding 23. With FINRA's assistance, the CBBH introduced organizational changes to improve performance of its financial stability, risk management, research, and communications functions. During seven out of twelve relevant KIIs, it was noted that FINRA’s assistance resulted in organizational changes within the CBBH, including improvements of organization of risk management function and practices, research function development, and strengthening of the public communications of the CBBH. With FINRA’s assistance, the CBBH: (i) established the RMC and introduced the position of CRO, and (ii) reorganized its research department by introducing the Unit for monetary and financial analyses and Unit for non-monetary analyses. Both organizational changes were informed by in-depth analyses produced by FINRA and approved by the CBBH. With FINRA’s assistance, the CBBH also strengthened its public communications and decided to upgrade its website and social media accounts (created a Facebook page, elevated its use of Twitter, and opened a YouTube channel). Finding 24. FINRA's ad-hoc demand-driven assistance provided during the COVID-19 pandemic is viewed as crucial to the CBBH Governor’s receiving the Vecernjakov Pecat Award for Public Service Excellence. As confirmed during three KIIs, advisory support provided by FINRA to the CBBH in crisis communications was of paramount importance for the Bank in addressing public and political pressures that emerged during the COVID-19 pandemic. This support included providing general guidance on best practices in central bank communication during a crisis, as well as drafting talking points and responses for public speeches and media statements. This support helped the Governor, as the country’s leading financial sector official, reassure the public regarding the stability of the banking system and the ability of the CBBH to manage any pressures that might arise. Such stable and confident crisis communication was recognized by the choice of the CBBH Governor as the recipient of the Vecernjakov Pecat Award for Public Service Excellence in June 2020. Finding 25. In the view of its key beneficiaries, FINRA's assistance improved the banking supervision agencies’ (BSAs’) compliance with EU standards and encouraged them to pursue EU regulatory equivalence recognition. Addressing 12 out of 20 CRs in total helped the entity banking agencies to improve their compliance with EU banking supervision practices. This alignment with the EBA standards allowed and encouraged the FBA and BARS to apply for the European Commission (EC) certification, as confirmed during four out of ten relevant KIIs. The procedure of establishment of equivalence includes a gap analysis that the EC conducts to determine a degree to which bank supervision compiles with the EU practices. To inform the analysis, the EC 21 | FINRA PERFORMANCE EVALUATION USAID.GOV team visited both entity banking agencies earlier this year. The analysis is not finalized yet and is expected in autumn 2020. Finding 26. FINRA assisted the CBBH and DIA to elevate their professional profile through a number of public events. According to the Activity documents, FINRA supported the PGEs in organizing and delivering several public events. Specifically, FINRA advised the CBBH to establish a regular annual conference with a policy and/or academic focus to strengthen its public image. FINRA also assisted the CBBH in organizing two annual conferences to date and is supporting the CBBH in preparing for its third annual conference expected in November 2020. As also confirmed by the Activity documents and during five KIIs, besides the annual conferences, FINRA also assisted the CBBH in establishing its Research Paper Series and organizing four Brown Bag seminars to date. FINRA also assisted the DIA in organizing a conference to mark the 20th anniversary of the Agency’s founding. This support included provision of advisory services as well as covering part of the event’s costs. CONCLUSIONS Overall, FINRA was well designed and responded to key priorities in the BiH financial sector identified by the international community in the relevant reports and documents published in 2015-2017 period. In addition, the Activity implementation has been consistent and flexible, thereby enabling accommodation of requests for specific assistance raised by the PGEs that had not been recognized at the time of the Activity design. Such flexible planning, along with skillful management, also helped the Activity to mitigate political challenges encountered during its implementation to date, including a major delay in adoption of the Law on Deposit Insurance. Smooth implementation of FINRA activities was facilitated by close partnerships, rather than conventional donor-recipient relationships, that FINRA established with the PGEs. Although formal cooperation agreements were never signed, the PGEs have shown remarkable commitment to working with FINRA and have continued to recognize the Activity’s contributions as useful for improving their institutions’ performance. Moreover, by actively engaging institutions in the design of its interventions and adjusting those to the PGEs’ absorption capacities, FINRA earned profound trust of its beneficiaries. Such an approach fostered local ownership and sustainability of the Activity results. FINRA’s interventions are complementary with the activities of other international organizations. Continuous communication and close working relationships established with the international organizations active in the BiH financial sector have ensured that the assistance provided by FINRA is well aligned with the contributions of other counterparts. This approach exemplifies good practice in donor coordination. FINRA's key strength is an extensive pool of professionals capable of providing technical assistance tailored to the needs of the beneficiary institutions. In addition to the knowledge of specific areas within the financial sector, FINRA’s consultants also demonstrate deep understanding of the country’s financial environment and ability to successfully adapt their experiences gained in the reform processes in their countries to the current situation in BiH. Most FINRA consultants speak the local language, which considerably facilitates communication with the PGEs. Finally, the Activity and its consultants demonstrated flexibility and willingness to provide ad-hoc assistance, even informally, when needed. 22 | FINRA PERFORMANCE EVALUATION USAID.GOV EVALUATION QUESTION 2A: HOW HAS FINRA ASSISTED ITS BENEFICIARIES TO RESPOND TO THE POTENTIAL ECONOMIC FALLOUT CAUSED BY THE COVID-19 PANDEMIC AND HOW DO BENEFICIARIES PERCEIVE THIS ASSISTANCE? Finding 27. The impact of the COVID-19 pandemic on the completion of FINRA deliverables has been limited. Beginning in March 2020, all levels of government in BiH declared a state of emergency because of the COVID-19 outbreak. The state of emergency was lifted in May 2020. However, the pandemic is still not over, with the broader implications for the economy and society still uncertain and difficult to measure. Some of the key measures introduced to combat the spread of the virus include restrictions related to movement of people and public gatherings. To address these restrictions, the Activity revised its Year 3 Work Plan and submitted a COVID-19 Risk Mitigation Plan to USAID/BiH. According to the revised Work Plan, of 63 deliverables envisaged for the Year 3, 12 were at no risk of completion due to COVID-19 (19%), mainly because they were finalized prior to the virus outbreak, 34 planned deliverables were assessed as being at low risk (54%), ten are medium risk (16%), four face high risk of non-completion (6%), and additional three (5%) relate to specific COVID-19 ad-hoc assistance to PGEs. Two of the high-risk deliverables related to EU standards and methodologies are delayed because they were conceived to draw on planned EBA activities, which have been postponed to 2021. The others have been postponed to late 2020 at the request of the PGEs because of higher-priority issues they faced during the COVID-19 crisis. In total, finalization of 28 out of 63 deliverables has been delayed (44 percent), with the completion of 20 of those expected in the FY2020 and remaining eight in the Year 4 of the Activity’s implementation period. In 16 of 19 KIIs during which COVID-19 consequences on FINRA implementation were discussed, the effect of the pandemic on FINRA was consistently described as limited. Finding 28. Whenever possible, FINRA mitigates the challenges in implementation caused by the COVID-19 pandemic by switching entirely to online work mode. Out of 51 deliverables planned but not yet completed by the time COVID-19 crisis began, the Activity The Grant Fund implementation is perceived as professional and satisfactory and the Activity ensured that all preconditions for successful execution of grants were met, including: (i) developing a Grant Manual, (ii) establishing a Grant Selection Committee, (iii) introducing the application process to the PGEs, (iv) leaving the call for grant applications open throughout the Activity duration to date, (v) leaving out cost-sharing requirements, and (vi) supporting the PGEs in the grant implementation process. Nevertheless, the demand for FINRA’s grant assistance was lower than projected, and, as a result, less than 50% of the funds available during the Base Period has been allocated. The C3 implementation arrangements limited FINRA’s options for systematic contribution to improvement of the audit of public financial resources and quality of government spending. Moreover, two of the three C3 interventions have encountered delays in implementation because of the COVID-19 pandemic, which considerably restricts the results that could be expected by the end of the Year 3, when C3 implementation will end. FINRA’s assistance capitalized on its successes and contributed to results beyond its original scope, thereby generating additional value for its beneficiaries and the country. These include improved communication among the PGEs as well as between the PGEs and other counterparts, strengthened risk and research functions in the CBBH, and encouragement to the FBA and BARS to pursue the EU regulatory equivalence recognition. 23 | FINRA PERFORMANCE EVALUATION USAID.GOV recognized that it was possible to move implementation of 20 of these (39%) online, while only six were assessed as not feasible to implement remotely (12%). The remaining deliverables are either expected to be completed on time or postponed for implementation in Year 3 or early Year 4. FINRA used various approaches to provide technical assistance, either planned or ad-hoc, online, including on-line short specialized trainings, mentoring and iterative consultations, conference calls, and email correspondence. Additional forms of capacity building support include revising methodology documents and providing organizational backing for the third CBBH annual conference due to take place in the autumn of 2020. Finding 29. FINRA responded to the beneficiary institutions’ requests for ad-hoc assistance and supported them by providing timely advisory support tailored to address the challenges caused by the COVID-19 crisis. According to two KIs from the beneficiary institutions, as well as 67 percent of surveyed PGE representatives and 58 percent of surveyed FINRA consultants, COVID-19 is currently the main challenge facing the BiH financial sector. While the crisis caused FINRA to adjust the planned implementation timelines for some of its activities in Year 3, it has also created a need for additional technical assistance to PGEs. Specifically, the Activity provided advisory support to all PGEs and supported them in addressing public and political pressures during the crisis. This included advising the FBA on introducing a partial moratorium on loan repayment and sharing with and presenting to both banking agencies the practices employed by EU banks regarding debt moratorium and guidelines for banks on how to work during the outbreak. The CBBH had also faced severe populist political pressures as the COVID crisis emerged, including pressures to reduce the bank reserve requirement and release commercial banks’ excess reserves held at the CBBH, and CBBH’s foreign exchange (FX) reserves, held to back the currency board, to support to the economy. FINRA provided expert advisory services to support the CBBH in managing these pressures and responding to the crisis adequately, while also reinforcing the CBBH’s crisis communications through assistance in preparing talking points for press releases and media statements. FINRA also advised the DIA concerning preparing the Agency for potential pandemic￾related pressures, mainly with regard to the prospective EBRD credit line of approximately $36 million to back up the DIF. Finding 30. The ad-hoc assistance provided during the COVID-19 crisis is perceived as useful and has been valued by the beneficiary institutions. FINRA's assistance to beneficiary institutions in terms of mitigating the COVID-19 crisis is perceived as useful. None of the KIs consider FINRA’s assistance unhelpful, with just two KIs from the PGEs declaring they did not need any assistance from the Activity during the crisis period. In addition to providing technical information and advice on activities and measures undertaken by EU countries in terms of relaxation of regulatory standards in order to maintain an adequate flow of finance to businesses and households, FINRA also actively counseled the PGEs on how to present their positions to the public. According to four KIIs, successful crisis communication by CBBH with FINRA’s support earned the CBBH Governor the Vecernjakov Pecat Award for Public Service Excellence (see Finding 24). CONCLUSIONS The COVID-19 has affected FINRA implementation. However, the Activity response to the newly encountered challenges has been managed well. While four out of 63 planned deliverables for Year 3 are assessed as being at high risk of non-completion, the work on remaining deliverables was initiated and/or will be continued remotely or in-person, with application of all safety measures, or was postponed for later in Year 3 or early in Year 4. 24 | FINRA PERFORMANCE EVALUATION USAID.GOV EVALUATION QUESTION 3: TO WHAT EXTENT DO STAKEHOLDERS AND BENEFICIARIES PERCEIVE PGES ARE ABLE TO ADVANCE COMPLIANCE WITH THE INTERNATIONAL REGULATORY REQUIREMENTS IN BANKING SUPERVISION, DEPOSIT INSURANCE, AND CENTRAL BANKING COORDINATION ACTIVITIES WITHOUT USAID ASSISTANCE? Finding 31. Beneficiary institutions perceive that their progress towards compliance with international standards would be much slower without FINRA. During six out of 30 relevant KIIs it was emphasized that, even if FINRA never happened, the PGEs would have had to work on implementing the IMF and EU requirements. However, the KIs in these interviews believe the progress would have been much slower without FINRA. In seven out of 14 KIIs with the PGEs, the KIs expressed satisfaction with the progress towards compliance with the relevant IMF and EU standards and requirements achieved with FINRA’s assistance. In addition, during five out of 14 KIIs with PGEs, it was noted that FINRA is among the best international projects they participated in due to focus on the sector priorities, flexible approach, and understanding of limited PGEs’ absorption capacities. Finding 32. Most KIs think additional FINRA support is needed, specifically with regards to compliance with the EU regulatory framework. Although PGE capacities are perceived to have improved with FINRA’s assistance (see Exhibit 8), during eight out of 11 KIIs with the PGEs we were told that the PGEs still lack capacity to independently continue their progress towards compliance with the relevant international standards and requirements. More broadly, in 17 out of 20 KIIs during which further FINRA support was discussed, it was confirmed that continuous assistance to progress towards compliance with international standards would be beneficial, especially concerning the continuously evolving EU regulations. EXHIBIT 8. PGEs REPRESENTATIVES’ PERCEPTION ABOUT IMPROVED CAPACITIES OF CBBH, DIA, FBA, AND BARS WITH FINRA’S ASSISTANCE FINRA supported PGEs in their efforts to mitigate the economic and financial impacts of COVID￾19, mainly by advising on appropriateness of potential response measures or applicability of EU practices. To date, the success of PGEs’ response to challenges encountered due to COVID-19 pandemic is best demonstrated by the preservation of the stability of the monetary and financial system. Importantly, additional advisory support by FINRA further reinforces already high level of trust between the PGEs and the Activity. “We really consider FINRA as our great support. So far from all the projects we've had, I can tell that the most specific assistance came from FINRA. We're really happy with FINRA’s approach so far.” PGE 25 | FINRA PERFORMANCE EVALUATION USAID.GOV Finding 33. USAID is perceived to be among most relevant international organizations supporting the BiH financial sector development. During 17 KIIs in which USAID engagement has been discussed, KIs regularly emphasized that the Agency is one of the key international organizations that supported the strengthening of the financial sector ever since the end of the 1995 war. This view is shared by all KIs from the PGEs, as well as three international organizations that discussed the USAID role. Some of the most highly-valued Agency contributions recognized during these KIIs include support to establishing the DIA and elimination of the Payment Bureaus, as well as advisory and financial assistance to financial sector institutions through the USAID/BiH’s PARE, regional Partners for Financial Stability (PFS) Program, and FINRA. Finding 34. Despite significant improvement in BiH banking regulation facilitated by FINRA, there are still substantial and unique shortcomings in the BiH regulatory environment. When asked to name financial sector issues that should be prioritized and addressed as soon as possible, KIs provided a wide range of responses. The most frequently mentioned recommendations include:  further improvement in harmonization with the EU financial regulatory standards;  capacity building of local employees in the financial sector;  financial education of the general population (e.g., money transfer, money laundering);  adoption of the Supervisory Review and Evaluation Process (SREP) methodologies;11  establishment of a Restructuring Fund;12 and  expanding access to finance, with special emphasis on non-banking finance.13 A full list of recommendations for future USAID work in the financial sector is provided in Annex XI. In addition, during the discussion with the KIs, the evaluation team identified the lack of ‘a lender of last resort’ to be a unique challenge for the BiH’s financial stability, especially in a hypothetical case of an imported crisis of confidence. Domestically owned banks are small enough to be liquidated and their deposit covered by the existent deposit insurance scheme, and therefore do not represent systemic risk. Large banks are members of international groups, and their owners can effortlessly support them in the case of a liquidity shortage. However, an issue may arise if the owner of a large BiH bank experiences a crisis in the home market, thereby affecting the credibility 11 Development and adoption of SREP methodology is facilitated by the World Bank through its BSSP program. FINRA assisted the banking agencies to develop and adopt the draft Internal Liquidity Adequacy Assessment Process (ILAAP) regulation, an important SREP area that was not covered by the World Bank BSSP. FINRA contributions on ILAAP were incorporated by the BSSP into its SREP documentation. 12 According to one KI, the World Bank Group has already tried to push forward the establishment of a Restructuring Fund, however, with no success mainly due to the lack of political consent. 13 Strengthening access to non-banking finance has been recognized as a potential area for further USAID assistance by the performance evaluation of USAID/BiH’s PARE as well. “The help they got from USAID early on was critical and there were two elements of the USAID support then: (i) they had very good technical advisors and (ii) they stayed in BiH for a long period, they were there for several years and so they got to know the people and the issues…and it’s that continuity that USAID provided that I didn’t see anybody else providing. In 1997 there was 76 registered banks in the country of 4 million people, but there was one of them that operated within the whole country and total balance sheet was 1 billion Deutsch marks in 76 banks. So, there were many banks but no banking system since the citizens didn’t use it. The technical advice that was given to the banking agencies was all done by USAID. And they transformed the system. So, USAID did a tremendous job and the fact that the banking system looks totally different today, it looks European, it’s sound and it’s such a huge change and USAID claim the major role there.” FINRA Consultant 26 | FINRA PERFORMANCE EVALUATION USAID.GOV of its BiH subsidiary. In case (i) the resolution of the affected bank and group is not adequately prepared, (ii) the resolution is not efficiently conducted, or (iii) the resolved bank does not commit to supporting its BiH subsidiary (as BiH is not integrated in EU resolution planning), a ‘lender of last resort’ would be the only solution left to support the stressed bank since other lenders would probably not be able or willing to step in. If this hypothetical case actually occurred, the bank could fail and it might profoundly affect the business of their clients and other institutions, thereby causing severe jeopardy for the BiH financial stability. CONCLUSIONS Beneficiary institutions' capacities have improved; however, they still recognize the need for international support in advancing towards compliance with relevant international standards and requirements, and particularly to continuously evolving EU regulations. Considering the USAID long-lasting engagement in the BiH financial sector and its role to date, the Agency is recognized as an important partner that can provide substantial contribution to further sector development. This reputation built not just among the PGEs, but also through cooperative relationship with international financial institutions, could be used to catalyze solutions for the priority issues that still cause difficulties to the financial sector. While the EC gap analysis, anticipated for late 2020, is expected to provide a clear picture on the status of BiH banking supervision compliance when compared to the EU practices, evaluation KIs identified a number of shortcomings that represent potential risks to the BiH regulatory environment. Some of these concerns are already being addressed by BiH institutions with support of the international financial institutions, while there still is no clear roadmap for resolving some very important issues, including a lack of a Restructuring Fund and absence of the ‘lender of last resort’ function. Both of these are complex issues with no simple and fast solutions, not just on account of the unique financial sector environment, but also because their resolution requires successful political dialogue and achieving consensus. 27 | FINRA PERFORMANCE EVALUATION USAID.GOV RECOMMENDATIONS The following recommendations are based on the evaluation findings and conclusions. These recommendations are devised to facilitate adoption and dissemination of CLA practices within the Mission and USAID/BiH should consider their application to the ongoing and future interventions whenever possible: MEL PRACTICES 1. Replicate FINRA’s model of engaging local stakeholders in assessing the Activity progress in other USAID/BiH interventions. The evaluation team found that this practice has already been encouraged among USAID’s existing portfolio of activities as well as those in the pipeline. However, the evaluation team still wishes to emphasize the need for facilitating the CLA practices that aim to improve MEL reporting. FINRA can serve as an example of good practice for bolstering local ownership and sustainability of the Activity results through the engagement of relevant stakeholders in assessing the Activity progress, which in turn provides a sound basis for fostering the country’s self-reliance. 2. Introduce the practice of providing feedback on final assessment results to key stakeholders to further strengthen local ownership and results sustainability and promote greater learning among the stakeholders. Active stakeholder engagement could be further developed by ensuring that feedback on final assessment results is systematically shared with key stakeholders. Such an approach has potential to promote collaboration and disseminate new knowledge among the stakeholders. The evaluation team also recognized this approach as a practice effective in further reinforcing local ownership and sustainability of the Activity results, which are of paramount importance for the country’s progress in its journey to self-reliance. 3. Consider modalities aimed to enhance measurability of the Activity’s performance in the remaining years of the implementation. At the end of Year 2 of its five-year contract period, the Activity exceeded or was close to exceeding its LoA performance indicators targets. For the purpose of the Activity’s further performance monitoring, USAID/BiH should consider modalities that would reflect FINRA’s real implementation potential and derive maximum benefits from the Activity’s sound design and the IP’s noteworthy implementation competence. Possible approaches include broadening Activity’s scope of work and/or revising the Activity objectives and/or targets. While potential contractual implications necessarily transcend the scope of this report, modifications along these lines may re-establish benchmarks for desired achievements in the remaining years of implementation and ensure continuity of guidance, permit monitoring, and maximize benefits for the beneficiaries, the IP’s success, and USAID’s achievement of results. For example, USAID/BiH could use the findings of the gap analysis that the EC is currently conducting to inform the focus of the Activity’s efforts in the coming years. In addition, in case of a MEL Plan modification, USAID/BiH should pay attention to ensuring the alignment with the J2SR agenda and take into account expected ramifications of the COVID-19 pandemic. Lastly, USAID/BiH should consider instructing the Activity to disaggregate its data on “Number of staff of Counterpart agencies and financial institutions trained with FINRA support” by sex to ensure full compliance with the USAID ADS requirements. 28 | FINRA PERFORMANCE EVALUATION USAID.GOV ACTIVITY DESIGN 4. Capitalize on FINRA’s model and expand application of its approach to consultation with key stakeholders to facilitate the Activity’s ability to respond to the main challenges in the subject area. Consultation with relevant stakeholders during the Activity design has been recognized as a key step in defining the adequate Activity scope of work, since reliance on relevant international reports that provide specific and in-depth regulatory recommendations and guidance is a unique rather than regular opportunity. Still, this approach has created for FINRA stable grounds for (i) developing partner relationship with the PGEs, therefore strengthening their commitment to the Activity implementation, and (ii) close collaboration with other international organizations active in the financial sector, which is recognized as essential for effective donor coordination. 5. Review the suitability of Activity design that combines heterogeneous areas. USAID/BiH should consider appropriateness of Activity design that incorporates work on widely disparate while very demanding thematic areas, such as the financial and fiscal sector, from the standpoint of potential effects of increased complexity on efficiency and effectiveness as well as possible added complications for resource allocation. USAID/BiH should also consider providing a more flexible timeframe for execution of planned interventions within its Activities to provide reasonable space for successful implementation of these interventions and increases likelihood of an effective response to possible challenges in the implementation. ACTIVITY IMPLEMENTATION 6. Adapt FINRA’s approach to donor coordination for use in future USAID interventions. The Mission’s strategic donor coordination efforts, characterized by regular communication and close relationships with other international organizations active in the financial sector, were validated by the successful outcomes of the FINRA preparation and design stages. This strategic approach was suitably complemented by operational coordination with the most active donor counterparts practiced by FINRA during its implementation period to date, with improved information sharing, prevention of overlaps in assistance interventions, and resulting in generally better-balanced international efforts in the financial sector. As USAID strives to allocate its resources most effectively, the Mission should consider adapting FINRA’s approach to donor coordination for application in other interventions as well. 7. Transfer a part of the unallocated FINRA’s Grant Fund to technical assistance which is in high demand and greatly valued by the beneficiary institutions. Since it is reasonable to conclude that FINRA’s Grant Fund budget will not be fully disbursed by the end of the Activity, USAID should consider transferring a portion of the Grant Fund budget to technical assistance which has been in considerably higher demand to date. This is well aligned with the Activity’s request to the Mission for adjusting the allocation of resources within the Activity total budget as well as with KIs’ perception about the usefulness of the technical assistance provided to date. 29 | FINRA PERFORMANCE EVALUATION USAID.GOV 8. Take into account challenges caused by the COVID-19 pandemic in planning and adjusting ongoing technical assistance and future interventions in the financial sector. The COVID-19 pandemic and its related economic crisis are recognized as the main current challenge for the financial sector. Although the monetary and financial system remain stable, the crisis continues, and its final implications remain uncertain and difficult to gauge. USAID/BiH should carefully assess potential consequences and consider those in planning or adjusting its future assistance. This should apply to the proposed revision of the FINRA MEL Plan, as suggested in Recommendation 3. USAID/BiH should also consider incorporating increased flexibility into assistance provided through various Activities to ensure its interventions can accommodate and adjust to any external shocks that are beyond the individual Activity or Mission’s control. FUTURE ASSISTANCE TO THE BIH FINANCIAL SECTOR 9. Provide continuing support to the BiH banking regulation institutions to preserve and further reinforce financial stability in BiH. Considering its reputation and trust it enjoys with both PGEs and international organizations, USAID should examine possibilities for facilitating further financial sector development in BiH. Although it is among the most developed sectors in the country, the financial sector still has shortcomings. USAID should consider using its public image and respect, accumulated knowledge, and proved-to-work practices in donor coordination and working with government institutions, to address the identified shortcomings and therefore accelerate BiH’s journey to self-reliance. 10. Consider mounting an effort to start or reopen the discussion about burning financial sector issues, including:  lack of the ‘lender of last resort’ function;  lack of a government backstop for the deposit insurance scheme or failing banks;  incomplete resolution framework; and  failure to define the list of strategically important banks. Since these are all issues emerging from present political and legal arrangements in BiH, they cannot be addressed through communication with the PGEs only. A wider circle of participants should be included in the discussion and make aware of the possible consequences of neglecting those issues. Addressing the burning financial sector issues would require facilitation of political dialogue, discussion, and persuasion to motivate them to open the discussion and move forward with further steps to build on the already achieved and encouraging level of financial stability, create a sound and healthy financial environment, and help prepare BiH for events that are low probability but may have severe consequences. At this point in time, it is not possible to propose a roadmap for improvement in these areas. It can only emerge as a consequence of the open and honest political discussion between decision makers in BiH. The strong reputation FINRA and USAID built in BiH, clearly ascertained in the course of this evaluation, could be a strong asset to successfully promote and encourage such a dialogue in cooperation with other international partners. 30 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEXES ANNEX I: STATEMENT OF WORK FOR PERFORMANCE EVALUATION OF THE USAID FINANCIAL REFORM AGENDA ACTIVITY (FINRA) IN BOSNIA AND HERZEGOVINA INTRODUCTION The United States Agency for International Development Mission in Bosnia and Herzegovina (USAID/BiH) has requested its Monitoring and Evaluation Support Activity (MEASURE II) to conduct a performance evaluation of the USAID/BiH’s Financial Reform Agenda Activity (FINRA). FINRA, implemented by Financial Markets International, Inc., started in September 2017 and is envisaged as a three-year intervention with the possibility of a two-year extension. The purpose of this performance evaluation is to assess FINRA’s progress towards achievement of its contractual obligations to date and inform any future USAID/BiH interventions in financial sector regulatory compliance in BiH. Exhibit 1 presents basic information about FINRA. Exhibit 9. Basic Information on the Financial Reform Agenda Activity (FINRA) Activity/Project Name Financial Reform Agenda Activity (FINRA) Contractor Financial Markets International, Inc. (FMI) Contract # AID-168-C-17-00003 Total Estimated Cost $5,999,136 Life of Activity September 29, 2017 – September 28, 2022 (assuming its two-year extension period option is exercised) Active Geographic Regions All of BiH Mission Development Objective (DO)/Intermediate Result (IR) DO 2: A competitive, market-oriented economy providing better economic opportunities for all its citizens IR 2.2: Improved economic aspects of governance relevant to business activity COUNTRY CONTEXT Maintaining financial stability is recognized as a key prerequisite for facilitating accelerated private sector growth and countering high unemployment in BiH, and therefore essential for fostering the country’s journey to self-reliance (J2SR). It is also of paramount importance for country progress in European integration processes. However, financial sector development and private sector access to finance in BiH face a number of issues that contribute to the vulnerability of the overall financial system. These issues are captured by the low score on Economic Policy sub-dimension in in BiH’s J2SR Roadmap.14 Specifically, financial sector development is hindered by the complex political situation and frequent turmoil that slow down and/or impede policy implementation and inter￾governmental cooperation that are required for effective operation of the financial system.15 The 14 USAID. 2020. Bosnia and Herzegovina – Journey to Self-Reliance: FY2020 Country Roadmap. Available at: https://selfreliance.usaid.gov/country/bosnia-and-herzegovina 15 IMF. 2018. Staff Report on Request for Extended Arrangement under the Extended Fund Facility (EFF). Available at: https://www.imf.org/en/Publications/CR/Issues/2018/02/13/Bosnia-and-Herzegovina-2017-Article-IV-Consultation-First￾Review-Under-the-Extended-45624 31 | FINRA PERFORMANCE EVALUATION USAID.GOV banking system, which constitutes a major segment of the financial arena, is adequately capitalized and liquid.16 In 2018, the banking sector’s assets as a share of the nominal gross domestic product (GDP) reached 91.6 percent.17 However, bank profitability in BiH is the lowest among the neighboring countries due to the fragmented market and high regulatory costs.18 According to the 2018 financial soundness indicators , published by the CBBH, profitability is the only declining segment of BiH’s financial system’s resilience to risks. While the rating in the asset quality segment remained unchanged when compared to the 2017, capitalization, liquidity, and foreign exchange risk indicators are improving.19 However, credit risk remains the BiH banking sector’s dominant risk.20 High non-performing loans (NPL), especially the segment of loans to legal entities, threaten to endanger the solvency of vulnerable banks, and prevent raising fresh capital.21 The share of NPLs as a percentage of total loans at the end of 2018 reached 8.8 percent and declined 1.3 percentage points compared to 2017.22 Nevertheless, an overhaul of the regulatory and legislative framework for NPL resolution is an essential step towards achieving financial stability. Although there has been some improvement in modernizing and harmonizing banking sector legislation, these reforms are still incomplete. For example, the Law on Deposit Insurance has still not been adopted, which impedes the use of the deposit insurance fund (DIF) in cases of bank failure, therefore, undermining the systemic risk oversight.23 According to a top-down stress test conducted by the CBBH in 2018, the resilience of the BiH banking sector is based on the high initial level of capital adequacy, current profitability of banks, and significant resilience to extreme shocks of the two largest banks in the system, which account for 35 percent of the total banking sector assets.24 The share of the non-banking sector as a portion of the overall financial sector reached a modest 11.5 percent in 2018, with insurance and reinsurance being its largest segments.25 Among other non￾banking financial institutions, microcredit and leasing sector profitability increased, while the total turnover of the two stock exchanges in BiH continued to decrease in 2018, for the third consecutive year.26 Following the adoption of the European Union’s (EU’s) Reform Agenda (RA) for BiH 2015-2018 in June 2015, BiH governments (BiH Council of Ministers (CoM), Federation BiH Government, and Republika Srpska (RS) Government) adopted detailed short-term action plans with specific financial sector measures. The IMF’s Extended Fund Facility (EFF) EUR 550 million agreement, which was approved in September 2016 and was crucial for mobilizing continued international financial 16 Ibid. 17 CBBH. 2018. Financial Stability Report. Available at: https://www.cbbh.ba/Content/Archive/575 18 IMF. 2018. Staff Report on Request for Extended Arrangement under the Extended Fund Facility (EFF). Available at: https://www.imf.org/en/Publications/CR/Issues/2018/02/13/Bosnia-and-Herzegovina-2017-Article-IV-Consultation-First￾Review-Under-the-Extended-45624 19 CBBH. 2018. Financial Stability Report. Available at: https://www.cbbh.ba/Content/Archive/575 20 Ibid. 21 IMF. 2018. Staff Report on Request for Extended Arrangement under the Extended Fund Facility (EFF). Available at: https://www.imf.org/en/Publications/CR/Issues/2018/02/13/Bosnia-and-Herzegovina-2017-Article-IV-Consultation-First￾Review-Under-the-Extended-45624 22 CBBH. 2018. Financial Stability Report. Available at: https://www.cbbh.ba/Content/Archive/575 23 IMF. 2018. Staff Report on Request for Extended Arrangement under the Extended Fund Facility (EFF). Available at: https://www.imf.org/en/Publications/CR/Issues/2018/02/13/Bosnia-and-Herzegovina-2017-Article-IV-Consultation-First￾Review-Under-the-Extended-45624 24 CBBH. 2018. Financial Stability Report. Available at: https://www.cbbh.ba/Content/Archive/575 25 Ibid. 26 Ibid. 32 | FINRA PERFORMANCE EVALUATION USAID.GOV assistance for BiH, was another key financial and policy lever for the RA implementation. However, BiH has made little progress in meeting the EU accession criteria for economic readiness and further progress in advancing financial stability remains a priority.27 BACKGROUND INFORMATION ABOUT THE FINANCIAL REFORM AGENDA ACTIVITY (FINRA) The general purpose of FINRA is to enhance the stability of the BiH financial sector. Specifically, FINRA’s assistance focuses on supporting financial sector reforms documented in the IMF’s EFF Letters of Intent, IMF Financial Sector Assessment Program (FSAP), the IMF/World Bank (WB) Financial Sector Stability Assessment (FSSA), and the EU Reform Agenda. Meeting policy benchmarks defined by these documents are expected to assist BiH to advance toward compliance with the EU accession requirements, therefore strengthening its European integration prospects. FINRA is a continuation of USAID/BiH’s assistance provided for supervision of the BiH financial sector over the past two decades. Specifically, the Activity contributes to USAID/BiH’s Country Development Cooperation Strategy (CDCS) 2012-2020 Development Objective (DO) 2: BiH is a competitive, market-oriented economy providing better economic opportunities for all its citizens, and in particular to Intermediate Result (IR) 2.2: Improved economic aspects of governance relevant to business activity. FINRA’s assistance is organized into three overarching components:  Banking Supervision and Insurance This Activity component entails assistance related to two functional areas of FINRA’s work, bank safety and soundness and problem bank management. Specifically, FINRA supports ongoing supervision of financial institutions to encourage their responsible governance and risk management. This will help ensure that the financial system is stable, provides access to finance for worthy borrowers, and protects depositors. FINRA also focuses on preparing supervisors to deal with individual banks identified as failing or at risk of insolvency, and on preventing systemic problems, including forestalling depletion of deposit insurance funds, or the need for government bailouts.  Central Banking FINRA’s activities under this component include capacity building efforts and assistance to align the work of the CBBH with EU requirements and best practices. This component also envisages assistance to advance the CBBH compliance with the IMF’s FSAP recommendations.  Audit of Public Financial Resources and Quality of Government Spending Under this component, FINRA provides assistance with public internal financial control strategies and procedures, strengthening internal audit functions in public sector institutions, and supporting government spending performance auditing by supreme audit institutions. This is a separately priced Contract Option (CLIN 0003) that the Activity, based on the Mission’s approval, began implementing in year 2. FINRA’s assistance is designed to address: i) the need for assistance in drafting the implementing standards (formal policies, laws, bylaws, directions, guidelines, and implementing procedures), ii) the need for assistance in improving supervisory institutional practices and tools, and iii) the need for assistance in building institutional capacities of BiH financial institutions/entities. FINRA’s key implementing partner government entities (PGEs) include key financial sector regulators in BiH: the 27 European Commission (EC). 2019. Commission Opinion on BiH's application for membership of the EU. Available at: https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/20190529-bosnia-and-herzegovina-opinion.pdf 33 | FINRA PERFORMANCE EVALUATION USAID.GOV CBBH, Deposit Insurance Agency (DIA), FBiH Banking Agency (FBA), and the Banking Agency of the RS (BARS). PLANNED AND ACHIEVED RESULTS Early during its implementation, FINRA developed a close working relationship with the PGEs. Specifically, every FINRA intervention was developed in collaboration and formally approved by the PGEs. Besides, FINRA established a working relationship with the IMF to ensure its efforts advanced the IMF’s priorities. Relationships were also established with the World Bank Group, European Bank for Reconstruction and Development (EBRD), and Swiss State Secretariat for Economic Affairs (SECO), to make sure the Activity’s interventions are complementary to the efforts of these international organizations. According to the Activity contract, a precondition for FINRA’s work is the adoption of regulatory frameworks, specifically the Federation Law on Banks, the Republika Srpska Law on Banks, the Law on Banking Agencies for both entities, and the Law on Deposit Insurance. While four of these five laws were adopted by the time FINRA started, the Law on Deposit Insurance was only passed in June 2020. FINRA operationalizes its work through task areas grouped according to specific Activity functional components. During the first two years of its implementation, FINRA developed scopes of work (SoWs) for 64 task areas. Of those, 28 are completed and 36 are still in the process. When disaggregated by the type of assistance provided across the 64 task areas, FINRA delivered 82 trainings, drafted 73 regulatory standards, and submitted 71 documented institutional operating improvements. FINRA’s Activity goal reflects USAID/BiH’s Economic Growth Project goal: a competitive, market￾oriented economy providing better economic opportunities for all of its citizens. Indicators that the Activity tracks to measure its progress towards the Activity goal are sourced from secondary data, and include: 1) nonperforming loans (NPLs) of commercial banks as a percent of their total gross loans28, and 2) domestic credit to the private sector as a percent of GDP.29 As defined in the Activity contract, FINRA’s purpose is improved financial sector stability. The Activity measures its progress towards achieving the life of Activity (LOA) target for improved financial sector stability by tracking the number of IMF/EU Financial Stability Compliance Requirements (CRs) significantly advanced toward fulfillment with its assistance. CRs refer to 20 provisions identified in nine relevant secondary sourced documents that address BiH financial stability. The list of CRs is provided in Exhibit 2. Exhibit 10. List of Compliance Requirements (CRs) Relevant for FINRA # Compliance Requirement Agencies Bank Safety and Soundness 1 Strengthen oversight of banks by the entity banking agencies FBA, BARS 2 Assure that banks continue working toward full compliance with the International Financial Reporting Standards (IFRS) FBA, BARS 3 Encourage conservative assumptions by banks; issue prudential guidance to promote adequate provisioning FBA, BARS 28 Sourced from the CBBH statistical reports 29 Sourced from the World Bank Databank 34 | FINRA PERFORMANCE EVALUATION USAID.GOV # Compliance Requirement Agencies 4 Implement the liquidity coverage ratio (LCR); fine tune parameters imbedded in the LCR and Net Stable Funding Ratio (NSFR) as to inclusion of “required reserves” in High-Quality Liquid Assets (HQLA); revise the liquidity ratio FBA, BARS 5 Add macroprudential analysis; select and calibrate the appropriate instruments to address systemic risk FBA, BARS 6 Identify ultimate beneficiary owners; enhance disclosure on group structure of banks and insider lending; transfer of significant ownership, major acquisitions FBA, BARS 7 Deepen the assessment of fit and proper FBA, BARS 8 Improve stress testing; conduct bottom-up stress tests at least once per year FBA, BARS Problem Bank Management 9 Adopt amendments to the Law on Deposit Insurance; adapt regulation DIA 10 Require all banks to have recovery plans; firm and timely action to deal with problem banks; develop a broader range of measures to expedite corrective action FBA, BARS 11 Establish resolution plans for all Systemically Important Banks (SIBs); facilitate restructurings and debt resolution as well as adoption of out-of-court restructuring guidelines FBA, BARS 12 Develop banking agency organizational structures emanating from the new bank resolution mandate FBA, BARS 13 Develop cross-border bank crisis resolution coordination arrangements for all SIBs FBA, BARS 14 Broaden the scope to draw on the DIF for funding bank resolution; assess the availability of suitable backup facilities given DIA’s potential expanded role DIA 15 Enable prompt depositor payout DIA Central Banking 16 Strengthen cooperation, coordination and information exchange among regulatory agencies, strengthen cooperation among financial safety net players under the Standing Committee for Financial Stability (SCFS) and Coordination for Banking Supervision group (CBS) CBBH, FBA, BARS, DIA 17 Improve national-level oversight of systemic risks; define the tools to be used for the identification of systemic risk define and collect additional information for the assessment of vulnerabilities CBBH 18 Improve the scope and exchange of information on key variables important for systemic liquidity 19 Strengthen, regularly review, and test bank contingency plans of SCFS members to harmonize domestic cooperation and information exchange; conduct crisis simulation exercise to test cooperation DIA, FBA, BARS, CBBH, MoFs 20 Establish a separate risk management function, independent of operational units, with a direct reporting line to the Governing Board CBBH To measure the number of CRs that substantially advanced toward fulfillment with its assistance, FINRA organizes Annual Strategic Review meetings that are attended by five CBEs and IMF. During this annual meeting, CBEs and IMF representatives fill out evaluation forms and assess progress on the CRs applicable to them. The indicator targets were met during the first year of FINRA implementation and surpassed in the second. FINRA is also tracking three outcome/output indicators informed by the Activity records or counterpart publications. Exhibit 3 provides FINRA’s tracking table (TT) in which these indicators 35 | FINRA PERFORMANCE EVALUATION USAID.GOV are listed and attached to specific results. The TT also presents annual and LOA targets for all indicators, as well as the actuals for the first two years of the Activity implementation. Exhibit 11. Activity Indicators, with Targets and Actuals for the FY2018 and FY2019, and Life of Activity Targets (LOA) Level of result Narrative Summary Indicators Targets (Actuals) Life of Activity Targets FY 2018 FY 2019 Activity Goal A competitive, market￾oriented economy providing better economic opportunities for all of its citizens Nonperforming loans (NPLs) of commercial banks as a percent of their total gross loans 12.1% (10.0%) 11.9% (8.9%) 11.3% Domestic credit to the private sector as a percent of GDP 56.7% (54.1%) 56.9% (58.6%) 57.5% Activity Purpose Improved financial sector stability Number of IMF/EU Financial Stability Compliance Requirements significantly advanced toward fulfillment with FINRA assistance* 5 (5) 10 (16) 18 Activity Outcome/ Output 1 Policies, laws, regulations, and implementing procedures (“Standards”) are drafted for adoption in the financial, audit and quality of government spending areas Number of Standards addressing IMF/EU Compliance Requirements and other areas of financial stability (i) drafted with FINRA assistance and (ii) adopted by BiH authorities* 10/8 (37/30) 20/15 (36/18) 65/53 Activity Outcome/ Output 2 Strengthened managerial organization and operational practices, procedures, and tools (“Institutional Improvements”) of Counterparts Number of Institutional Improvements documented by FINRA and proposed to Counterpart government agencies 4 (27) 20 (44) 49 Activity Outcome/ Output 3 Strengthened professional capacities of Counterpart agencies and financial institution staff Number of staff of Counterpart agencies and financial institutions trained with FINRA support 50 (120) 70 (124) 140 * Project indicator EVALUATION PURPOSE, QUESTIONS, DESIGN, AND LIMITATIONS The main purpose of this performance evaluation is to assess FINRA’s progress towards achievement of its contractual obligations to date. Specifically, the evaluation aims to generate the knowledge within the Mission about the magnitude and determinants of FINRA’s performance and accrue evidence on best approaches to facilitate programming in the financial sector regulatory compliance. Evaluation findings, conclusions, and recommendations are expected to cultivate collaboration, learning, and adapting (CLA) within the Mission, as well as among other evaluation stakeholders. For the implementing partner (IP), the evaluation’s findings will offer a valuable independent reflection on approaches and success in fulfilling the Activity’s goals that inform improvements in their future efforts. The governments in BiH and their relevant agencies will receive independent insight into their progress towards meeting the IMF’s and EU policy benchmarks for the financial sector that are integral for the European integration process. The European Union (EU) and leading international financial institutions (IFIs), notably the WBG and IMF, whose priorities and conditions 36 | FINRA PERFORMANCE EVALUATION USAID.GOV provided the core basis for the design of the Activity, will benefit from the evaluation by receiving an external overview of progress in the financial sector reforms that they are contributing to. EVALUATION QUESTIONS The FINRA performance evaluation will answer the following evaluation questions (EQs): EQ1: What progress has been accomplished in reaching contract targets? EQ2: How is FINRA assistance perceived by beneficiaries and other stakeholders from the standpoint of its design and implementation? EQ2a: How has FINRA assisted its beneficiaries in responding to the potential economic fallout caused by the COVID-19 pandemic and how do beneficiaries perceive this assistance? EQ3: To what extent do stakeholders and beneficiaries perceive PGEs are able to advance compliance with the international regulatory requirements in banking supervision, deposit insurance, and central banking coordination activities without USAID assistance? DESIGN AND METHODOLOGY The evaluation design will employ a mixed method approach and will triangulate data across the following sources to answer the evaluation questions. The proposed methodology will ensure systematic and efficient collection of data. The evaluation team will make use of the following data sources:  FINRA’s design and implementation documentation and databases, including Activity contract, work plans, quarterly and annual reports, MEL documentation, Activity deliverables, and any other documentation and databases of FINRA.  Secondary documentation relevant to the financial sector, including documentation from international organizations, such as the EU, IMF and World Bank Group, as well as documentation from relevant BiH government/public institutions.  Key informant interviews (KIIs) with USAID/BiH, the FINRA IP, relevant international and donor organizations, government agencies and institutions, and other stakeholders. The full list of key informants and the semi-structured interview guide will be presented in the evaluation work plan and subject to USAID/BiH comments.  Online survey of FINRA’s beneficiaries/stakeholders. A draft survey questionnaire will be included in the evaluation work plan and subject to USAID/BiH comments. The evaluation team will have access to relevant FINRA documentation, including award documents, work plans, quarterly reports, annual reports, MEL documentation, databases, and other relevant documents and deliverables. Exhibit 4 presents the Evaluation Matrix, which details the methodological approach to be used to answer each of the three evaluation questions. The evaluation team will use Activity and secondary documentation, as well as primary data collected through KIIs and an online survey to gather and triangulate information and best inform the evaluation findings, conclusions, and recommendations. The evaluation team will review the Activity’s documentation; record, transcribe, and code the KIIs; analyze and compare the survey data across different groups of Activity stakeholders; review the secondary data and compare them against the primary data. 37 | FINRA PERFORMANCE EVALUATION USAID.GOV Exhibit 12. Evaluation Matrix EVALUATION LIMITATIONS Potential limitations of this evaluation include:  Possible overstatement of positive effects by the Activity PGEs: the evaluation team will triangulate data across multiple data sources to verify the credibility of results.  Effects of the FINRA interventions on the financial sector’s performance and economy as a whole may be difficult to separate from the influence of larger market and environmental forces (such as the economic crisis triggered by the COVID-19 pandemic).  Possible inability to conduct in-person KIIs due to the COVID-19 pandemic and related safety measures: the evaluation team will mitigate this limitation by leveraging remote rather than in￾person interviews; the evaluation team will use phone and/or online tools to meet with the KIs depending on the KIs’ preferred or available means of communication. EVALUATION IMPLEMENTATION DELIVERABLES AND SCHEDULE All deliverables will be submitted electronically and in English. The deliverables will include: 1. Detailed evaluation work plan and data collection instrument(s) The evaluation work plan will include: (1) a detailed evaluation design matrix (including the key questions, methods, and data sources used to address each question and the data analysis plan for each question); (2) draft data collection instruments (interview guides and questionnaires) (3) the list of potential interviewees; (4) known limitations to the evaluation design; (5) a dissemination plan; (6) the anticipated schedule and logistical arrangements; and EVALUATION QUESTION (EQ) DATA SOURCES/DATA COLLECTION METHODS RESEARCH DESIGN What progress has been accomplished in reaching contract targets? 1. Activity documentation (MEL Plan and progress reports) 2. Secondary documentation (government and international community reports and documents) 3. KIIs 4. Online survey of FINRA’s beneficiaries/stakeholders Mixed methods How is FINRA assistance perceived by beneficiaries and other stakeholders from the standpoint of its design and implementation? How has FINRA assisted its beneficiaries to respond to the potential economic fallout caused by the COVID-19 pandemic and how do beneficiaries perceive this assistance? 1. Activity documentation (progress reports) 2. Secondary documentation (government and international community reports and documents) 3. KIIs 4. Online survey of FINRA’s beneficiaries/stakeholders Mixed methods To what extent do stakeholders and beneficiaries perceive PGEs are able to advance compliance with the international regulatory requirements in banking supervision, deposit insurance, and central banking coordination activities without USAID assistance? 1. Activity documentation (progress reports) 2. Secondary documentation (government and international community reports and documents) 3. KIIs 4. Online survey of FINRA’s beneficiaries/stakeholders Mixed methods 38 | FINRA PERFORMANCE EVALUATION USAID.GOV (7) a list of the members of the evaluation team, with their respective roles and responsibilities. 2. Presentation of preliminary findings A presentation of preliminary findings to USAID/BiH will include a summary of preliminary findings and recommendations to USAID/BiH. 3. Draft evaluation report The draft evaluation report will be consistent with the USAID Evaluation Report Requirements (ADS REFERENCE 201MAH, at https://www.usaid.gov/ads/policy/200/201mah), USAID’s evaluation policy (https://www.usaid.gov/sites/default/files/documents/1870/USAIDEvaluationPolicy.pdf) and take into account criteria to ensure the quality of the evaluation report specified in ADS REFERENCE 201MAA (https://www.usaid.gov/ads/policy/200/201maa). Once the initial draft report is submitted, USAID/BiH will have 15 working days to review and comment on the initial draft and submit the consolidated comments to the research team. The research team will address the consolidated comments and submit a revised final draft report within 10 days of receiving USAID/BiH comments. 4. Final evaluation report The evaluation team will take no more than 10 calendar days to respond/incorporate the final comments from USAID/BiH. The evaluation team co-lead will then submit the final report. The final evaluation report will be up to 30 pages long, excluding any annexes. 5. Evaluation follow-up workshop Upon the Mission’s approval of the final evaluation report, MEASURE II will organize a follow-up workshop to discuss utilization of evaluation findings and conclusions, as well as application of recommendations to ongoing and/or future USAID/BiH development programming. The workshop will strengthen use of evidence and facilitate improved CLA practices for USAID/BiH and FINRA. Exhibit 5 provides a tentative evaluation schedule. Exhibit 13. Tentative evaluation schedule Tentative Dates Tasks and Deliverables June 30, 2020 Submit draft evaluation work plan to USAID/BiH July 1 to July 6, 2020 Logistical preparation, scheduling KIIs interviews, online survey preparation, piloting data collection instruments July 6 to July 24, 2020 Data collection through KIIs and online survey Interview transcription Initial data analysis Review of Activity documentation Review of secondary data July 27, 2020 Recommendations’ discussion workshop with USAID/BiH (if needed) July 27 to August 7, 2020 Continue and finalize transcribing interviews Continue and finalize data analysis Report drafting 39 | FINRA PERFORMANCE EVALUATION USAID.GOV August 7, 2020 Presentation to USAID/BiH to discuss the preliminary findings and recommendations August 14, 2020 Submit Draft Evaluation Report to USAID 10 days upon receiving USAID/BiH comments on the draft report Submit Final Evaluation Report to USAID September 15, 2020 Evaluation Follow-Up Workshop TENTATIVE TEAM COMPOSITION AND KEY QUALIFICATIONS The evaluation team is expected to include six members. The tentative team composition and team members’ key qualifications are illustrated in Exhibit 6. Exhibit 14. Team Composition and Team Members’ Key Qualifications Position Key Qualifications Subject Matter Expert Team Co-Lead Team and project management skills; subject matter expertise in banking supervision, risk management, deposit insurance, central banking, and EU and international regulations; report drafting skills Evaluation Expert/Team Co-Lead Team and project management skills; technical expertise in evaluation methodologies, evaluation report drafting skills MEASURE Research Fellow with local financial sector expertise Data collection and data analysis skills; expertise in local regulations (banking supervision and deposit insurance or government audit, government structures, and public finances) Senior Research Analyst (SRA) Data collection and data analysis skills Two Research Assistants Organizational skills and ability to ensure smooth data collection and processing The team composition and level of effort for each team member will be elaborated in detail in the evaluation work plan. 40 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX II: LIST OF REVIEWED DOCUMENTATION ACTIVITY DOCUMENTATION: 1. Activity’s Contract no. AID-168-C-17-00003, September 27, 2017 2. Activity’s Contract modification no.: AID-168-C-17-00003/05, September 13, 2019 3. USAID FINRA Year 1 Annual Report 4. USAID FINRA Year II Annual Report 5. USAID FINRA Year I First Quarterly Report 6. USAID FINRA Year I Second Quarterly Report 7. USAID FINRA Year I Third Quarterly Report 8. USAID FINRA Year I Fourth Quarterly Report 9. USAID FINRA Year II First Quarterly Report 10. USAID FINRA Year II Second Quarterly Report 11. USAID FINRA Year II Third Quarterly Report 12. USAID FINRA Year II Fourth Quarterly Report 13. USAID FINRA Year III First Quarterly Report 14. USAID FINRA Year III Second Quarterly Report 15. USAID FINRA Year IV Third Quarterly Report 16. USAID FINRA Year I Work Plan 17. USAID FINRA Year II Work Plan 18. USAID FINRA Year III Work Plan 19. USAID FINRA Monitoring and Evaluation Plan, March 13, 2018 20. USAID FINRA Monitoring and Evaluation Plan (revised version), February 15, 2019 21. USAID FINRA Consultants Database 22. USAID FINRA Training Database 23. USAID FINRA Compliance Requirements Progress Evaluation Forms 24. FINRA Report on FINRA Activities Related to Bank Recovery, Resolution and Deposit Insurance 25. FINRA Grants Manual 26. MEERS and FMF Grants approval document from the CO 27. DIA Grants approval document from the CO 28. BARS Grants approval document from the CO 29. FINRA Grant Approval Memorandum for MEERS and FMF 30. Grant Agreement 168-019-001 (BARS) 31. Grant Agreement 168-019-002 (MEERS) 32. Grant Agreement 168-20-003 (DIA) 33. DIA Public Awareness Survey Final Report 34. FINRA workshops and seminars summaries 35. FINRA Review of Planning and Guidance for FINRA Component 3 36. Pi Consulting Assessment of BiH Needs in Quality of Government Spending 37. MEERS Registry Support Request 38. FINRA Basic elements of the potential technical assistance for Sarajevo Canton 39. FINRA Scopes of Work 40. FINRA SOWs and Outputs Tracker 41 | FINRA PERFORMANCE EVALUATION USAID.GOV OTHER RELEVANT DOCUMENTATION: 1. USAID/BiH Partnership for Advancing Reforms in the Economy (PARE) Performance Evaluation 2. IMF Financial Sector Assessment Program (FSAP) Bosnia and Herzegovina 3. CBBH Financial Stability Report 2018 4. IMF Staff Country Report 2017 Bosnia and Herzegovina 5. European Union Reform Agenda for Bosnia and Herzegovina 2015-2018 6. OECD Update on COVID-19 situation, July 2020 7. Agency for Statistics of BiH, BiH International Trade in Goods Statistics: January-June 2020 8. World Bank, Global Economic Prospects 2020 9. EBRD, Resilience to the COVID-19 shocks in the EBRD regions 42 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX III: LIST OF KEY INFORMANT INTERVIEWS # TYPE INSTITUTION NAME FUNCTION 1 DONOR USAID/BiH Dobrila Vukmanović FINRA COR 2 Erol Mujanović Deputy Director, USAID/BiH EDO Office 3 Elma Bukvić Jusić Senior Development Assistance Specialist 4 CONTRACTOR FINRA David King Chief of Party 5 Amil Kamenica Deputy Chief of Party, Grant Fund Manager 6 SUB￾CONTRACTOR Pi Consulting Emina Kadrić Director 7 Naida Čaršimamović Macro-Finance and Monitoring and Evaluation Expert 8 BENEFICIARIES CBBiH Senad Softić Governor 9 Ernadina Bajrović Vice-Governor 10 Milica Lakić Vice-Governor 11 Vesna Papić Head of Financial Stability Department 12 Belma Čolaković Chief Economist 13 Amir Hadžiomeragić Head of Department for Statistics 14 DIA Krešimir Šoljić Director 15 Gorana Krunić Coordinator for Insurance Limit Assessment and Bank Resolution 16 FBA Edvard Kotorić Director assistant 17 Seid Konjhodžić Senior Expert Associate 18 Rizvan Mehmedbašić Supervisor 19 Hrvoje Orešković Supervisor 20 BARS Danijela Nježić￾Buzadžija Head of Regulation Department 21 Ljiljana Miljević Head of the Small and Medium Supervision Department 22 Nenad Tepić Director of Restructuring Unit 23 Olivera Talijan Koprena Head of the Department of Information and Communication Technologies 24 FMF Alija Aljović Assistant Minister 25 MEERS Slobodanka Dubravac Assistant Minister 26 INTERNATIONAL ORGANIZATIONS IMF Andrew Jewell Resident Representative for Bosnia and Herzegovina 27 WB Ismael Ahmad Fontan Technical Input Lead (FINSAC) 28 EBRD Tamir Mostarac Associate Director 29 KfW Adam Drosdzol Director 43 | FINRA PERFORMANCE EVALUATION USAID.GOV 30 KfW Dželila Huremović Deputy Director 31 Graduate Institute Geneva Nikhil Ray Programme Manager 32 CONSULTANTS FINRA STTA Marko Škreb Central Banking Lead 33 Bank of Slovenia Andraž Južnič Secretary General 34 Joint Vienna Institute Adam Geršl Financial Stability Analysis Expert 35 Bank of Slovenia Tomaž Rotovnik Consultant 36 Maxima Consulting HR Čedo Maletić Senior Partner 37 Arhivanalitika Velimir Šonje Director 38 NA Peter Nicholl Independent banker 39 National Bank of Serbia Đorđe Jevtić Director of Administration for Supervision of Financial Institutions 40 Central Bank of Ireland Branko Sučević Senior Policy Specialist 41 NON￾BENEFICIARIES APRMBiH Amir Softić Executive Director 42 Amar Brkan General Secretary 43 UBBiH Berislav Kutle Director 44 Nova Banka Srđan Kondić Director 44 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX IV: KEY INFORMANT INTERVIEW GUIDES INTRODUCTION The interview guides are intended to serve as semi-structured guides for conversations with key stakeholders of FINRA. Do not read the questions or probes word for word. Instead, adapt the wording to match the phrasing used by the respondent and ask only those questions which have not been already addressed by the interviewees during earlier part of the interview. Take notes on key terms or phrases used by the respondents that may be helpful in coding the interview data. Ask for clarification and definitions as needed. Familiarize yourself with the interview protocol guides in advance of your meeting. The questions in bold are the questions you will prioritize if the respondent’s time is limited. Be respectful of the respondent’s time and keep the interview to the agreed length of time. Follow up by phone or email for more information as needed. In addition:  Take notes during the discussion. To ensure you accurately report what is discussed during the interview, record the session as well.  As necessary, tailor all questions to fit the individual stakeholders’ relationship with FINRA.  Keep the discussion under sixty minutes.  The research team must ensure that the information shared through these interviews remain strictly confidential. INTERVIEW GUIDE FOR USAID/BIH 1. Describe the country and sector context at the time of the Activity design (2016) and compare it to the current situation.  How would you describe the political climate in terms of improving the BiH financial sector stability? How does it compare to the situation from three years ago?  What were the main issues facing the BiH financial sector system at the moment of FINRA design and what are the main issues today?  What prominent issues can you think of with regard to banking supervision, deposit insurance, central banking, and audit of public financial resources/quality of government spending (then and now)?  What are the ramifications of this situation for economic development and progress toward the EU integration? 2. Describe the Activity’s development hypothesis/theory of change, expected results, and implementation mechanisms (technical assistance, grants).  How are the Activity’s theory of change, goals, and activities aligned with the PGEs objectives?  Were the PGEs consulted in the design and development of the Activity’s theory of change? 3. Would you say that FINRA interventions are needed to improve the BiH financial sector stability and compliance with the IMF/EU CRs?  Were these areas the primary priorities in the financial sector at the time?  What about now? 4. Describe the Activity’s design process.  Did the FINRA design accommodate any of the recommendations emphasized in the evaluation report of its ancestor Activity – Partnership for Advancing Reforms in the Economy (PARE)? (The 45 | FINRA PERFORMANCE EVALUATION USAID.GOV evaluation team will provide list of PARE’s evaluation recommendations in short bullets in case the KIs cannot recall these recommendations.)  Why was the Activity’s third component, focusing on the audit of public financial resources and quality of government spending, designed as an optional component?  What was the process for making decision on whether to approve the implementation of this component? How was this decision informed? 5. How would you assess the implementation of FINRA?  Were the FINRA interventions implemented so far tailored to the needs of the beneficiary institution(s)? How is collaboration with beneficiary institutions documented/agreed?  Were the interventions in which you participated complementary to the activities of other international donors (e.g., IMF, World Bank Group)? Were these interventions designed in collaboration with other donors? What was the format of this collaboration and how do you assess it? 6. What is your opinion of the manner and quality of management and implementation of FINRA interventions (in banking supervision, deposit insurance, central banking and/or audit of public financial resources/quality of government spending)? Describe good practices and areas for improvement.  Policy development interventions (i.e., standards, policies, laws, guidelines)  Capacity building interventions a. Expertise and pedagogical skills of trainers b. Quality of training  Institutional improvements interventions 7. Has FINRA faced any challenges in management or implementation, overall or due to the COVID-19 crisis?  If yes, how has the Activity mitigated these challenges?  How has the Activity supported its beneficiaries in coping with the pandemic? How would you assess this support? (e.g., effective and timely, inadequate) 8. How do you assess FINRA’s approach and effectiveness in administration of its Grant Fund?  Describe the grant award process from your standpoint?  Were the grant amounts provided sufficient to meet the needs of grant recipients? Were there any grant applications to which FINRA could not respond? If yes, why?  Has FINRA conducted any follow-ups to assess grants’ utilization/effectiveness? 9. Has the Activity faced any obstacles related to MEL reporting (e.g., data collection, data analysis) for any of the following performance indicators: a. Number of staff of Counterpart agencies and financial institutions trained with FINRA support b. Number of institutional improvements documented by FINRA and proposed to Counterpart government agencies c. Number of Standards addressing IMF/EU Compliance Requirements and other areas of financial stability (i) drafted with FINRA assistance and (ii) adopted by BiH authorities d. Number of IMF/EU Financial Stability Compliance Requirements significantly advanced toward fulfillment with FINRA assistance 46 | FINRA PERFORMANCE EVALUATION USAID.GOV 10. Why did the Activity overperform on performance indicators?  Do the actuals reported in the progress report correspond to the actuals in the BIHPERFORM system? If not, why?  How did the Activity define its targets? 11. Where would you say that BiH stands at the moment in terms of financial sector stability? How has FINRA contributed to these changes?  How do you assess BiH’s progress towards compliance with relevant international standards (IMF, EU)?  To what extent has FINRA made progress in improving the audit of public financial resources/quality of government spending? 12. How has FINRA ensured local ownership and sustainability of their results? 13. What lessons have you learned from the FINRA implementation?  Is there any aspect of the Activity implementation that you consider extremely successful or not so successful?  Is there any aspect of the Activity implementation that distinguishes FINRA from other USAID Activities, either as good or not so good example?  If you had an opportunity to design such an intervention or to implement it from the beginning, what would you have done differently? 14. Were there any unintended effects of FINRA? 15. Overall, to what extent would you say that PGEs are prepared to continue making progress with the IMF/EU CRs banking supervision, deposit insurance, and central banking coordination without FINRA assistance? 16. Is there a need for FINRA assistance in the area of audit of public financial resources/quality of government spending in the upcoming period? If yes, what should this assistance entail? 17. In your view, are there any adaptations that could improve or compliment the FINRA approach (technical assistance, grants) to strengthen the stability of the BiH financial sector? 18. What are the future Mission intentions in terms of supporting the BiH financial stability? INTERVIEW GUIDE FOR THE IMPLEMENTING PARTNER 1. Describe the country and sector context at the time of the Activity design (2016) and compare it to the current situation.  How would you describe the political climate in terms of improving the BiH financial sector stability? How does it compare to the situation from three years ago?  What are the ramifications of this situation for economic development and progress toward the EU integration? 2. Describe the Activity’s development hypothesis/theory of change, expected results, and implementation mechanisms (technical assistance, grants).  How are the Activity’s theory of change, goals, and activities aligned with the PGEs objectives?  Were the PGEs consulted in the design and development of the Activity’s theory of change? 47 | FINRA PERFORMANCE EVALUATION USAID.GOV 3. Describe the Activity’s design process.  Did the FINRA design accommodate any of the recommendations emphasized in the evaluation report of its ancestor Activity – Partnership for Advancing Reforms in the Economy (PARE)? (The evaluation team will provide list of PARE’s evaluation recommendations in short bullets in case the KIs cannot recall these recommendations.)  Why was the Activity third component, focusing on the audit of public financial resources and quality of government spending designed as an optional component?  What was the process for making decision on whether to approve the implementation of this component? How was this decision informed? 4. Describe the Activity implementation process.  Were the FINRA interventions implemented so far tailored to the need of the beneficiary institution(s)? How is collaboration with beneficiary institutions documented/agreed?  Were the FINRA interventions complementary to the activities of other international donors (e.g., IMF, World Bank Group)? Have these interventions been designed in collaboration with other donors? What was the format of this collaboration and how do you assess it? 5. Has FINRA faced any challenges in management or implementation, overall or due to the COVID-19 crisis?  If yes, how has the Activity mitigated these challenges?  How has the Activity supported its beneficiaries in coping with the pandemic? 6. Describe FINRA’s approach in administration of its Grant Fund.  Describe the grant award process from your standpoint?  Were the grant amounts provided sufficient to meet the needs of grant recipients? Were there any grant applications to which FINRA could not respond? If yes, why?  Has FINRA conducted any follow-ups to assess grants’ utilization/effectiveness? 7. Has the Activity faced any obstacles related to MEL reporting (e.g., data collection, data analysis) for any of the following performance indicators: e. Number of staff of Counterpart agencies and financial institutions trained with FINRA support f. Number of institutional improvements documented by FINRA and proposed to Counterpart government agencies g. Number of Standards addressing IMF/EU Compliance Requirements and other areas of financial stability (i) drafted with FINRA assistance and (ii) adopted by BiH authorities h. Number of IMF/EU Financial Stability Compliance Requirements significantly advanced toward fulfillment with FINRA assistance 8. Why did the Activity overperform on performance indicators?  Do the actuals reported in the progress report correspond to the actuals in the BIHPERFORM system? If not, why?  How did the Activity define its targets? 9. Where would you say that BiH stands at the moment in terms of financial sector stability? How has FINRA contributed to these changes? 48 | FINRA PERFORMANCE EVALUATION USAID.GOV 10. To what extent has FINRA made progress in terms of the IMF/EU CRs in these areas? What has changed in practice? a. Banking supervision and insurance (bank safety and soundness; problem bank management) b. Central banking 11. To what extent has FINRA made progress in improving the audit of public financial resources/quality of government spending? 12. How have the PGE’s institutional managerial, organization, and operational practices and procedures changed due to FINRA assistance?  What evidence do you have to substantiate that the PGE changes you mentioned have been made and FINRA’s contribution to those changes? 13. To what extent have the capacities of Partner Government Entities been improved due to FINRA assistance in the following areas…? Please elaborate. a. Banking supervision and insurance (bank safety and soundness; problem bank management) b. Central banking c. Audit of public financial resources/quality of government spending  Which skills did the trained PGE’s staff improve? (ask about the specific area below) a. Banking supervision and insurance (bank safety and soundness; problem bank management) b. Central banking c. Audit of public financial resources/quality of government spending  What evidence do you have to substantiate that the PGE changes you mentioned have been made and FINRA’s contribution to those changes? 14. What financial policies/processes have been improved due to FINRA assistance that had not been established before? To what extent have they been implemented?  If any such policies/processes have not been finalized yet, what is the estimated time for FINRA to complete them? 15. How has FINRA ensured local ownership and sustainability of their results? 16. What lessons have you learned from the FINRA implementation?  Are there any aspects of the Activity implementation that you consider extremely successful or not so successful?  If you had an opportunity to design such an intervention or to implement it from the beginning, what would you have done differently? 17. Were there any unintended effects of FINRA? 18. Overall, to what extent would you say that PGEs are prepared to continue making progress with the IMF/EU CRs banking supervision, deposit insurance, and central banking coordination without FINRA assistance? 19. Is there a need for FINRA assistance in the area of audit of public financial resources/quality of government spending in the upcoming period? If yes, what should this assistance entail? 20. In your view, are there any adaptations that could improve or compliment the FINRA approach (technical assistance, grants) to strengthen the stability of the BiH financial sector? 49 | FINRA PERFORMANCE EVALUATION USAID.GOV 21. Based on your knowledge of the current status and issues in the BiH financial sector, what type of interventions would you recommend that USAID implements in the future to support improving the compliance with the IMF/EU requirements and bolster the stability of the country’s financial sector? INTERVIEW GUIDE FOR BENEFICIARY GOVERNMENT INSTITUTIONS 1. Describe the country and sector context at the time of the Activity design (2016) and compare it to the current situation.  How would you describe the political climate in terms of improving the BiH financial sector stability? How does it compare to the situation from three years ago?  What are main issues facing the BiH financial sector system at the moment?  What about the banking supervision/deposit insurance/central banking/audit of public financial resources/quality of government spending?  What are the ramifications of this situation for economic development and progress toward the EU integration? 2. Describe the Activity’s design process.  How have you first heard about FINRA? Were you involved in the concept stage, design and or/ implementation?  How was your cooperation defined/documented? 3. In which FINRA interventions did you participate?  Could you describe how the possibility to participate in the intervention was introduced to you and how was the participation agreed?  To what extent were the FINRA interventions tailored to the need of your institution?  Were the interventions in which you participated complementary to the activities of other international donors (e.g., IMF, World Bank)? Have these interventions been designed in collaboration with other donors? 4. Would you say that FINRA interventions focusing on banking supervision/deposit insurance/central banking/audit of public financial resources and quality of government spending were well selected?  Would you say that FINRA interventions are: a. necessary, and b. sufficient to improve the BiH financial sector stability and compliance with the IMF/EU CRs?  Were these areas the primary priorities in the financial sector at the time? What about priorities of your institution?  What about now? 5. What is your opinion of the manner and quality of management and implementation of FINRA interventions (in banking supervision, deposit insurance, central banking and/or audit of public financial resources/quality of government spending)? Describe good practices and areas for improvement.  Policy development interventions (i.e., standards, policies, laws, guidelines)  Capacity building interventions c. Expertise and pedagogical skills of trainers d. Quality of training e. Relevance/usefulness of training 50 | FINRA PERFORMANCE EVALUATION USAID.GOV  Institutional improvements interventions  Other (if applicable) 6. Has any assistance that FINRA was providing to you or planned to provide affected by the COVID-19 crisis?  If yes, how has the Activity mitigated these challenges?  Has the Activity provided any specific assistance to your institution in coping with the COVID-19? If yes, how and how would you assess this support? (e.g., effective and timely, inadequate) 7. Are you familiar with the FINRA Grant Fund?  Have you applied for a grant from FINRA? If yes, could you describe the grant award process from your standpoint?  Was your grant application approved? For which purpose?  How would you assess FINRA’s approach and grant assistance? 8. Where would you say that BiH stands at the moment in terms of financial sector stability? How has FINRA contributed to these changes? 9. To what extent has FINRA made progress in terms of the IMF/EU CRs in these areas? What has changed in practice?  Banking supervision and insurance (bank safety and soundness; problem bank management)  Central banking (The evaluation team may provide a list of CRs to help KIs answer this question; PGEs should respond only to those areas for which their institution is responsible for) 10. To what extent has FINRA made progress in improving the audit of public financial resources/quality of government spending? (The question should be asked only to Ministries of finance involved in the Activity)? 11. To what extent have the capacities of your institution been improved due to FINRA assistance? Please elaborate.  Which skills did your staff improve or acquire due to FINRA capacity building workshops? 12. What financial policies/processes have been developed/introduced due to FINRA assistance that had not been established before? To what extent have they been implemented?  If any such policies/processes have not been finalized yet, what is the estimated time for FINRA to complete them? 13. How have the institutional managerial, organization, and operational practices and procedures changed due to FINRA assistance? 14. How has FINRA ensured local ownership and sustainability of their results? 15. What lessons have you learned from the FINRA implementation?  If you had an opportunity to design such an intervention or to implement it from the beginning, what would you have done differently? 16. Were there any unintended effects of FINRA? 51 | FINRA PERFORMANCE EVALUATION USAID.GOV 17. Overall, to what extent would you say that your institution is prepared to continue making progress with the IMF/EU CRs banking supervision/deposit insurance/central banking coordination without FINRA assistance?  Do you have any action plan and/or dedicated resources to continue making progress with the IMF/EU CRs banking supervision/deposit insurance/central banking coordination? 18. Is there a need for FINRA assistance in the area of audit of public financial resources/quality of government spending in the upcoming period? If yes, what should this assistance entail? (The question should be asked only of the Ministries of finance involved in the Activity) 19. In your view, are there any adaptations that could improve or compliment the FINRA approach (technical assistance, grants) to strengthen the stability of the BiH financial sector? 20. Based on your knowledge of the current status and issues in the BiH financial sector, what type of interventions would you recommend that USAID implements in the future to support improving the compliance with the IMF/EU requirements and bolster the stability of the country’s financial sector? INTERVIEW GUIDE FOR INTERNATIONAL ORGANIZATIONS 1. Describe the country and sector context at the time of the Activity design (2016) and compare it to the current situation.  How would you describe the political climate in terms of improving the BiH financial sector stability? How does it compare to the situation from three years ago?  What were the main issues facing the BiH financial sector system at the moment of FINRA design and what are the main issues today?  What prominent issues can you think of with regard to banking supervision, deposit insurance, central banking, and audit of public financial resources/quality of government spending?  What are the ramifications of this situation for economic development and progress toward the EU integration? 2. Would you say that FINRA interventions focusing on banking supervision/deposit insurance/central banking/audit of public financial resources and quality of government spending were well selected?  Would you say that FINRA interventions are needed to improve the BiH financial sector stability and compliance with the IMF/EU CRs?  Were these areas the primary priorities in the financial sector at the time?  What about now? 3. How have you first heard about FINRA? Were you involved in the concept stage, design or implementation?  How was your cooperation defined/documented?  How would you assess your cooperation with FINRA?  In which FINRA interventions were you involved and in what capacity?  Were the interventions you are acquainted with tailored to the need of the beneficiary institution(s)?  Were the interventions in which you participated complementary to your activities or the activities of other international donors? Have these interventions been designed in collaboration with other donors? 52 | FINRA PERFORMANCE EVALUATION USAID.GOV 4. What is your opinion of the manner of implementation of FINRA interventions (in banking supervision, deposit insurance, central banking and/or audit of public financial resources/quality of government spending)? Discuss good practices and areas for improvement.  Policy development interventions (i.e., standards, policies, laws, guidelines)  Capacity building interventions  Institutional improvements interventions  Other (if applicable) 5. Are you familiar with the FINRA Grant Fund? If yes, how would you assess FINRA’s approach and grant administration? 6. Where would you say that BiH stands at the moment in terms of financial sector stability? How has FINRA contributed to these changes? 7. To what extent has FINRA made progress in terms of the IMF/EU CRs in these areas? What has changed in practice?  Banking supervision and insurance (bank safety and soundness; problem bank management)  Central banking (The evaluation team may provide a list of CRs to help KIs answer this question; PGEs should respond only to those areas for which their institution is responsible for) 8. To what extent has FINRA made progress in improving the audit of public financial resources/quality of government spending? 9. What financial policies/processes have been developed/introduced due to FINRA assistance that had not been established before? To what extent have they been implemented? 10. How have the institutional managerial, organization, and operational practices and procedures changed due to FINRA assistance? 11. How has FINRA ensured local ownership and sustainability of their results? 12. Are you familiar with any unintended effects of FINRA? If yes, describe those. 13. Overall, to what extent would you say that BiH institutions are prepared to continue making progress with the IMF/EU CRs banking supervision/deposit insurance/ central banking coordination without FINRA/donor assistance? 14. In your view, are there any adaptations that could improve or compliment the FINRA approach (technical assistance, grants) to strengthen the stability of the BiH financial sector? 15. Based on your knowledge of the current status and issues in the BiH financial sector, what type of interventions would you recommend that USAID implements in the future to support improving the compliance with the IMF/EU requirements and bolster the stability of the country’s financial sector? 53 | FINRA PERFORMANCE EVALUATION USAID.GOV INTERVIEW GUIDE FOR FINRA CONSULTANTS 1. Describe the country and sector context at the time of the Activity design (2016) and compare it to the current situation.  How would you describe the political climate in terms of improving the BiH financial sector stability? How does it compare to the situation from three years ago?  What were the main issues facing the BiH financial sector system at the moment of FINRA design and what are the main issues today?  What prominent issues can you think of with regard to banking supervision, deposit insurance, central banking, and audit of public financial resources/quality of government spending (then and now)?  What are the ramifications of this situation for economic development and progress toward the EU integration? 2. Would you say that FINRA interventions focusing on banking supervision/deposit insurance/central banking/audit of public financial resources and quality of government spending were well selected?  Would you say that FINRA interventions are needed to improve the BiH financial sector stability and compliance with the IMF/EU CRs?  Were these areas the primary priorities in the financial sector at the time?  What about now? 3. Describe the Activity implementation process.  Were the FINRA interventions implemented so far tailored to the need of the beneficiary institution(s)?  Were the FINRA interventions complementary to the activities of other international donors (e.g., IMF, World Bank Group)? 4. Has FINRA faced any challenges in management or implementation, overall or due to the COVID-19 crisis?  If yes, how has the Activity mitigated these challenges?  How has the Activity supported its beneficiaries in coping with the pandemic? 5. Have you been involved in the FINRA grants administration? If yes, describe FINRA’s approach in administration of its Grant Fund.  Describe the grant award process from your standpoint?  Were the grant amounts provided sufficient to meet the needs of grant recipients? Were there any grant applications to which FINRA could not respond? If yes, why?  Has FINRA conducted any follow-ups to assess grants’ utilization/effectiveness? 6. Where would you say that BiH stands at the moment in terms of financial sector stability? How has FINRA contributed to these changes? 7. To what extent has FINRA made progress in terms of the IMF/EU CRs in these areas? What has changed in practice? c. Banking supervision and insurance (bank safety and soundness; problem bank management) d. Central banking 8. To what extent has FINRA made progress in improving the audit of public financial resources/quality of government spending? 54 | FINRA PERFORMANCE EVALUATION USAID.GOV 9. To what extent have the capacities of Partner Government Entities been improved due to FINRA assistance in the following areas…? Please elaborate. d. Banking supervision and insurance (bank safety and soundness; problem bank management) e. Central banking f. Audit of public financial resources/quality of government spending  Which skills do the trained PGE’s staff have now that they did not have before? (ask about the specific area below) d. Banking supervision and insurance (bank safety and soundness; problem bank management) e. Central banking f. Audit of public financial resources/quality of government spending 10. What financial policies/processes have been developed/introduced due to FINRA assistance that had not been established before? To what extent have they been implemented?  If any such policies/processes have not been finalized yet, what is the estimated time for FINRA to complete them? 11. How has FINRA ensured local ownership and sustainability of their results? 12. What lessons have you learned from the FINRA implementation?  Is there any aspect of the Activity implementation that you consider extremely successful or not so successful?  Is there any aspect of the Activity implementation that distinguishes FINRA from other USAID/donor Activities, either as good or not so good example?  If you had an opportunity to design such an intervention or to implement it from the beginning, what would you have done differently? 13. Were there any unintended effects of FINRA? 14. Overall, to what extent would you say that PGEs are prepared to continue making progress with the IMF/EU CRs banking supervision, deposit insurance, and central banking coordination without FINRA assistance? 15. Is there a need for FINRA assistance in the area of audit of public financial resources/quality of government spending in the upcoming period? What should this assistance entail? 16. In your view, are there any adaptations that could improve or compliment the FINRA approach (technical assistance, grants) to strengthen the stability of the BiH financial sector? 17. Based on your knowledge of the current status and issues in the BiH financial sector, what type of interventions would you recommend that USAID implements in the future to support improving the compliance with the IMF/EU requirements and bolster the stability of the country’s financial sector?  In your specific area (banking supervision/deposit insurance/central banking)  In the financial stability sector overall 55 | FINRA PERFORMANCE EVALUATION USAID.GOV INTERVIEW GUIDE FOR FINRA NON-BENEFICIARIES 1. Describe the country and sector context at the time of the Activity design (2016) and compare it to the current situation.  How would you describe the political climate in terms of improving the BiH financial sector stability? How does it compare to the situation from three years ago?  What were the main issues facing the BiH financial sector system at the moment of FINRA design and what are the main issues today?  What prominent issues can you think of with regard to banking supervision, deposit insurance, and central banking?  What are the ramifications of this situation for economic development and progress toward the EU integration? 2. Are you familiar with the FINRA implementation? If yes:*  Would you agree that FINRA interventions are needed to improve the BiH financial sector stability and compliance with the IMF/EU CRs?  Were these areas the primary priorities in the financial sector at the time? 3. Where would you say that BiH stands at the moment in terms of financial sector stability? How has FINRA contributed to these changes?* 4. Has BiH made progress in terms of the IMF/EU CRs in these areas since 2018? What has changed in practice? e. Banking supervision and insurance (bank safety and soundness; problem bank management) f. Central banking (The evaluation team may provide a list of CRs to help KIs answer this question)  Can these changes be attributed to FINRA assistance?* 5. How has the audit of public financial resources/quality of government spending changed since 2019? Do you believe FINRA contributed to this and, if yes, how?* 6. To what extent have the capacities of Partner Government Entities been improved since 2018 in the following areas…? Please elaborate. g. Banking supervision and insurance (bank safety and soundness; problem bank management) h. Central banking i. Audit of public financial resources/quality of government spending  Can these improvements be attributed as accomplishments of FINRA?* 7. Are you familiar with any financial policies/processes developed/introduced due to FINRA assistance? To what extent have they been implemented?* 8. Overall, to what extent would you say that PGEs are prepared to continue making progress with the IMF/EU CRs banking supervision, deposit insurance, and central banking coordination without further donor assistance? 9. In your view, are there any adaptations that could improve or compliment the FINRA approach (technical assistance, grants) to strengthen the stability of the BiH financial sector?* *Questions or sub-questions that should be asked only if the KI is familiar with the FINRA work. 56 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX V: SURVEY QUESTIONNAIRES ONLINE SURVEY QUESTIONNAIRE FOR FINRA CONSULTANTS 1. Please state the main problems in the BiH’s financial sector from three years ago: UP TO TWO RESPONSES a. Lack of compliance with the international (IMF, EU) standards b. Insufficiently developed bank supervision system c. Low bank profitability d. Exposure to credit risk/high share of non-performing loans (NPLs) as a percentage of total loans e. Low domestic credit to the private sector as the share of GDP f. Limited powers of the BiH Central Bank g. Lack of cooperation and information-sharing among the main financial stability institutions h. Poor public understanding and perception of work of the main financial stability institutions i. Insufficient harmonization of the entity laws and regulations governing financial stability j. Other, please specify: 2. Please state the main problems in the BiH’s financial sector today: UP TO TWO RESPONSES a. Lack of compliance with the international (IMF, EU) standards b. Insufficiently developed bank supervision system c. Low bank profitability d. Exposure to credit risk/high share of non-performing loans (NPLs) as a percentage of total loans e. Low domestic credit to the private sector as the share of GDP f. Limited powers of the BiH Central Bank g. Lack of cooperation and information sharing among main financial stability institutions h. Poor public understanding and perception of work of the main financial stability institutions i. Insufficient harmonization of the entity laws and regulations governing financial stability j. Potential economic fallout caused by the COVID-19 k. Other, please specify: 3. How would you rate the political climate from the standpoint of improving the BiH financial sector stability? a. Very unfavorable b. Unfavorable c. Favorable d. Very favorable 4. When only financial sector is assessed, would you say BiH is further advanced in the EU integrations than three years ago? a. Strongly disagree b. Somewhat disagree c. Somewhat agree d. Strongly agree 5. To what extent do you agree or disagree with the following statements: BiH made a significant progress toward meeting the IMF and EU compliance requirements compared to three years ago in terms of… Completely disagree Somewhat disagree Somewhat agree Completely agree DK/NA Banking supervision 1 2 3 4 0 Deposit insurance 1 2 3 4 0 Central banking 1 2 3 4 0 Stability of BiH financial sector in general 1 2 3 4 0 57 | FINRA PERFORMANCE EVALUATION USAID.GOV 6. How would you rate the following aspects of FINRA on a scale from 1 to 5, where 1 is poor and 5 is excellent/extremely successful? 1 2 3 4 5 DK/NA Design 1 2 3 4 5 0 Relevance for improving the financial sector’s stability in BiH 1 2 3 4 5 0 Management 1 2 3 4 5 0 Implementation of trainings 1 2 3 4 5 0 Implementation of grant awards 1 2 3 4 5 0 Implementation of the policy development component (standards, laws, bylaws…) 1 2 3 4 5 0 Flexibility of implementation 1 2 3 4 5 0 Response to the COVID-19 crisis 1 2 3 4 5 0 7. Were the activities you were engaged in tailored to the needs and developed in collaboration with the beneficiary institutions? a. Yes b. No c. I am not sure 8. Were the activities you were engaged in complementary to the activities of other donors/international organizations supporting BiH financial sector? a. Yes b. No c. I am not sure 9. Were there any unintended effects of FINRA? a. Yes b. No 10. If YES, can you describe those effects? Open-ended question 11. Overall, to what extent do you agree or disagree that PGEs are prepared to continue making progress with the IMF/EU CRs in terms of …? Completely disagree Somewhat disagree Somewhat agree Completely agree DK/NA Banking supervision 1 2 3 4 0 Deposit insurance 1 2 3 4 0 Central banking 1 2 3 4 0 Stability of the BiH financial sector in general 1 2 3 4 0 12. Overall, to what extent do you agree or disagree that following institutions are prepared to continue strengthening their capacities without further FINRA or other donor/international organizations’ support? Completely disagree Somewhat disagree Somewhat agree Completely agree DK/NA Central Bank of BiH 1 2 3 4 0 Agency for Deposit Insurance of BiH 1 2 3 4 0 Banking Agency of FBiH 1 2 3 4 0 58 | FINRA PERFORMANCE EVALUATION USAID.GOV Banking Agency of RS 1 2 3 4 0 13. What would you emphasize as FINRA’s major achievement to date (either related to the activities you were involved in or more broadly): Open-ended question 14. What financial policies/processes have been introduced/improved due to FINRA’s assistance (either related to the activities you were involved in or more broadly)? Open-ended question 15. Is there a need for FINRA’s assistance in the area of audit of public financial resources/quality of government spending in the upcoming period? a. Yes b. No 16. What should this assistance entail? Open-ended question 17. To what extent do you agree that USAID should retain the approach that FINRA used (technical assistance, grants) in its future interventions to strengthen the stability of the BiH financial sector? a. Completely agree b. Somewhat agree c. Somewhat disagree d. Completely disagree 18. What adaptations to the current approach used by FINRA (technical assistance, grants) would you recommend to further facilitate strengthening the stability of the BiH financial sector? Open-ended question 19. What are the current priorities in the BiH financial sector that should be addressed to achieve progress in fulfillment of the IMF and EU requirements? Open-ended question 20. Based on your knowledge of the current status and issues in the BiH financial sector, what type of interventions would you recommend that USAID implements in the future to improve the compliance with the IMF/EU requirements and therefore, bolster the stability of the country’s financial sector? Open-ended question 59 | FINRA PERFORMANCE EVALUATION USAID.GOV RESULTS OF ONLINE SURVEY OF FINRA CONSULTANTS 58.3% 20.8% 0.0% 12.5% 8.3% 8.3% 25.0% 12.5% 16.7% 12.5% Lack of compliance with the international (IMF, EU) standards Insufficiently developed bank supervision system Low bank profitability Exposure to credit risk/high share of non￾performing loans (NPLs) as a percentage of total loans Low domestic credit to the private sector as the share of GDP Limited powers of the BiH Central Bank Lack of cooperation and information sharing among the main financial stability institutions Poor public understanding and perception of work of the main financial stability institutions Insufficient harmonization of the entity laws and regulations governing financial stability Other, please specify: Please state the main problems in the BiH financial sector from 3 years ago: 20.8% 8.3% 8.3% 4.2% 8.3% 12.5% 16.7% 8.3% 20.8% 58.3% 4.2% Lack of compliance with the international (IMF, EU) standards Insufficiently developed bank supervision system Low bank profitability Exposure to credit risk / high share of non￾performing loans (NPLs) as a percentage of total loans Low domestic credit to the private sector as the share of GDP Limited powers of the BiH Central Bank Lack of cooperation and information sharing among main financial stability institutions Poor public understanding and perception of work of the main financial stability institutions Insufficient harmonization of the entity laws and regulations governing financial stability Potential economic fallout caused by the COVID-19 Other, please specify: Please state the main problems in the BiH financial sector today: 12.5% 50.0% 37.5% 0.0% Very unfavorable Unfavorable Favorable Very favorable How would you rate the political climate from the standpoint of improving the BiH financial sector stability? 60 | FINRA PERFORMANCE EVALUATION USAID.GOV 4.2% 0.0% 66.7% 29.2% Strongly disagree Somewhat disagree Somewhat agree Strongly agree When only financial sector is assessed, would you say BiH is further advanced in the EU integrations than three years ago? 0.0% 0.0% 0.0% 0.0% 4.2% 8.3% 4.2% 12.5% 41.7% 41.7% 41.7% 62.5% 37.5% 29.2% 33.3% 12.5% 16.7% 20.8% 20.8% 12.5% Banking supervision Deposit insurance Central banking Stability of BiH financial sector in general BiH made a significant progress toward meeting the IMF and EU compliance requirements compared to three years ago in terms of… (on a scale from 1 - completely disagree to 4 - completely agree) Completely disagree Somewhat disagree Somewhat agree Completely agree Don't know 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 4.6% 4.4% 4.2% 4.2% 4.2% 4.2% 4.2% 8.3% 0.0% 0.0% 4.2% 4.2% 4.2% 8.3% 8.3% 0.0% 40.9% 30.4% 25.0% 29.2% 16.7% 29.2% 33.3% 25.0% 40.9% 65.2% 62.5% 37.5% 20.8% 37.5% 54.2% 13.6% 12.5% 0.0% 4.2% 25.0% 54.2% 20.8% 0.0% 54.2% Design Relevance for improving the financial sector’s stability in BiH Management Implementation of trainings Implementation of grant awards Implementation of the policy development component (standards, laws, bylaws…) Flexibility of implementation Response to the COVID-19 crisis How would you rate the following aspects of FINRA on a scale from 1 to 5, where 1 is poor and 5 is excellent/extremely successful? 1 2 3 4 5 Don't know 61 | FINRA PERFORMANCE EVALUATION USAID.GOV 95.8% 0.0% 4.2% Yes No I am not sure Were the activities you were engaged in tailored to the needs and developed in collaboration with the beneficiary institutions? 78.3% 4.4% 17.4% Yes No I am not sure Were the activities you were engaged in complementary to the activities of other donors/international organizations supporting BiH financial sector? 13.0% 87.0% Yes No Were there any unintended effects of FINRA? 62 | FINRA PERFORMANCE EVALUATION USAID.GOV 4.4% 4.4% 4.4% 4.4% 4.4% 0.0% 0.0% 0.0% 21.7% 21.7% 30.4% 43.5% 43.5% 30.4% 26.1% 17.4% 26.1% 43.5% 39.1% 34.8% Banking supervision Deposit insurance Central banking Stability of the BiH financial sector in general Overall, to what extent do you agree or disagree that PGEs are prepared to continue making progress with the IMF/EU CRs in terms of… ? Completely disagree Somewhat disagree Somewhat agree Completely agree Don't know 4.4% 0.0% 0.0% 4.6% 17.4% 21.7% 34.8% 31.8% 30.4% 21.7% 30.4% 27.3% 4.4% 0.0% 0.0% 0.0% 43.5% 56.5% 34.8% 36.4% Central Bank of BiH Agency for Deposit Insurance of BiH Banking Agency of FBiH Banking Agency of RS Overall, to what extent do you agree or disagree that following institutions are prepared to continue strengthening their capacities without further FINRA or other donor/international organizations’ support? Completely disagree Somewhat disagree Somewhat agree Completely agree Don't know 70.0% 30.0% Yes No Is there a need for FINRA’s assistance in the area of audit of public financial resources/quality of government spending in the upcoming period? 63 | FINRA PERFORMANCE EVALUATION USAID.GOV 69.6% 21.7% 8.7% 0.0% Completely agree Somewhat agree Somewhat disagree Completely disagree To what extent do you agree that USAID should retain the approach that FINRA used (technical assistance, grants) in its future interventions to strengthen the stability of the BiH financial sector? 64 | FINRA PERFORMANCE EVALUATION USAID.GOV ONLINE SURVEY QUESTIONNAIRE FOR CAPACITY BUILDING PARTICIPANTS 1. Please state the main problems in the BiH’s financial sector from 3 years ago: UP TO TWO RESPONSES a. Lack of compliance with the international (IMF, EU) standards b. Insufficiently developed bank supervision system c. Low bank profitability d. Exposure to credit risk/high share of non-performing loans (NPLs) as a percentage of total loans e. Low domestic credit to the private sector as the share of GDP f. Limited powers of the BiH Central Bank g. Lack of cooperation and information-sharing among the main financial stability institutions h. Poor public understanding and perception of work of the main financial stability institutions i. Insufficient harmonization of the entity laws and regulations governing financial stability j. Other, please specify: 2. Please state the main problems in the BiH financial sector today: UP TO TWO RESPONSES a. Lack of compliance with the international (IMF, EU) standards b. Insufficiently developed bank supervision system c. Low bank profitability d. Exposure to credit risk/high share of non-performing loans (NPLs) as a percentage of total loans e. Low domestic credit to the private sector as the share of GDP f. Limited powers of the BiH Central Bank g. Lack of cooperation and information sharing among main financial stability institutions h. Poor public understanding and perception of work of the main financial stability institutions i. Insufficient harmonization of the entity laws and regulations governing financial stability j. Potential economic fallout caused by the COVID-19 k. Other, please specify: 3. How would you rate the political climate from the standpoint of improving the BiH financial sector stability? a. Very unfavorable b. Unfavorable c. Favorable d. Very favorable 4. When only financial sector is assessed, would you say BiH is further advanced in the EU integrations than three years ago? a. Strongly disagree b. Somewhat disagree c. Somewhat agree d. Strongly agree 5. To what extent do you agree or disagree with the following statements: BiH significantly advanced its progress toward the IMF and EU compliance requirements compared to three years ago in terms of… Completely disagree Somewhat disagree Somewhat agree Completely agree DK/NA Banking supervision 1 2 3 4 0 Deposit insurance 1 2 3 4 0 Central banking 1 2 3 4 0 Stability of the BiH financial sector in general 1 2 3 4 0 65 | FINRA PERFORMANCE EVALUATION USAID.GOV 6. How would you rate the quality of the following FINRA interventions? Very unfavorable Unfavorable Favorable Very favorable DK/NA Policy development (i.e., standards, regulations, laws, bylaws) 1 2 3 4 0 Institutional improvement interventions 1 2 3 4 0 Assessments and analyses 1 2 3 4 0 7. How would you rate the following aspects of FINRA? Very unfavorable Unfavorable Favorable Very favorable DK/NA Design 1 2 3 4 0 Relevance for improving the financial sector’s stability in BiH 1 2 3 4 0 Management 1 2 3 4 0 Implementation of trainings 1 2 3 4 0 Implementation of grant awards 1 2 3 4 0 Implementation of the policy development component (standards, laws, bylaws…) 1 2 3 4 0 Tailoring interventions based on institutions’ needs 1 2 3 4 0 Flexibility of implementation 1 2 3 4 0 Coordination with other donor or government projects 1 2 3 4 0 Usefulness of the activity for your institution 1 2 3 4 0 Response to the COVID-19 crisis 1 2 3 4 0 8. Did you participate in any FINRA trainings in the last three years? a. Yes - continue b. No 9. In which FINRA training(s) did you participate? a. Bank Resolution/Resolution Planning/Restructuring/Recovery b. BARS Organizational structure c. Cash Sorting d. CBBH Network analysis e. CBBH Public Communications f. DIA Financial Model g. DIA Management h. Early Intervention and Recovery i. Financial Macroeconomics for Supervisors j. Financial Reporting (FINREP) k. FINRA Grant Workshop l. Internal Capital Adequacy Assessment Process (ICAAP)/Supervisory Review and Evaluation Process (SREP) m. Internal Governance Workshop n. Internal Liquidity Adequacy Assessment Process (ILAAP) o. International Financial Reporting Standards 9 (IFRS9) p. Liquidity coverage ratio (LCR) q. Macroeconomic Forecasting r. Macroeconomic Imbalances s. Risk-based Premia t. Stress Testing 66 | FINRA PERFORMANCE EVALUATION USAID.GOV 10. What is your opinion on the following aspects of the training(s): Very unfavorable Unfavorable Favorable Very favorable DK/NA Overall quality of training 1 2 3 4 0 Trainer’s expertise 1 2 3 4 0 Trainer’s pedagogical skills 1 2 3 4 0 Relevance of training topic 1 2 3 4 0 Timeliness of training 1 2 3 4 0 Applicability of training 1 2 3 4 0 11. To what extent has the capacity of your institution to achieve progress toward the IMF/EU compliance requirements improved due to FINRA assistance in the following areas? Not at all Slightly Moderately Extremely I did not participate in such interventions Banking supervision 1 2 3 4 0 Deposit insurance 1 2 3 4 0 Central banking 1 2 3 4 0 12. To what extent have your institution’s practices and procedures improved due to FINRA assistance? a. Not at all (go to question 14) b. Slightly (go to question 14) c. Moderately (go to question 13) d. Extremely (go to question 13) 13. What procedures have been introduced or significantly improved with the help of the FINRA in your institution? Open-ended question 14. What are the current priorities in the BiH financial sector that should be addressed to achieve progress in fulfilment of the IMF and EU requirements? Open-ended question 15. Based on your knowledge of the current status and issues in the BiH financial sector, what type of interventions would you recommend that USAID implements in the future to improve the compliance with the IMF/EU requirements and therefore, bolster the stability of the country’s financial sector? Open-ended question 67 | FINRA PERFORMANCE EVALUATION USAID.GOV RESULTS OF ONLINE SURVEY OF CAPACITY BUILDING PARTICIPANTS 33.3% 11.1% 0.0% 25.0% 16.7% 16.7% 16.7% 38.9% 11.1% 2.8% Lack of compliance with the international (IMF, EU) standards Insufficiently developed bank supervision system Low bank profitability Exposure to credit risk/high share of non-performing loans (NPLs) as a percentage of total loans Low domestic credit to the private sector as the share of GDP Limited powers of the BiH Central Bank Lack of cooperation and information-sharing among the main financial stability institutions Poor public understanding and perception of work of the main financial stability institutions Insufficient harmonization of the entity laws and regulations governing financial stability Other, please specify: Please state the main problems in the BiH financial sector from 3 years ago: 2.8% 5.6% 0.0% 13.9% 13.9% 22.2% 11.1% 36.1% 11.1% 66.7% 0.0% Lack of compliance with the international (IMF, EU) standards Insufficiently developed bank supervision system Low bank profitability Exposure to credit risk / high share of non-performing loans (NPLs) as a percentage of total loans Low domestic credit to the private sector as the share of GDP Limited powers of the BiH Central Bank Lack of cooperation and information sharing among main financial stability institutions Poor public understanding and perception of work of the main financial stability institutions Insufficient harmonization of the entity laws and regulations governing financial stability Potential economic fallout caused by the COVID-19 Other, please specify: Please state the main problems in the BiH financial sector today: 9.4% 68.8% 21.9% 0.0% Very unfavorable Unfavorable Favorable Very favorable How would you rate the political climate from the standpoint of improving the BiH financial sector stability? 68 | FINRA PERFORMANCE EVALUATION USAID.GOV 2.8% 16.7% 52.8% 27.8% Strongly disagree Somewhat disagree Somewhat agree Strongly agree When only financial sector is assessed, would you say BiH is further advanced in the EU integrations than three years ago? 11.4% 6.5% 3.2% 11.8% 2.9% 3.2% 9.7% 2.9% 37.1% 58.1% 38.7% 55.9% 45.7% 9.7% 12.9% 20.6% 2.9% 22.6% 35.5% 8.8% Banking supervision Deposit insurance Central banking Stability of the BiH financial sector in general BiH significantly advanced its progress toward the IMF and EU compliance requirements compared to three years ago in terms of… Completely disagree Somewhat disagree Somewhat agree Completely agree Don't know 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 48.6% 50.0% 41.2% 37.1% 38.2% 35.3% 14.3% 11.8% 23.5% Policy development (i.e., standards, regulations, laws, bylaws) Institutional improvement interventions Assessments and analyses How would you rate the quality of the following FINRA interventions? Very unfavorable Unfavorable Favorable Very favorable Don't know 69 | FINRA PERFORMANCE EVALUATION USAID.GOV 0.0% 0.0% 0.0% 0.0%2.8% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%2.8% 0.0% 0.0%2.8% 0.0% 0.0% 0.0%8.6% 57.1% 52.9% 41.7% 38.9% 38.9%50.0% 33.3% 36.1% 36.1% 31.4% 31.4% 17.9% 41.2% 44.4%58.3% 16.7% 33.3%47.2%58.3% 30.6% 62.9% 17.1%25.0% 5.9%13.9% 0.0% 41.7% 16.7% 16.7% 5.6% 33.3% 5.7% 42.9% Design Relevance for improving the financial sector’s stability in BiHManagement Implementation of trainings Implementation of grant awards Implementation of the policy development component (standards, laws, bylaws…) Tailoring interventions based on institutions’ needs Flexibility of implementation Coordination with other donor or government projects Usefulness of the activity for your institution Response to the COVID-19 crisis How would you rate the following aspects of FINRA? Very unfavorable Unfavorable Favorable Very favorable Don't know 12.5% 6.3% 12.5% 9.4% 3.1% 21.9% 12.5% 12.5% 28.1% 6.3% 0.0% 6.3% 15.6% 46.9% 40.6% 18.8% 6.3% 15.6% 3.1% 12.5% Financial Macroeconomics for Supervisors DIA Financial Model Financial Reporting (FINREP) FINRA Grant Workshop CBBH Public Communications Liquidity coverage ratio (LCR) Macroeconomic Forecasting Macroeconomic Imbalances International Financial Reporting Standards 9 (IFRS9) DIA Management CBBH Network analysis BARS Organizational structure Risk-based Premia Internal Capital Adequacy Assessment Process (ICAAP)/Supervisory Review and Evaluation Process (SREP) Internal Liquidity Adequacy Assessment Process (ILAAP) Internal Governance Workshop Early Intervention and Recovery Bank Resolution/Resolution Planning/Restructuring/Recovery Cash Sorting Stress Testing In which FINRA training(s) did you participate? 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 3.1% 0.0% 3.1% 0.0% 43.8% 31.3% 43.8% 28.1% 34.4% 34.4% 56.3% 68.8% 53.1% 71.9% 62.5% 65.6% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% Overall quality of training Trainer’s expertise Trainer’s pedagogical skills Relevance of training topic Timeliness of training Applicability of training What is your opinion of the following aspects of the training(s): Very unfavorable Unfavorable Favorable Very favorable Don't know 70 | FINRA PERFORMANCE EVALUATION USAID.GOV 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 41.2% 26.9% 18.5% 26.5% 7.7% 14.8% 32.4% 65.4% 66.7% Banking supervision Deposit insurance Central banking To what extent has the capacity of your institution to achieve progress toward the IMF/EU compliance requirements improved due to FINRA assistance in the following areas? Not at all Slightly Moderately Extremely NA 0.0% 2.9% 57.1% 40.0% Not at all Slightly Moderately Extremely To what extent have your institution’s practices and procedures improved due to FINRA assistance? 71 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX VI: EVALUATION MATRIX EVALUATION QUESTION (EQ) DATA SOURCES DATA COLLECTION METHODS RESEARCH DESIGN What progress has been accomplished in reaching contract targets? Activity documentation: Contract; MEL Plan; Activity progress reports; Activity Tracking Table Documents delivered by FINRA; BiHPERFORM system Mixed methods Secondary documentation: PGEs reports, international organizations’ reports Online available resources Key informants: USAID/BiH, IP, PGEs, STTAs, international organizations, non-beneficiaries, trainees KIIs and online surveys How is FINRA assistance perceived by beneficiaries and other stakeholders from the standpoint of its design and implementation? How has FINRA assisted its beneficiaries to respond to the potential economic fallout caused by the COVID-19 pandemic and how do beneficiaries perceive this assistance? Activity documentation: Contract; Activity work plans; Activity progress reports; Documents delivered by FINRA Mixed methods Secondary documentation: PGEs reports, international organizations’ reports Online available resources Key informants: USAID/BiH, IP, PGEs, STTAs, international organizations, non-beneficiaries, trainees KIIs and online surveys To what extent do stakeholders and beneficiaries perceive PGEs are able to advance compliance with the international regulatory requirements in banking supervision, deposit insurance, and central banking coordination activities without USAID assistance? Activity documentation: Activity progress reports Documents delivered by FINRA Mixed methods Secondary documentation: PGEs reports, international organizations’ reports Online available resources Key informants: USAID/BiH , IP, PGEs, STTAs, international organizations, non-beneficiaries, trainees KIIs and online surveys 72 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX VII: LIST OF RELEVANT KEY FINANCIAL STABILITY-RELATED DOCUMENTS LIST OF IMF AND EU FINANCIAL STABILITY RELATED DOCUMENTS Author Title Date Code IMF Financial System Stability Assessment June 2015 FSSA IMF Financial Sector Assessment Program: Banking Sector Supervision Core Principles – Technical Note July 2015 FSAP-B IMF Financial Sector Assessment Program: Technical Note – Systemic Liquidity Management, Financial Safety Net, Insolvency Framework, and Macroprudential Policy July 2015 FSAP-L IMF Financial Sector Assessment Program: Detailed Assessment of Observance of the CPMI-IOSCO Core Principles for Financial Market Infrastructures July 2015 FSAP-P IMF Financial Sector Assessment Program: Insurance Sector – Technical Note July 2015 FSAP-I IMF Financial Sector Assessment Program: Banking Sector Stress Testing – Technical Note July 2015 FSAP-S IMF Request for Extended Arrangement under the Extended Fund Facility – Press Release; Staff Report; and Statement by the Executive Director for BiH (Key Measures; Structural Benchmarks) September 2016 EFF WB/IMF Financial Sector Assessment – BiH June 2015 FSA EU Reform Agenda for Bosnia & Herzegovina 2015-2018, together with Reform Agenda Implementation – Proposal for Phase II Priorities 2015, 2017 EURA 73 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX VIII: RECOMMENDATIONS FROM DQA SESSION WITH FINRA INDICATORS RECOMMENDATIONS STATUS Number of IMF/EU Financial Stability Compliance Requirements significantly advanced toward fulfillment with FINRA assistance 1. FINRA should consider including in PIRS definition the explanation that IMF will review the stakeholders’ self￾assessments. 2. FINRA should make an assessment for planning for future years, as many stakeholders are involved in these interventions, and there is also a pressure from IMF and other institutions to work on these requirements. Not addressed Not addressed Number of Standards addressing IMF/EU Compliance Requirements and other areas of financial stability drafted with FINRA assistance and adopted by BiH authorities OPTION 1 1. FINRA should consider revising the definition in PIRS, to tight the definition to be more precise. They should consider counting only regulations or in fact regulations. 2. FINRA should consider revising targets as they will count standards as such. 3. FINRA should make an assessment for planning for future years OPTION 2 1. FINRA should count here broader standards, keep the targets as they are now, but revise the actual. 2. FINRA should make an assessment for planning for future years Addressed Addressed Not addressed Not addressed Not addressed Number of institutional improvements documented by FINRA and proposed to Counterpart government agencies OPTION 1 1. FINRA should consider revising the definition in PIRS, to explain that they will count all improvements which can occur in different stages in the same area. 2. FINRA should consider revising targets as they will count improvements as such. OPTION 2 1. FINRA should count here broader improvements, improvements that happen in one institutional area, keep the targets as they are now, but revise the actual. Addressed Addressed Not addressed Number of staff of Counterpart agencies and financial institutions trained with FINRA support 1. As there were many trainings that were not planned at the beginning, FINRA should adjust the targets for future years. Addressed Hours of training delivered to Counterpart agencies and financial institutions staff with FINRA support 1. PIRS should specify more clearly how many hours should have a specific intervention to be counted under this indicator. 2. As there were many trainings that were not planned at the beginning, FINRA should adjust the targets for future years. 3. FINRA should consider adjusting the definition of this indicator, where they will count only hours delivered, without multiplying with number of participants. The number of participants is counted under indicator Number of staff of Counterpart agencies trained. Not addressed Addressed Not addressed 74 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX IX: LIST OF COMPLIANCE REQUIREMENTS RELEVANT FOR FINRA # Compliance Requirement Agencies Final assessment results (averages) Bank Safety and Soundness Y1 Y2 1 Strengthen oversight of banks by the entity banking agencies FBA, BARS N/A 3.7 2 Assure that banks continue working toward full compliance with International Financial Reporting Standards (IFRS) FBA, BARS 3.3 4.0 3 Encourage conservative assumptions by banks; issue prudential guidance to promote adequate provisioning FBA, BARS 3.3 4.3 4 Implement the liquidity coverage ratio (LCR); fine tune parameters imbedded in the LCR and Net Stable Funding Ratio (NSFR) as to inclusion of “required reserves” in High￾Quality Liquid Assets (HQLA); revise the liquidity ratio FBA, BARS 2.3 3.0 5 Add macroprudential analysis; select and calibrate the appropriate instruments to address systemic risk FBA, BARS N/A 2.8 6 Identify ultimate beneficiary owners; enhance disclosure on group structure of banks and insider lending; transfer of significant ownership, major acquisitions FBA, BARS N/A N/A 7 Deepen the assessment of fit and proper FBA, BARS N/A 3.0 8 Improve stress testing; conduct bottom-up stress tests at least once per year FBA, BARS N/A 3.7 Problem Bank Management Y1 Y2 9 Adopt amendments to the Law on Deposit Insurance; adapt regulation DIA 3.0 N/A 10 Require all banks to have recovery plans; firm and timely action to deal with problem banks; develop a broader range of measures to expedite corrective action FBA, BARS 2.3 4.7 11 Establish resolution plans for all Systemically Important Banks (SIBs); facilitate restructurings and debt resolution as well as adoption of out-of-court restructuring guidelines FBA, BARS 2.3 4.3 12 Develop banking agency organizational structures emanating from the new bank resolution mandate FBA, BARS N/A 5.0 13 Develop cross-border bank crisis resolution coordination arrangements for all SIBs FBA, BARS N/A N/A 14 Broaden the scope to draw on the DIF for funding bank resolution; assess the availability of suitable backup facilities given DIA’s potential expanded role DIA N/A 3.5 15 Enable prompt depositor payout DIA 3.0 4.5 Central Banking Y1 Y2 16 Strengthen cooperation, coordination and information exchange among regulatory agencies, strengthen cooperation among financial safety net players under the Standing Committee for Financial Stability (SCFS) and Coordination for Banking Supervision group (CBS) CBBH, FBA, BARS, DIA 2.3 3.2 17 Improve national-level oversight of systemic risks; define the tools to be used for the identification of systemic risk CBBH 2.5 3.5 75 | FINRA PERFORMANCE EVALUATION USAID.GOV define and collect additional information for the assessment of vulnerabilities 18 Improve the scope and exchange of information on key variables important for systemic liquidity N/A 4.0 19 Strengthen, regularly review, and test bank contingency plans of SCFS members to harmonize domestic cooperation and information exchange; conduct crisis simulation exercise to test cooperation DIA, FBA, BARS, CBBH, MoFs N/A N/A 20 Establish a separate risk management function, independent of operational units, with a direct reporting line to the Governing Board CBBH 2.0 4.5 76 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX X: RECOMMENDATIONS FROM IMF FSAP/FSSA 2015 RECOMMENDATIONS AND AUTHORITY RESPONSIBLE FOR IMPLEMENTATION STATUS Banking Oversight Develop a remedial action program focusing on new tools, earlier step-up enforcement and heavier fines to expedite corrective action (FBA, BARS, relevant Ministries of Finance and Justice). Addressed by FINRA Strengthen provisioning under IAS by issuing standards to encourage conservative assumptions on impairment by banks (FBA, BARS). Addressed by FINRA Enact new Laws on Banks and amend relevant legislation addressing deficiencies in supervisory powers, consolidated supervision, and identification of ultimate beneficiary owners (FBA, BARS, CBBH, Ministries). Completed prior FINRA Conduct additional AQRs in banks with weak solvency and liquidity indicators (FBA, BARS). Completed prior FINRA Assure that banks continue implementing IFRS and external auditor implement IAS (FBA, BARS). Addressed by FINRA Insurance Oversight The appointment of the FBiH-ISA director should be based on the relevant law (Government). Completed prior FINRA The new insurance law in the FBiH should be approved if it shows improved convergence towards the EU insurance directives (FBiH-ISA). Completed prior FINRA Introduce a formal channel of information sharing with banking agencies (FBiH-ISA). Not addressed Update solvency regime by a gradual incorporation of risk elements, develop an early warning system, including prompt corrective actions. Capacity building is required (ISAs). Not addressed Financial Safety Net, Resolution of NPLs, and Systemic Liquidity Management Revise the reserve requirements, maturity mismatch, and the liquidity ratio (CBBH, FBA, BARS). Completed prior FINRA Design and implement the LCR, adapted to BiH (FBA, BARS). Completed prior FINRA Revise the consequences to banks for not complying with reserve requirement (CBBH). Completed prior FINRA Enable prompt depositor pay-out (DIA). Addressed by FINRA Provide resolution powers to the FBA and BARS for banks in their respective jurisdictions on the new Laws on Banks, following the FSB Key Attributes and EU BRRD (FBA, BARS, DIA, relevant Ministries) Addressed by FINRA Broaden the scope to draw on the existing DIF for funding bank resolution using least-cost solution (DIA) Addressed by FINRA 77 | FINRA PERFORMANCE EVALUATION USAID.GOV Establish a Financial Stability Fund (FSF)—under the DIA—to provide for open bank resolution and limited and temporary liquidity support in systemic crisis (DIA, CBBH, FBA, BARS, Governments). Not addressed Strengthen, regularly review, and test bank contingency plans of SCFS members (SCFS). Addressed by FINRA Strengthen the cooperation between DIA and financial safety net players (DIA, CBBH, FBA, BARS). Addressed by FINRA Add macroprudential analysis and policy to the coordination MoU. Define and collect additional information for the assessment of vulnerabilities and macroprudential tools (CBBH, FBA and BARS). Addressed by FINRA Streamline collateral execution procedures by allowing a final auction at no reserve price if previous rounds of auctions failed and specify realistic criteria for asset market values (RS/FBiH Ministry of Justice). Not addressed Consider tools and incentives to facilitate restructurings and debt resolution as well as adoption of out- of-court restructuring guidelines. (RS/FBiH/BiH MoF, FBA, BARS). Addressed by FINRA Revise the insolvency framework by introducing incentives to initiate proceedings early and expand the insolvency framework to cover businesses run by individuals. (RS/FBiH MoJ). Not addressed Financial Market Infrastructure Strengthen the legal framework to designate payment systems, and to protect settlement finality and netting in line with international standards (CBBH). Completed prior FINRA Develop a comprehensive risk management framework for the RTGS system (CBBH). Completed prior FINRA Stress test the RTGS system, including the default of the largest participant and affiliates (CBBH). Completed prior FINRA Establish a recovery time objective for the RTGS system following disruptive events (CBBH). Completed prior FINRA Establish a new oversight unit in Payment Systems Department, which is staffed with sufficient resources to carry out oversight responsibilities (CBBH). Completed prior FINRA Macroprudential Policy Define and collect additional information for the assessment of vulnerabilities in the financial system and the design of macroprudential tools (CBBH, FBA, BARS) Addressed by FINRA Define the tools to be used for the identification of systemic risk, select and calibrate the appropriate instruments to address, and implement (by the corresponding institution) Addressed by FINRA *Source: FINRA MEL Plan 78 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX XI: FINANCIAL SECTOR PRIORITIES IDENTIFIED THROUGH KIIs AND ONLINE SURVEYS STAKEHOLDER RECOMMENDATION Beneficiaries  Establishment of a restructuring fund  Expanding the cooperation and support to other regulators of BiH financial sector, for example RS Committee for Coordination of Supervision of the Financial Sector  Continuous capacity building through training  Conducting extensive analyses focusing on risk assessments of companies and population as the two main economic subjects  Creating activity that would increase the transparency of the economy, examine liquidity and debt of companies and households  Continuous and efficient IT support  Software upgrading  Planning of stress and unexpected scenarios, stress testing  Supporting the Investment Committee through education, expanding the horizons regarding the investments of new instruments  Education on preventing money laundry  Financial education on money transfer, educating the population, especially diaspora on making money transactions which are not physical  Financial inclusion and education  Continuous IT applications development and money laundry education  Complete implementation of international and EU standards and strong software support  Cooperation with the non-governmental sector  Revision of the IT systems in BiH banks  Improving the financial stability, evaluation of human resources  Developing the methodologies for systemic important banks  Developing monetary instruments of CBBH  Further development of research and forecasting function in CBBH International organizations  Continuation of bringing Bosnia and Herzegovina in line with European Union standards  Adopting SREP building the capacity in terms of macro-economic analyses and micro forecasting  Higher-level donor cooperation, co-funding, and co-implementation of projects  More intensive bank monitoring, paying attention to the weaker banks because of the pandemic  Capacity building of local employees, education through study visits, funding these study visits of the employees  More proactive approach  Supporting the Deposit Insurance Agency in recognizing early warning signals in the banks  Even closer cooperation with banking agencies  Comprehensive midterm analyses about raising the insured deposits, the number of insured deposits, based on all relevant positions or variables Consultants  Increasing the capacity of the Central Bank by increasing the number of people dealing with financial sector issues, enlarging the department that challenges the supervisors, creating a variety of perspectives and data  Establishment of a restructuring fund  Amending the Bosnian banking Act in a way that it should provide a legal basis for the changes in regulation framework, amended in line with banking directives in European Union  Supporting the agencies in the development of their strategies  Pushing the banking agencies towards using the SREP methods  Constant improvement in regulations, monitoring the EU behavior and solutions, being compliant with these behaviors and regulations  Aiding the agencies with periodical analyses of the banking sector regarding the liquidity and solvency  Improving cyber security and operational financial risks procedures  Prioritizing the financial stability questions and improving the cooperation between the two banking agencies  Merging the BiH insurance sector into a bigger stock exchange  Developing the insurance and security markets 79 | FINRA PERFORMANCE EVALUATION USAID.GOV  Designing the activity focusing on the fiscal sector (FINRA C3 should have been a separate activity  Centralization of database regarding the bank supervision and tracking  Aiding BiH towards adopting EU regulations  Creating the resolution fund  Establishing a technical secretariat as a strong support for coordination in the frame of financial sector architecture and financial stability  Reaching full compliance with relevant restructuring regulation (BRRD2, including establishment of bank resolution fund)  Reaching significant improvement of inter-agency cooperation within financial stability architecture (in particular, strengthening role of Standing Committee for Financial Stability and Banking Coordination, supported by modernized version of comprehensive Memorandum of Understanding)  Improving communication of all agencies with banks and general public  Reaching implementation of macro-prudential policies within coordinated setting  Bringing the Law on Deposit Insurance in full compliance with EU Directive (DGSD)  Broadening scope of financing instruments for deposit insurance fund  Reaching implementation of risk-based deposit insurance premium Non-beneficiaries  Focusing on the areas in which the BiH banking sectors are not harmonized with the EU regulatory framework  Implementing the macro prudential instruments in an appropriate manner  Digitalization of banks, especially in the cyber-criminal part  Finalizing the equivalence process- completely aligning the BiH regulations with EU regulations Banks via Bank Association  Focusing on the areas in which the BiH banking sectors are not harmonized with the EU regulatory framework  Implementing the macro prudential instruments in an appropriate manner  Digitalization of banks, especially in the cyber-criminal part  Introduction of a "two-tier" system for collection of negative fee for free funds with the CBBH above the Reserve Requirement  Complete abolition of the Maturity Compliance Limit on financial assets and liabilities  Introduction of REPO transactions with the Central Bank  FX EUR/BAM SWAP transactions with the Central Bank  Relaxation and enabling the purchase/sale of BAM/EUR regardless of the time period  Limits to affiliated banks (Decision on large exposures, Decision on calculation of capital)  To increase the limit from 5% to 10% for collateral-free exposures  To ensure that differences in prudential provisions are reduced through capital rather than through RDIG  To extend the deadlines in which placements without collateral in the non-income portfolio must be 100% reserved and equal to the EBA regulations (calendar provisioning of the 2nd year)  To abolish the recovery period from Stage 2 to Stage 1 (same as not defined by EBA regulations)  To equate the recovery period from the non-income to the income portfolio with the recovery period as defined by the EBA regulations 80 | FINRA PERFORMANCE EVALUATION USAID.GOV ANNEX XII: DISCLOSURE OF CONFLICT OF INTEREST FOR EVALUATION TEAM MEMBERS Name Damir Odak Title External Expert Organization USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II) Evaluation Position? Team Leader Team member Evaluation Award Number USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II), implemented by IMPAQ International, LLC, Contract Number: AID-167-I-17-00004 USAID Project(s) Evaluated USAID/BiH Financial Reform Agenda Activity (FINRA), implemented by Financial Markets International, Inc. (FMI), Contract Number: AID-168-C-17-00003 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 9, 2020 81 | FINRA PERFORMANCE EVALUATION USAID.GOV Name Anela Kadić Abaz Title Deputy Chief of Party Organization USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II) Evaluation Position? Team Leader Team member Evaluation Award Number USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II), implemented by IMPAQ International, LLC, Contract Number: AID-167-I-17-00004 USAID Project(s) Evaluated USAID/BiH Financial Reform Agenda Activity (FINRA), implemented by Financial Markets International, Inc. (FMI), Contract Number: AID-168-C-17-00003 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 9, 2020 82 | FINRA PERFORMANCE EVALUATION USAID.GOV Name Jasmina Mangafić Title External Expert Organization USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II) Evaluation Position? Team Leader Team member Evaluation Award Number USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II), implemented by IMPAQ International, LLC, Contract Number: AID-167-I-17-00004 USAID Project(s) Evaluated USAID/BiH Financial Reform Agenda Activity (FINRA), implemented by Financial Markets International, Inc. (FMI), Contract Number: AID-168-C-17-00003 I have real or potential conflicts of interest to disclose. Yes x No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 9, 2020 x x 83 | FINRA PERFORMANCE EVALUATION USAID.GOV Name Haris Mešinović Title Senior Research Analyst Organization USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II) Evaluation Position? Team Leader Team member Evaluation Award Number USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II), implemented by IMPAQ International, LLC, Contract Number: AID-167-I-17-00004 USAID Project(s) Evaluated USAID/BiH Financial Reform Agenda Activity (FINRA), implemented by Financial Markets International, Inc. (FMI), Contract Number: AID-168-C-17-00003 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 19. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 20. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 21. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 22. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 23. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 24. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 9, 2020 84 | FINRA PERFORMANCE EVALUATION USAID.GOV Name Amela Kurta Title Analyst Organization USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II) Evaluation Position? Team Leader Team member Evaluation Award Number USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II), implemented by IMPAQ International, LLC, Contract Number: AID-167-I-17-00004 USAID Project(s) Evaluated USAID/BiH Financial Reform Agenda Activity (FINRA), implemented by Financial Markets International, Inc. (FMI), Contract Number: AID-168-C-17-00003 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 25. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 26. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 27. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 28. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 29. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 30. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 9, 2020 85 | FINRA PERFORMANCE EVALUATION USAID.GOV Name Amer Čekić Title Analyst Organization USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II) Evaluation Position? Team Leader Team member Evaluation Award Number USAID/BiH Monitoring and Evaluation Support Activity (MEASURE II), implemented by IMPAQ International, LLC, Contract Number: AID-167-I-17-00004 USAID Project(s) Evaluated USAID/BiH Financial Reform Agenda Activity (FINRA), implemented by Financial Markets International, Inc. (FMI), Contract Number: AID-168-C-17-00003 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 31. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 32. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 33. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 34. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 35. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 36. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date July 9, 2020 MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE II) Fra Anđela Zvizdovića 1 UNITIC Tower B, Floor 21 71000 Sarajevo Bosnia and Herzegovina Phone: +(387) 033 941 676 contacts@measurebih.com www.measurebih.com