October 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared by an independent team consisting of David T. King (Team Leader), Nancy Fronczak, and Jasmina Mangafić. PERFORMANCE EVALUATION REPORT: BOSNIA & HERZEGOVINA PARTNERSHIP FOR ADVANCING REFORMS IN THE ECONOMY (PARE) ACTIVITY USAID PARE Evaluation Report PERFORMANCE EVALUATION: USAID BOSNIA & HERZEGOVINA PARE ACTIVITY PARTNERSHIP FOR ADVANCING REFORMS IN THE ECONOMY (PARE), IMPLEMENTED FROM APRIL 2008 THROUGH SEPTEMBER 2013 FINAL REPORT January 7, 2014 Evaluation Order Number: AID-168-O-13-00041 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID PARE Evaluation Report CONTENTS I. Executive Summary...................................................................................................................... 1 Evaluation Purpose and Evaluation Questions .................................................................................... 1 Activity Background..................................................................................................................... 1 Evaluation Design, Methods and Limitations ...................................................................................... 2 Findings, Conclusions, & Recommendations ...................................................................................... 2 II. Evaluation Purpose & Evaluation Questions ....................................................................................... 6 Evaluation Purpose ...................................................................................................................... 6 Evaluation Questions ................................................................................................................... 7 III. Activity BackgrounD.................................................................................................................... 8 General Background and Development Purpose................................................................................. 8 Status at Activity Outset: Banking Supervision.................................................................................... 8 Status at Activity Outset: Deposit Insurance...................................................................................... 9 IV. Evaluation methods & limitations.................................................................................................. 10 Evaluation Design and Methods .................................................................................................... 10 A. Rationale for methods........................................................................................................ 10 B. Data sources and collection methods and tools........................................................................ 10 C. Data Analysis ................................................................................................................... 11 Evaluation Limitations................................................................................................................. 12 V. Findings, Conclusions, & Recommendations..................................................................................... 14 Question 1: Banking Supervision .................................................................................................. 14 Background to Activity Objectives in Banking Supervision ............................................................... 14 Banking Supervision: Findings.................................................................................................... 15 Banking Supervision: Conclusions ............................................................................................. 19 Question 2: Deposit Insurance .................................................................................................... 20 Background to Activity Objectives in Banking Supervision ............................................................... 20 Deposit Insurance: Findings...................................................................................................... 21 Deposit Insurance: Conclusions ................................................................................................ 22 Question 3: Obstacles............................................................................................................... 23 Obstacles to Implementation: Findings........................................................................................ 23 Obstacles to Implementation: Conclusions .................................................................................. 25 - - i USAID PARE Evaluation Report Question 4: Donor coordination ................................................................................................. 26 Donor Coordination: Findings .................................................................................................. 26 Donor Coordination: Conclusions............................................................................................ 27 Question 5: Recommendations.................................................................................................... 27 - - ii USAID PARE Evaluation Report ACRONYMS BARS Banking Agency of Republika Srpska BiH Bosnia & Herzegovina BSA Banking Supervision Agency CBBH Central Bank of Bosnia & Herzegovina CRD Capital Requirements Directive (EU) DIA Deposit Insurance Agency of Bosnia & Herzegovina EBA European Banking Authority ECB European Central Bank FBA Federation of BiH Banking Agency FBiH Federation of Bosnia & Herzegovina FSAP Financial System Stability Assessment Report (IMF) IADI International Association of Deposit Insurers IMF International Monetary Fund KfW German Development Bank (Kreditanstalt für Wiederaufbau) KI Key Informant KII Key Informant Interview MoFT BiH Ministry of Finance and Treasury NPL Non-Performing Loan PFS Partners for Financial Stability (USAID regional project) PMI Performance Management Indicator ROSC Report on the Observance of Standards and Codes (IMF, World Bank, OECD) RS Republika Srpska SBA Stand-By Agreement (IMF) UBBiH Association of Banks of BiH (Udruženje Banaka Bosne i Herzegovine) WB World Bank - - iii USAID PARE Evaluation Report I. EXECUTIVE SUMMARY This is a performance evaluation of the USAID Partnership for Advancing Reforms in the Economy activity (PARE, or “the Activity”), implemented in Bosnia & Herzegovina from April 2008 through September 2013, with a total budget of $9.6 million. The Activity was set up with a two-year base period, followed by a two-year option period, and finally an 18-month extension period. Evaluation Purpose and Evaluation Questions The basic purpose of this PARE evaluation is to inform USAID and other reviewers as to the extent to which the Activity accomplished its specific results objectives and achieved development impact. In addition, the evaluation provides recommendations as to gaps and priorities for further attention in BiH financial sector development. This performance review and recommendations may be used by USAID and other donors to inform decisions as to future interventions. Depending on policies as to dissemination, the audience for this evaluation includes, in addition to USAID, counterpart government agencies in BiH, counterpart private sector institutions in BiH, major supporters of BiH financial sector development including the World Bank, the IMF, and KfW, as well as the public. The evaluation focused on the two primary areas of focus of the PARE activity: advancing systems, practices, and professional capabilities for banking supervision, and for deposit insurance. Specifically, it was charged with addressing five key questions: (1) To what extent has PARE contributed to the Banking Supervision Agencies (BSAs) abilities to move toward gradual implementation and enforcement of a regulatory framework that is compliant with Basel II/III and relevant EU Directives? (2) (a) To what extent has PARE contributed to improving the overall capacity of the BiH deposit insurance agency? (b) To what extent has the PARE activity contributed to the BiH deposit insurance agency’s compliance with International Association of Deposit Insurers (IADI) standards? (3) What were the main obstacles to implementing PARE and how did the activity address them? (4) How well was PARE’s implementation coordinated/synchronized with other donors’ activities aimed at helping BiH more effectively supervise and maintain the stability of the financial system in BiH? (5) Based upon the expert opinions of PARE counterparts and key stakeholders, what priority reforms in the BiH financial services sector (specifically banking) need to be addressed? Activity Background The broad background for the PARE activity was the long history of support that USAID had provided in establishing sound systems for banking supervision and deposit insurance in BiH. In the highly coordinated donor assistance environment that was set up at the conclusion of the war of the early 1990s, USAID took primary responsibility for providing technical and material assistance for banking supervision through the two entity banking supervision agencies (BSAs), the Federation Banking Agency (FBA) and the Banking Agency Republika Srpska (BARS). USAID provided continuous support to the BSAs for a decade thereafter. Similarly, USAID, along with KfW, contributed the initial capital for the BiH deposit insurance system, insuring its independence, and delivered intensive technical assistance to - - 1 USAID PARE Evaluation Report the Deposit Insurance Agency (DIA). In addition, USAID provided technical assistance in various areas to the Central Bank of Bosnia & Herzegovina (CBBH). However, given the ongoing evolution of international standards and best practices in these areas, by the time the Activity was planned, supervisory authorities in both banking and deposit insurance were falling behind. They requested continued assistance from USAID. The PARE activity was designed to be demand-driven, responding to these clear and pressing needs. Evaluation Design, Methods and Limitations This evaluation was carried out by a team of three independent consultants, who spent approximately two weeks in desk review and two-and-one-half weeks in field work in August-September 2013, as the Activity was ending. The evaluation process was designed to triangulate information, in order to independently and objectively validate findings and conclusions. It was based on three primary data sources: (i) Documents: Relevant reports were available from a wide range of sources: the Activity’s own progress reports, the technical assessments and reports of the Activity’s expert consultants, reviews and evaluations from other donors working in related areas (principally the IMF), and documents and reports of the principal counterpart agencies – the two BSAs, the DIA, and the CBBH. These separate sources enabled triangulation/validation within the document set itself. The Evaluation Team reviewed over 500 pages from 56 different documents (Annex 1, Table 5). (ii) Interviews with Key Informants (KIs): Key informants were interviewed one-on-one or in groups within an institution, in almost all cases with at least two Evaluation Team members present. KIs were drawn from all four of the principal agency counterparts, and included their chief executives (for the two BSAs and DIA) or senior vice presidents (for the CBBH). They also came from the Activity itself (permanent professional staff and short-term technical advisor), the private sector (individual banks, the BiH Association of Banks, a major accounting firm, and independent financial professionals), and USAID and other donors connected to financial sector development. The Evaluation Team conducted 25 separate KI field interviews, consisting of a total of 33 individuals (Annex 1, Table 1). (iii) Survey of trainees: Because one of the most important purposes of the Activity was to build professional capabilities in the counterpart agencies for understanding, tailoring, and implementing advanced supervisory requirements, independent feedback from participants in PARE capacity-development activities was essential for triangulation and validation of effectiveness. A survey to gauge the quality and job usefulness of PARE trainings was distributed to all identified participants, of whom a representative sample of over one-third responded. Information from these three principal sources was collated by the Evaluation Team and reviewed in daily team meetings. Provisional findings were developed and discussed, in the end producing full agreement within the team on findings, conclusions, and recommendations. Findings, Conclusions, & Recommendations The principal findings and conclusions of the Evaluation Team, organized according to the five key questions, are as follows: Question 1: Progress in Banking Supervision Findings - Accomplishments - - 2 USAID PARE Evaluation Report With the encouragement and assistance of the Activity, the two BSAs developed parallel strategies for advancing toward and eventually implementing Basel II, with further adjustments for Basel III and advanced EU directives. The Activity also delivered 58 capacity-development events related to banking supervision, and professional capacities in bank supervision were significantly strengthened over its course. PARE provided expert and appropriate short-term technical consultants to the BSAs. The Activity helped design and implement a range of institutional improvements within the BSAs and in private sector support for them to advance progress toward the standards. Finally, PARE was instrumental in preparing a significant number of “building blocks” of the standards framework, which were either in an advanced state of readiness or actually implemented by the end of the Activity. Findings – Issues The Activity initially had a very broad mandate, covering a number of financial subsectors, and was slow in getting to the point of rapid accomplishments in banking supervision, most of which occurred in its last 18 months (the extension period). It faced an extremely turbulent operating environment almost as soon as it began, when the global financial crisis hit, which quickly moved the Activity goalposts with respect to regulatory regime well beyond where they were when it started. Furthermore, the BSAs initially had some misunderstanding of how PARE would work – whether simply as a consulting service to them, or a trainer/facilitator, or some combination. PARE fell well short of its PMI targets with respect to actual adoption of regulations, which though not in its actual power led to a period of intense pressure on the BSAs from the Activity to get laws and decrees passed and gazetted. This pressure created frictions both within the Activity team and between the Activity and the BSAs (particularly the FBA). In addition, the objective of a unified State-level banking supervision agency was unachieved and abandoned, due to intense political opposition to it. Also, with respect to the PMI related to number of supervisors trained, the results reported are misleading if they are taken as unique individuals instead of simply the sum of attendees, since most officials attended multiple trainings. Conclusions on Banking Supervision The BSAs achieved substantial progress toward the advanced banking supervision standards framework over the course of the Activity, and much of this progress is attributable to PARE’s efforts. Given the importance of the advanced standards framework for financial sector stability and SME access to finance in BiH, and for eventual accession to the EU, we conclude that the overall development objectives of PARE were in banking supervision were to a significant extent fulfilled. Through its capacity-building activities, PARE contributed substantially to the knowledge and readiness of the professional personnel in the two BSAs to implement the advanced standards framework. BSA staff now have the know-how to proceed effectively. While concrete progress during the base and option periods was slow, there were reasons for this outside the control of the Activity, including the conservative attitude of the BSAs engendered by the global financial crisis, and their preoccupation with rapidly rising NPLs at BiH banks in its wake. Strong pressure for adoption of decrees from the Activity led to pushback, from the larger and more bureaucratic FBA at least, that may have delayed progress for a short period. However, we believe that the pressure was justified, and may have helped stimulate the substantial progress seen in the extension period. The fact that the Activity’s PMIs as to actual adoption of standards clearly were not met raises the issue of the appropriateness of the measurement indicators themselves, since it is in fact not within the power of any donor project to actually adopt reforms. Question 2: Progress in Deposit Insurance Findings - Accomplishments - - 3 USAID PARE Evaluation Report PARE identified serious problems with the DIA’s investment policies and practices at the beginning of the activity and provided technical assistance to substantially improve them. Formal investment guidelines were adopted and much closer management and board oversight of Reserve Fund investments was implemented. In addition, the DIA’s model for establishing deposit insurance system financial parameters was updated and improved. The Activity implemented a number of capacity-building activities that strengthened DIA staff and management capacities. It also provided extensive technical assistance in preparing a risk-based deposit insurance premium framework, including developing and testing an analytical model and fully specifying the software application required for eventual implementation of risk-based premia. Finally, a comprehensive assessment of the BiH deposit insurance system’s compliance with IADI standards was carried out, and gaps were identified and addressed, fully or as much as they could be without additional legal reforms beyond the authority of the DIA. Findings – Issues Despite the progress achieved toward the risk-based premium regime, and the fact that it was an explicit PARE objective, its actual implementation was not possible, for two reasons outside the direct mandate or influence of the Activity – a lack of uniform application of international accounting standards on the part of BiH banks, and a lack of clarity in the two entity banking laws as to the process of resolution and recovery of a failed bank. Required amendments to the deposit insurance law were drafted and adopted, but those for the entity banking laws, while drafted, were not passed. Conclusions on Deposit Insurance For almost all of its objectives with respect to deposit insurance, PARE’s assistance was instrumental and successful. It contributed strongly to development of professional capacities within the DIA, and delivered a range of legal/regulatory elements and operating tools necessary for a sound, IADI compliant deposit insurance system. It is probable that PARE’s support for improving the Reserve Fund’s investment policies and practices saved it from suffering significant losses during the financial crisis, which could have damaged public confidence in the entire deposit insurance system. As was the case for banking supervision, the DIA has intensive day-to-day responsibilities that make it difficult for it to plan and prepare systemic improvements. PARE served DIA as a critical outside resource for these purposes, and it we believe it would not have been possible for it to make the substantial progress that it did over this period without the Activity. Question 3: Obstacles to Progress Findings PARE faced a lack of will on the part of counterparts to move aggressively in addressing changes advocated by the IMF and PARE, particularly in banking supervision. As noted above, there was a complete lack of political will to move toward unified supervision, and indeed, several KIs thought that the independence from politics of the BSAs and the CBBH had deteriorated over the past several years. Some thought that the CBBH in particular had failed to take on responsibilities, which would have helped unify financial sector oversight in some areas, that it could have under its legal mandate. In addition, the BSAs displayed hesitancy due to a lack of staff understanding in the earlier years of the Activity, and due to conservatism about bank safety and soundness given the impact of the global financial crisis on bank loan quality. PARE responded by focusing on technical issues, at the technician level, and organizing the great majority of its capacity-development events with representation across relevant public and private sector counterparts. Slowness in forward movement was also related to the daily workload of BSA staff, much of whose time is taken up in ongoing regular offsite and onsite supervisory responsibilities. The BSAs are both supervisors and regulators, and the demands of the supervisory function tend to crowd out the critical capacity - - 4 USAID PARE Evaluation Report development and planning work involved in advancing the regulatory system. In response, PARE’s services essentially filled this role over the course of the Activity. The fact that PARE itself had no ability to actually adopt legal changes or decrees was an obstacle that is inherent in a activity of this type, even though actual adoption is its stated goal. In response, the Activity drafted and designed building blocks of the system, encouraged with some success overall financial sector oversight in the State-level MoFT at least at the technician level, supported private sector advocacy for change within the banking industry, as well as public awareness, to a limited extent. Conclusions on Obstacles In general, PARE adapted its strategies to address these obstacles. PARE’s role as an outside resource for developing and delivering trainings, organizing subject-specific teams, bringing representatives from all relevant areas together, and preparing drafts and tools for specific elements of the advanced regulatory framework, helped greatly to address the day-to-day supervisory workload problem faced by the BSAs and DIA, to keep things moving forward. This is one of the key reasons that overall progress was achieved over the period. USAID’s role, through PARE, as an external, independent force for change also provided screening from politics and created pressure on the BSAs and DIA to move forward. There is more scope for encouraging public pressure for accelerating movement toward EU alignment in general and greater financial stability access in particular, through “social marketing” strategies and improved small business and consumer awareness and education. Question 4: Donor Coordination Findings USAID has always played a lead role in advancing banking supervision and deposit insurance in BiH, an outgrowth of original donor coordination structures established in the wake of the late war. As a result, there are few organizations working in the financial sector in BiH, and no other on-the-ground projects to support the banking supervision and deposit insurance agencies in meeting Basel and EU financial standards. However, the Activity’s baseline was defined as the IMF’s 2006 FSAP, and the IMF and World Bank in particular were regularly informed of PARE’s progress. KfW, which had originally collaborated with USAID in capitalizing the deposit insurance fund, was also kept abreast of developments in that area. PARE also collaborated on occasion with other USAID activities, in particular the Partners for Financial Stability (PFS), a regional project, in complementary work with the DIA. Regionally, there was extensive collaboration with the Center of Excellence in Finance (CEF) in Slovenia, and, by making available key short-term technical experts, the Central Bank of Slovenia. Conclusions on Donor Coordination There was little overlap or duplication between PARE’s activities and those of other donors. The Activity had contacts and information-sharing with all identified donors or other external groups relevant to supporting BiH movement toward target international standards in banking supervision and deposit insurance issues. Its work was fully harmonized with the recommendations of the IMF and World Bank. Question 5: Recommendations We have four principal recommendations: 1. Is a follow-on to PARE needed? Though the value of a USAID “presence” is regularly cited by counterparts, the real contribution of the PARE activity was as an outside resource for the BSAs and DIA, who are consumed by their day-to-day supervisory tasks, to manage and press forward the - - 5 USAID PARE Evaluation Report complex regulatory reform program and the external expertise and professional capacity development it requires. This need will certainly continue, but we believe that thanks to the capacities developed by PARE, it does not require another U.S.-based contractor to manage. USAID can provide support through a FORWARD-consistent conditioned contract or grant through the CBBH to engage a local team to serve as a secretariat to the counterparts in carrying the program forward to full implementation. The key counterparts – the two BSAs, the DIA, and the CBBH – would be expected to share in the costs of such a activity. 2. The BiH financial system is entirely dominated by commercial banks, and in turn these are dominantly foreign-owned. This leaves SMEs in BiH with a narrow range of options for business finance. USAID should consider a activity focused actively on significantly broadening SME access to finance by developing the nonbank financial sector. This could include seed funding, innovative DCA guarantees, and public-private partnerships, to stimulate a wide range of nonbank financial sources – equipment leasing, factoring, targeted investment pools, and in capital markets, corporate/mortgage/ SME-fund bonds, insurance, and pensions. Such a activity would also develop critical financial sector supporting services including individual and business credit rating services, collections services, and collateral liquidation services. 3. USAID activites need greater focus on building broad political support for EU-related reforms. Even in financial sector development, social marketing methods can explain to the public the value of regulatory reform and its connection to access to credit and job creation. 4. The fact that PARE did not achieve its specific PMIs with respect to the actual adoption of laws and regulations, even though it made great progress, calls for greater sophistication in crafting PMIs for business environment reform projects. Looked at comprehensively, adoption is not black and white. It is the final step in a process that has a number of critical prior milestones. Future activities should be designed and PMIs and workplans developed with well-defined and agreed benchmarks leading up to and including the ultimate goal of adoption/implementation. This would in fact make projects more accountable, and would avoid potentially unproductive pressure for actual adoption before adequate industry discussion, public review, and counterpart training. Also with respect to PMIs, that for number of trainees needs to be refined to distinguish participations in trainings (which can be multiple for each individual) from unique individual trainees. II. EVALUATION PURPOSE & EVALUATION QUESTIONS Evaluation Purpose The basic reason for this performance evaluation is to assist USAID in determining the extent to which the USAID PARE activity achieved its overall objectives of helping BiH adopt economic reforms and build institutional capacity on its path toward joining the EU. In particular, it seeks to evaluate the impact of the Activity in its two areas of greatest technical focus and investment: banking supervision and deposit insurance. In these areas, the aim was to support the adoption of international standards and best practices, and to build the professional capacities of agency staff so that they could understand and effectively implement these regulations and guidelines. The evaluation is being conducted at this time because the Activity, which started in April 2008, came to a close on September 25, 2013. The findings, conclusions, and recommendations from the evaluation will be used by USAID in considering and developing its further assistance programs in Bosnia & Herzegovina. - - 6 USAID PARE Evaluation Report While the primary audience for this evaluation is the USAID BiH mission, which solicited it, we believe that it is of interest to other USAID missions and donors considering projects that are focused primarily on achieving reforms in the legal and regulatory environment, whether for the financial sector or other areas. Evaluation Questions The evaluation purpose as described above is clearly expressed in the five Key Questions contained in the statement of work that it is to answer, which are well- focused and will directly inform USAID considerations and decisions. These are: (1) To what extent has PARE contributed to the Banking Supervision Agencies (BSAs) abilities to move toward gradual implementation and enforcement of a regulatory framework that is compliant with Basel II/III and relevant EU Directives? This question reflects the desired results expressed in the original RFP for PARE for the Banking Supervision area of Activity component 1: (a) that BiH is closer to full compliance with Basel II Core Principals (with progress measured against the priorities outlined in the 2006 FSAP and ROSC for BiH) Report on Observance of Standards and Codes on Banking Supervision), and (b) that Banking supervision is effectively consolidated at the State level. It was explicitly acknowledged that the second item depended on political will. Intended results in these areas were further detailed in at the time of the extension of the PARE contract. (2) (a) To what extent has PARE contributed to improving the overall capacity of the BiH deposit insurance agency? (b) To what extent has the PARE activity contributed to the BiH deposit insurance agency’s compliance with International Association of Deposit Insurers (IADI) standards? The original RFP for PARE indicated the following key desired results related to this question: (i) that the Deposit Insurance Agency’s investment strategy is revised to ensure it incorporates best practices of EU member states, (ii) that an updated model is developed and implemented regarding eligible insured deposits thresholds (to incorporate considerations of EU standards), and that (iii) institutional risk is incorporated into the calculation of deposit insurance premiums. Additional desired results were stipulated at the time of the contract extension. (3) What were the main obstacles to implementing PARE and how did the activity address them? PARE was designed to implement activities using a demand-driven approach, responding to government requests for technical support, in coordination with partners and counterparts. It is recognized that ultimately, implementing institutional change is not within the power of the Activity. However, the evaluation looks for evidence that the Activity identified when its interventions and/or their results were not achieving the desired outcomes, and that there was active and rational strategizing within the Activity to overcome the obstacles and to improve results. (4) How well was PARE’s implementation coordinated/synchronized with other donors’ activities aimed at helping BiH more effectively supervise and maintain the stability of the financial system in BiH? The evaluation looks for evidence of coordination with other donor activities throughout all years of implementation, and for evidence that Activity activities and strategies were perceived by partners as of high technical quality and in keeping with overall objectives for strengthening the financial system. - - 7 USAID PARE Evaluation Report (5) Based upon the expert opinions of PARE counterparts and key stakeholders, what priority reforms in the BiH financial services sector (specifically banking) need to be addressed? Based on analysis of findings and conclusions for evaluation questions 1-4, and additional views gathered from experts, the evaluation provides recommendations for consideration of future intervention in financial sector development and related areas of economic growth in BiH. III. ACTIVITY BACKGROUND General Background and Development Purpose The basic rationale for the PARE activity was to advance financial sector development in BiH. While a broad range of financial subsectors and institutions were covered, the primary focus was on strengthening banking supervision and deposit insurance, the subject areas of this evaluation. USAID had been the primary supporter of reform in both banking supervision and deposit insurance in BiH since 1996, immediately following the end of the late war. It had provided extensive on-the-ground technical assistance to the two entity banking agencies – the Federation Banking Agency (FBA) and the Banking Agency of Republika Srpska (BARS) – and the Deposit Insurance Agency (DIA), as well as the Central Bank of BiH (CBBH). Counterparts in BiH pointed to further unmet needs for technical assistance, and requested it from USAID given its historical efforts in these areas, which were regarded by counterparts as highly valuable and productive. PARE was designed specifically with BiH’s eventual accession to the EU in mind, following the signing of its Stabilization and Association Agreement (SAA) in 2008. Therefore, it was concerned not just with global standards for banking supervision (the Basel Accords) and deposit insurance (IADI standards), but with relevant EU Directives. The principal development purposes of the Activity were: (i) facilitation of BiH accession to the EU, (ii) strengthening the safety and stability of the BiH financial system, and (iii) fostering banking sector development in order to expand access to finance for BiH enterprises. The development hypothesis, or causal logic, began with demand-driven technical assistance to the relevant agencies to develop the professional capacities to understand advanced banking supervision and deposit insurance standards, and to draft, design, and install the legal, regulatory, and operating elements required to implement the standards. By preparing and adopting these standards, strengthening the institutions and individuals responsible for them, and actually rolling them out, PARE’s development purposes would be advanced. Status at Activity Outset: Banking Supervision When the PARE activity started, the background and environment for banking supervision was as follows: • Some elements of Basel I were in place. The BSAs did have the intention of fully achieving Basel I standards over the following years, and had in place the outline of a plan to get there. There was little or nothing of the Basel II revisions to core principles in place, nor any concrete plan for implementing them. • There had been very rapid growth in bank credit over previous 5 years – in the range of 25-30% per annum. However, there were weaknesses in loan classification practices (partly because Basel standards were not yet implemented), and bank credit quality during the boom deteriorated. As a result, nonperforming loans (NPLs) were understated, which was to magnify pressures on supervisors later. (Indeed, NPLs have continued to grow, well after the crisis – similar to the situation in other SEE - - 8 USAID PARE Evaluation Report economies – this problem is now the main focus of concern for WB/IMF reviews and technical assistance.) • Cooperation with external supervisors of foreign-owned banks was lacking, because banking law in BiH did not properly provide for confidentiality in the exchange of information on individual bank risks. • There was no unified banking supervision agency, and there was a lack of overall financial sector oversight. Banking supervision was divided between the two independent entity BSAs. The CBBH had a quasi-formal role in overseeing developments in the banking sector, without actual authority. There was no State-level body to monitor the stability of the financial system covering all of its subsectors. • In the two BSAs, the managerial/work organization was largely unspecialized, and there was a low level of professional knowledge and capacity for implementing Basel II. • Finally, the Activity was launched on the eve of the financial crisis. Virtually as soon as it had developed its initial plans, the Basel II standards that PARE was contracted to help implement were considered no longer adequate, having failed to prevent the crisis. Status at Activity Outset: Deposit Insurance For deposit insurance, the following points summarize baseline conditions when the PARE activity started: • The DIA’s investment policy was very weak. Essentially, management of deposit insurance reserves was delegated to the Frankfurt-based external fund manager, without diligent oversight from DIA management. Reserve funds, and therefore the ability pay out if necessary, were exposed to instability in asset markets. • Not all BiH banks were in the deposit insurance system. This left depositors with having to figure out if their deposits were or were not insured, and was a clear violation of IADI core principle #8, “Compulsory Membership”. • There was a lack of legal clarity in process of payout of insured deposits in case of bank failure. This is important because the adequacy of a deposit insurance system’s reserves, and therefore the premiums it charges banks, depends in part on its ability to recover assets from the failing bank. • The DIA was using an out-of-date model for insured thresholds and premium rates. The model and data standards had served since the founding of the DIA but needed updating and improvement of data standards and estimation procedures. • The DIA had no capacity or tools for risk-based premium setting, which is now the global best management practice for deposit insurance. Deposit insurance premia should vary from bank to bank depending on the assessment of their riskiness. This connects advances in deposit insurance with improved banking supervision standards. As reviewed below, PARE was able to address, fully or partly, the majority of these baseline issues, while others were beyond its ability to rectify. - - 9 USAID PARE Evaluation Report IV. EVALUATION METHODS & LIMITATIONS Evaluation Design and Methods A. Rationale for methods This was a performance evaluation that used several methods to triangulate information in order to inform the evaluation conclusions and recommendations. Data collection methods and tools were designed to provide the following: • Independent validation of information reported by the Activity in its monitoring and evaluation and periodic progress reports related to the five key evaluation questions. • Evidence that the Activity process and interventions have moved the financial sector toward EU standards in regulation, banks supervision, and deposit insurance. • Evidence of activity strategies being proactively revised to improve results, where change expected from processes and interventions did not materialize or where there were barriers to implementing strategies. • Evidence of sustainability for activities supported by PARE that are important for continued progress toward meeting EU standards and sustaining advances in the financial sector. B. Data sources and collection methods and tools 1. Documents and reports Documents from the following sources were reviewed. (See Annex 1, Table 5 for a detailed listing.) i. Activity documents: These include routine reports, special technical and/or consultant reports, and activity planning and monitoring document. ii. Government financial agency documents and websites: These included copies of official decisions, decrees and laws. iii. Documents and reports of related donor (World Bank, IMF, EU, KfW, and others) iv. Documents to provide objective evidence of reported activities (e.g., minutes from meetings, reports on workshops, media publications, training curricula and schedules, memoranda on study trips, etc). 2. Key informant interviews (KIIs) Key informant interviews were conducted with 28 persons, for in-depth information and insights on issues relevant to the evaluation. Structured questionnaires (see Annex 4) were developed for the Key Informant Interviews that followed a generic interview guide, with adaptations to each major category of key informant. The objective of the guides was to ensure focus, objectivity, consistency, and comparability of responses. The Key Informants were identified by PARE Activity, USAID, and by other interviewees. (See Annex 1, Table 1 for a list of persons interviewed). The key informants fell under the following categories: i. Counterpart Government Agencies: Informants from the Banking Supervision Agency of FBiH (FBA), Banking Agency Republika Srpska (BARS), BiH Deposit Insurance Agency (DIA), and the Central Bank of BiH (CBBH) were interviewed. Group meetings were held with informants from the CBBH, BARS, and DIA. (16 informants were interviewed in 6 different meetings). ii. PARE Professional Staff: Key technical staff of the Activity, including the Chief of Party (4 - - 10 USAID PARE Evaluation Report informants). iii. PARE Technical Advisors: Consultants who provided technical assistance to the activity in development of elements of the reform framework and in capacity-building (4 informants). iv. Other Donors: Representatives of the IMF and the World Bank (2 informants). v. Banks: Representatives from the banking sector (including the Association of Banks) who must implement advanced bank supervision and deposit insurance standards (3 informants). Almost all key informant interviews were conducted with at least two Evaluation Team members present, both of whom took notes independently. Most interviews were tape recorded (with the permission of the interviewees) to enable the evaluation team to recheck information for clarifications if needed. Survey of government participants in activity capacity-building events A structured survey form (see Annex 4) was developed to gather information from participants on the number of PARE events they attended, and a self-assessment on the degree to which they felt their capacity had been strengthened in various topics related to risk-based supervision, EU standards, and DIA compliance with IADI guidelines. The survey form was shared with one FBA and one BARS Key Informant for comments and suggestions prior to finalization. It was then translated into Bosnian. Survey sample: Among the approximately 600 contacts from PARE sponsored/supported capacity building events, the evaluation team identified 121 unique participants from government agencies (CBBH, both BSAs, DIA). Participants from the government were the focus of the survey since capacity-building for the agencies was a principal activity objective. Each agency was then asked to verify the list. Persons who no longer work with the agency or who were on special leave were excluded, and two were added, leaving a total of 97 persons who were sent the email survey. Nine of these had non￾functional email addresses; attempts to correct the emails were not successful. The response rate was 36%, with the CBBH, FBA, BARS response rates around 30% and the DIA response rate 70%. Male and female eligible respondents were approximately equal, but the response rate for females (41%) exceeded that of males (31%). (See Annex 2 for a full description of the survey respondents and results.) 3. PARE Activity self assessment Prior to commencing the field work for this evaluation, the PARE Activity was provided with a questionnaire for completing a self assessment related to the Evaluation questions. In addition, the Activity was asked to identify verifiable results and other results the Activity thought were relevant to their making progress toward achieving the objectives. C. Data Analysis 1. All three evaluation team members participated in Key Informant Interviews and document review, as there is much overlap among the evaluation questions. The team was divided, however, so that one team member was assigned leadership for each question to allow sufficient time for in-depth focus on particular issues where relevant. 2. During the field work period, the Evaluation Team met daily to discuss and consider the results of their review of documents, and it met after each KII to discuss and consider the information gained. These active discussions enabled collation of information, identification of commonalities and contrasting of points of view, and resolution of issues in analysis of data. Themes raised and discussed in key informant interviews are organized by issue and type of interviewee in Table 7, Annex 1, Table 7. This tally was abstracted from notes on the KIIs taken by Evaluation Team members and recorded transcripts. The KII’s were conducted with individuals and - - 11 USAID PARE Evaluation Report with groups of individuals from the same agency. Each sitting (conducting the KII with either a group from the same agency or with one person) was counted as a single KII, since the groups did not express diverse opinions, but rather the different discussion points were addressed by the person(s) within the group most familiar with the topic being discussed. Where persons from the same organization were interviewed separately each interview was counted separate since the responses reflected the views of that sitting without influence from others. Issues identified from the KIIs are those that arose either in response to a question or spontaneously. Not mentioning an issue is not a negative or positive but rather can be neutral, meaning the issue might not have arisen in the interview. This can be interpreted as a view that the issue was not a major issue for the respondent. 3. Survey responses were entered into a SPSS database and cross-tabulations used to collate the responses. 4. Triangulation: Information from (i) observations from reports and documents (from Activity, government agency, and independent sources), (ii) notes from key informant interviews and (iii) trainee surveys, was triangulated to provide more depth and perspective for conclusions and recommendations. Evaluation Limitations The following are the principal limitations of this evaluation, and how they were mitigated: 1. Limited Baseline Information against which to measure progress: The IMF 2006 FSAP report and ROSC recommendations provided general information on the status of BiH with regards to EU standards at the start of the activity, and provided recommendations. However, over the course of the Activity, standards changed (Basel III and EU CRDs 3 and 4 were introduced) and demand￾driven priority activities were influenced by IMF conditions for continuing Stand-By Agreements (SBAs). Mitigation: The evaluation team assessed general progress toward capacity building for implementing EU/international financial standards, how far the Activity moved agencies along a continuum from initial awareness to actual legal implementation of changes toward EU standards, and strategies that were initiated by the Activity to address obstacles that arose. 2. Dependence on Activity reports: PARE Activity reports of activities and results were an important source of information for the evaluation. Mitigation: Validation (or lack thereof) through Key Informant interviews, trainees, and the documents of independent donors and agencies. 3. Willingness of institutions to share data and documents that provided evidence of the effectiveness of Activity activities and the institutionalization of desired changes may limit the ability of the Evaluation to validate reported information. Mitigation: Review of sensitive documents in hard copy form; signing of confidentiality agreements; Activity requests for cooperation. 4. Inconsistency in interview methods. For an evaluation of this type, key informant interviews have the potential to be inconsistent according to information sought and the person conducting the interview. Mitigation: Clearly defined interview protocols; careful review and agreement by the Evaluation Team of interview protocols; conduct of interviews by at least two Evaluation Team members - - 12 USAID PARE Evaluation Report independently taking notes; recording of interviews for later reference. 5. Bias in identification of key informants and their availability for interview. Key Informants were identified by the Activity and by USAID. The Activity was closed prior to the end of the evaluation. Mitigation: The Evaluation Team probed to identify other relevant Key Informants to ensure a well-rounded representative range of interviewees. Consultants and other identified informants who were not in-country were contacted by email and when possible interviewed by phone/skype. Activity staff made themselves available to respond to information needs by the evaluation team despite the fact that the Activity and office were closed. 6. Willingness for respondents to provide honest responses: Mitigation: In order to encourage honest responses, informed consents were a part of all interviews. The informed consent reassured respondents that they will not be identified in any report, and that responses will be aggregated so that the responses of specific interviewees cannot be attributed to specific persons 7. Survey response bias. Response to the online survey request may be inadequate or unrepresentative. . An assessment of differences between respondents and non-respondents showed that the overall response rate was 36%, with a higher rate for females than males (41% vs 31%). A rough summary of names of participants across registration forms showed that the respondents represented participants who attended more events~ than non-respondents, probably representing more active agency personnel. Mitigation: During Key informant Interviews the Evaluation team advised the managers from each agency of the upcoming survey and asked them to encourage their staff to participate. The agency managers verified the list of persons to contact and where email addresses were wrong, the agency managers were asked to provide correct addresses. Telephone follow-up with non￾respondents was not feasible since respondents had been assured that the results would be confidential and there was no personal identification information on some of the returned responses. 8. Simultaneity / spillover: PARE was implemented at the same time that other donors and IFIs, as well as BiH government agencies themselves, were engaging on a variety of fronts in financial sector development in BiH. For some objectives attribution of progress to Activity activities is difficult. Mitigation: Key informant interviews included probing to identify the role played by PARE in specific achievements or activities. 9. Difficulty in generalizing findings / conclusions. For a mainly qualitative evaluation, it is possible that findings may be specific to certain individuals or beneficiaries and not easily generalized. Mitigation: The Activity’s objectives were clearly focused, its key beneficiaries were few, and its activities clearly aimed. Triangulation of information from multiple sources was used to minimize the risk of conclusions being based on non-representative views. Annex 3 provides a matrix with a summary of the Evaluation Methodology as applied to each of the five Evaluation Questions. - - 13 USAID PARE Evaluation Report V. FINDINGS, CONCLUSIONS, & RECOMMENDATIONS Question 1: Banking Supervision To what extent has PARE contributed to the Banking Supervision Agencies (BSAs) abilities to move toward gradual implementation and enforcement of a regulatory framework that is compliant with Basel II/III and relevant EU Directives? Background to Activity Objectives in Banking Supervision The PARE activity’s effort to advance standards, practices, and professional capacities for banking supervision in BiH accounted for over half of the entire Activity budget. Unlike all other countries in the world, BiH has two completely independent authorities for bank supervision, one for each of the two entities – the Federation Banking Agency (FBA), and the Banking Agency of Republika Srpska (BARS). USAID has been by far the most important donor in supporting bank supervision in BiH virtually continuously over the past 17 years, and for almost all of that time has worked in parallel with both agencies, to ensure coordinated progress and harmonized standards. The Central Bank of BiH (CBBH), as a State-level institution responsible in law for upholding financial stability, has also always played a role in facilitating coordination between the two bank supervision agencies (BSAs). The principal goals of the Activity with respect to banking supervision may be summarized as follows: • Advance banking supervision toward Basel standards and relevant EU Directives. • Advance unification / harmonization of bank supervision in BiH • Build the professional capacities of bank supervision officials. The Basel standards focus on (i) the preservation of the funds of depositors, and (ii) the protection of the overall stability of the financial system, for both primarily via establishment of minimum capital standards for banks. The standards (formally called Accords) are established by the Basel Committee on Bank Supervision (BCBS), which is housed at the Bank for International Settlements (BIS) in Basel, Switzerland. The first set of standards, “Basel I” (1988) focused on credit risk – that is, the risk of default or nonpayment on bank loans and investments. “Basel II” (2004) refined concepts of credit risk and added standards for operational risk (risk mainly related to the way the bank internally operates, but also from non￾market-related external events such as natural disasters or cyber attack) and market risk (the risk of loss due to price changes in financial markets – stocks, interest rates, currencies, and commodities). Basel III (2010/11) standards were designed mainly for advanced economy banks and introduce standards for minimum bank liquidity and maximum leverage. The Basel standards are organized around three basic “pillars” for managing risk: • Pillar 1, Minimum Capital Requirements, aligns minimum capital requirements to each bank’s actual risk of loss, based on assessment of credit risk, operational risk, and market risk. • Pillar 2, Supervisory Committee, enables banking supervisors to evaluate the activities and risk profiles of individual banks, and assign them higher capital requirements than Pillar 1 minimums, and take remedial actions. • Pillar 3, Market Discipline, requires public transparency in bank reporting, to provide market pressure for prudent bank management. - - 14 USAID PARE Evaluation Report Basel III did not exist when the PARE activity started. The Activity was charged with assisting BiH in “gradual progress” toward Basel II. When Basel III was introduced it was incorporated into the objectives to some extent. However, according to the BCBS prescription for even advanced economies, the timeline for full implementation of Basel III extends to March 2018. Finally, there are EU directives relevant to bank supervision, which Activity objectives also incorporated. The European Banking Authority (EBA) has over time issued its own series of Basel-consistent capital requirements directives (CRDs), as well as a common reporting framework (COREP) for standardized reporting of credit, operational, and market risk, and the calculation of capital adequacy ratios. Banking Supervision: Findings Activity Focus The PARE activity addressed all three pillars of the Basel standards framework to some extent, but mainly concentrated on Pillar 1. The key informants (KIs) from the BSAs and the Activity explained that in the BiH context, credit risk was a significantly higher priority for supervision and regulation than operational risk and market risk. Operational risk is the second priority. KIs from the BSAs consistently noted that market risk, while of interest for capacity development and eventual readiness, was not considered particularly serious for BiH banks. Activity documents and KIs also consistently noted the BSAs concern about the effects of actually implementing Basel II/III minimum capital requirements in BiH. KIs from the BSAs explained that they feared that the application of these standards might allow BiH banks actually to reduce their capital requirements, with which they were not comfortable.1 KIs from all government agencies, the Activity, and banks all expressed opinions that the BSAs have a quite conservative attitude toward bank risk, attributed in part to the result of widespread losses on the public’s deposits during the late war. Findings – Accomplishments The following are positive developments in the area of banking supervision that took place over the course of the PARE activity and to which Activity activities contributed. • Parallel banking supervision strategies developed: Both entity BSAs developed, and their governing boards formally adopted, strategies for complete implementation of Basel II standards by the end of 2018. Furthermore, these strategies were largely parallel and harmonized. These strategies also incorporated the adoption of EBA CRD and COREP directives, as eventual requirements for BiH accession to the EU. PARE assisted in the development of these strategies, by providing international expert consulting assistance early in the Activity’s term to assess compliance with Basel II standards, to prioritize needs for achievement of the standards, and to prepare specific action plans for progress toward them. PARE also facilitated parallel development of the strategies, by organizing joint collaborative meetings among the two BSAs and the CBBH, at both the working group and governing board levels. 1 This concern is shared by other countries. The European Bank Coordination Initiative (EBCI), formed under World Bank-IMF leadership in 2009 to focus on issues of banking sector stability in the countries of Central and Southeast Europe, has explicitly recognized that the prudential rules of Basel III could have negative consequences for banking in “emerging Europe”. - - 15 USAID PARE Evaluation Report • Professional capacities in bank supervision strengthened: According to KIs from banks, agencies, and the Activity at the beginning of the PARE activity, banking supervision staff were reasonably proficient with Basel I standards, but unprepared for Basel II or the new standards and reporting requirement from the BCBS and EBA resulting from the global financial crisis which arose shortly after Activity launch. The KIs noted that prior to PARE involvement the BSAs, preoccupied with their ongoing supervisory responsibilities (review of monthly bank reports, onsite examinations of banks), lacked a plan or focus toward capacity-building in regulatory standards. Over its course PARE organized 58 banking supervision-related capacity-building events in a variety of subject areas for the two BSAs and CBBH, in many cases with joint attendance. (See Annex 1, Table 2, for a listing.) These included training seminars workshops, study tours to foreign countries (Slovenia, Germany, USA), and planning meetings/roundtables. In the first years of the Activity, training was focused on the assessment of regulatory improvement needs and on the Basel II conceptual framework. In later years, especially during the extension period, training focused on practical implementation of standards and directives – procedures for measuring the various types of risk and calculating minimum capital requirements, including the impact of applying Basel II standards in BiH. For training, PARE helped connect the BSAs with credible regional consultants, by identifying and engaging them and providing their services. These consultants had work experience at the central banks of Slovenia and Croatia and their efforts to move toward Basel II standards and beyond, and spoke the local language. A survey of participants in various capacity-building events (total eligible persons from the BSAs=74, responses=23 – overall response rate 31%) showed high levels of satisfaction with the technical expertise of facilitators/trainers, and the relevance of the capacity building events to their work. Opinions with regards to quality of capacity building events and a self-perceived perception of knowledge pre-PARE and post-PARE, by respondents also were positive. Most respondents reported some level of awareness of topics such as risk assessments and Basel II prior to the PARE activity and an increased understanding and depth of knowledge after exposure to the PARE events and work with the Activity. Although positive, the FBA respondents were a bit less strongly positive than the BARS respondents. Annex 2 provides details on the survey results by agency. • Institutional changes to support Basel standards and EU Directives implemented: The Basel II/III/EU Directives system of regulations is extensive and complex, requiring specialization in its particular areas if adequately deep expertise is to be developed. According to interviews with PARE consultants, at the beginning of the Activity work organization within the BSAs was not conducive to progress in implementing Basel II. Several observed that responsibilities were diffuse and unspecialized within the BSAs, and that the depth of expertise required for key components of the standards called for dividing into teams and specializing, which had not been the case. Furthermore, KIs from the Association of Banks of BiH (UBBiH) noted that the private sector was not adequately included in strategy for implementing new standards. This theme was also identified in PARE initial assessment documents. In institutional development for banking supervision, PARE encouraged and facilitated: o The formation of specialized technical teams within each of the BSAs – one for each of the basic risk areas defined by the Basel standards – credit risk, operational risk, and market risk. BSA staffers are dedicated to each of three teams and have developed specialized expertise. KIs from the BSAs noted this was an important development, strengthening their organizational capacity to address these issues. o Ongoing working group coordination between the BSAs, both between each pair of risk teams and at higher levels, including the governing boards. - - 16 USAID PARE Evaluation Report o Private sector inclusion in the standards reform and implementation process, by stimulating the formation of the Banking Reform Committee within the Association of Banks of BiH (UBBiH), and facilitating the inclusion of UBBiH representatives in PARE activities. o Founding of the Association of Professional Risk Managers (APRM), a PARE initiative, a private sector organization made up of senior risk managers at BiH banks. The APRM will focus on the development of a cadre of risk managers, trained and certified in evaluating the range of risks covered by Basel standards and EU Directives. • “Building blocks” of the Basel II/III/EU Directives framework: Once assessments were completed and the conceptual training done, PARE assisted the BSAs in developing a comprehensive range of actual elements required by the advanced banking supervision framework, some of which were prepared and others actually adopted or implemented. These were: o A set of 13 draft decrees to be promulgated to banks for implementation of Basel II/III and EU Directives. These decrees were planned and designed to address priorities identified in the IMF’s 2006 FSAP report and subsequent assessments by PARE and the BSAs. They are detailed drafts that are ready for review, refinement, and eventual adoption. o Confidentiality legislation, required so that BiH bank supervisors can fully participate in international “colleges” of bank supervisors, which are formed for all European cross-border banks, in order to oversee them on a consolidated basis for all of the countries where they operate. This legislation has been adopted by parliament (gazetted) in the RS and at this writing is in the parliamentary process in the Federation and expected to be passed shortly. o Three decrees to improve bank management boards – covering remuneration of managers, suitability of managers, and diligence of managers. These decrees were formally adopted and are now in force. o The set of COREP forms that banks will be required by EU Directives to use for measuring and reporting risks. o The BiH Quantitative Impact Study (QIS), a testing procedure used in many EU countries to estimate the effect of implementation of advanced standards on banks’ actual capital requirements. The QIS required extensive development of forms and training on procedures, amounting to a kind of “trial run” for Basel II implementation. It was carried out during the extension period, covering all BiH banks. A report on its results was expected soon after the close of the Activity. KIs and Banks all acknowledged the importance of this activity. o A joint BSAs website was prepared with Activity assistance, as a shared knowledge management resource for posting documents related to the banking supervision reform process. Essentially all respondents to the survey on PARE capacity building (97%) said they had used the website. Among those using the website, 65% described it as “very useful” or “an excellent resource”. Annex 2 provides details from the survey. With regard to these “building blocks”, KIs from the government agencies, banks, and the Activity explicitly noted that PARE’s preparation of drafts of documents for BSAs to respond to, and organization of capacity-building events focusing on international standards and specific issues related to risk assessments and risk-based supervision, were essential for increasing the pace and achievements toward adopting international standards. Findings – Issues The evaluation team identified the following issues that potentially compromised further accomplishments of the Activity. - - 17 USAID PARE Evaluation Report • The process of getting to the phase of rapid progress was relatively slow. For example, less than half of the capacity-building events in banking supervision occurred in the first four years of the Activity (the base and option periods), while 55% of them were organized in its last 18 months (the extension period). Similarly, most of the concrete building blocks of the framework, described above, were prepared in the extension period. The reasons for this included: (i) the need to analyze status and prepare assessments at the beginning of the project; (ii) the need to gather key counterparts and gain their trust and buy-in over a series of meetings to develop prioritized strategies and formally adopt action plans; (iii) a heavy focus on “conceptual” trainings; (iii) frictions created by pressure for progress between PARE and the FBA, and eventually a change in the key contact there; and (iii) the Activity’s very broad contractual mandate – too broad, in our view – which initially diverted attention and resources from banking supervision to other objectives not covered in this evaluation. • The Activity faced an unexpectedly turbulent operating environment, which was completely unforeseen during its design at USAID. Almost as soon as it started it ran into the teeth of the global financial crisis, and the international banking supervision standards it was expected to advance underwent substantial redefinition. The crisis also magnified risk-averse attitudes in the BSAs and CBBH. • One of the original objectives of the Activity, advancing the formation of a unified banking supervision agency at the State level, which was also an IMF recommendation, was entirely unachieved. The original PARE contract explicitly acknowledged that the achievement of this objective was subject to political will, and in the event there was outright political opposition to this idea. • The Activity did fall well short of its PMI targets with respect to actual adoption of regulations. Adoption means the gazetting of law or formal promulgation of decrees. In an attempt to meet its targets, PARE pushed very strongly for the adoption of the decrees it helped develop toward the end of the option period (i.e. during the fourth year). This was ahead of the timeline in the FBA’s agreed strategic plan. Three KIs (two Activity consultants and one from the FBA) believe that this pressure created friction between the Activity and the BSAs, potentially slowing progress. • This bears on the question of PARE’s achievement of its two life of Activity indicators for Component 1, which were as follows in the original scope of work: o Percentage of harmonized standards, procedures, and regulations adopted by institutions with PARE assistance in both entities and at the state level. o Number of EU accepted standards, Acquis Communautaire legislation and regulations adopted by targeted institutions as a result of PARE assistance. Neither of these indicators was in fact tracked by PARE: in practice, a different formulation was used, which was in keeping with the “F indicators”, and this was accepted by USAID. This was Activity PMI #1: “The number of internationally-recognized financial sector standards adopted as a result of USG assistance.” The “percentage” life-of-Activity formulation in the original SOW was not realistic, because the standards were a moving target, so the denominator of any percentage measure would have been changing constantly. As for the EU/Acquis standards for bank supervision, these would be subsumed under PMI #1 in its counting of “internationally-recognized financial sector standards”. Again, however, even as revised, PMI #1 did call for “adoption”. As it became clear that PARE itself could not control formal / final adoption, especially given the evolution during the Activity from Basel II to Basel III, this led to extensive back-and-forth between the contractor and USAID, and resulted in a compromise measurement protocol based on degree of progress toward adoption. There were in fact some important formal adoptions: amendments to banking agency law as well as three qualitative decrees in both the FBiH and RS. In addition, based on preparation of 13 broad decrees - - 18 USAID PARE Evaluation Report covering the elements of the Basel and EU standards, PARE was able to claim over-achievement of its target for PMI #1, even though the latter were not formally adopted by the close of the Activity. • The measurement of some other PMIs was questionable. (This finding applies also to Question 2.) For example, the measure of participant satisfaction with training included not just ratings from attendees, but single post-event letters of appreciation from counterpart agency management summarizing an entire event. More seriously, indicators for “number of persons trained” simply added up the attendees to all trainings, which meant that the same persons were multiple times, because essentially the same set of supervisors needed multiple trainings to cover many complex areas. In its final report, PARE reports over 900 financial sector supervisors trained, while according to our careful analysis of the attendance sheets of training events, obtained in hard copy from the Activity, there were only some 120 unique financial sector supervisory trainees from the two BSAs, the DIA, and the CBBH. • Several KIs from the BSAs identified early misunderstandings on how PARE should or would work. Was it to be simply a consulting resource for the BSAs, or a trainer / facilitator, or some combination of the two? It took some time for the BSAs to realize that the Activity was not designed simply to do their work for them, but more to build their capacities. • The PARE team – its permanent professional staff – did not always agree within itself during the first three years as to the design and pace of Activity interventions. A key Activity team leader felt that there should have been greater pressure for progress and that there were unnecessary delays in on￾boarding consultants. This led to frictions within the Activity and between the Activity and counterparts. Banking Supervision: Conclusions • While the two BSAs had received considerable support in the past, mainly from USAID, and while they had the intention to progress toward Basel II standards and were developing strategies for doing so on their own, and while they understood the need for harmonization and had for many years collaborated, we firmly do not believe that they would have made nearly as significant progress in readiness to adopt the standards over the past five years without the support of the PARE activity. In this sense, given the importance of the implementation of these standards for financial sector stability and SME access to finance in BiH, and for eventual accession to the EU, the development objectives of PARE were in banking supervision were to a substantial extent fulfilled. • PARE contributed substantially to the knowledge and readiness of the professional personnel in the two BSAs to implement Basel II/III standards and EU Directives. Extensive and specialized trainings through the many capacity-building events outlined above, including highly practical case study and exercise-based trainings in the second half of the project, have provided BSA staff with the basic understanding and capabilities to carry implementation forward, with or without another PARE-like activity (about which we provide our recommendation in Question 5). • There are two sides as to the issue of whether or not PARE could have stimulated more progress than it did in moving toward and adopting advanced standards. It should be noted that the strategies of the BSAs themselves anticipated preparation of all standards in 2016 and full implementation as of 2018. It is also the case that the extreme instability caused by the global financial crisis certainly made the BSAs more conservative in moving rapidly. However, given these issues, the early focus on assessments and conceptual trainings led by North American consultants, and frictions within the Activity and between the Activity and at least one BSA, concrete progress during the base and option periods was slow. Eventually, as the frictions were cleared, the global crisis passed and strategies settled down, and highly practical regional consultants were engaged, progress greatly accelerated. It is difficult to say whether, in the end, more might have been accomplished. On balance, given all of these influences, we think that if there was any shortfall of actual versus potential progress toward Basel II/III/EU Directives implementation, it was not substantial. - - 19 USAID PARE Evaluation Report • The fact that the Activity’s PMIs as to actual adoption of standards clearly were not met raises the issue not of whether it achieved its intended development impact, but of the appropriateness of the measurement indicators themselves. Clearly, it is not within the power of any donor activity to actually adopt reforms – this can only be done by a country’s government institutions. We recognize that standard “F” indicators point to actual adoption, that this is in fact the ultimate goal, and that having this explicit goal does create constructive pressure on a project to carry out interventions to achieve it. However, as our above findings and conclusions have demonstrated, there are a number of steps that must be taken before any law or regulation can even be considered for adoption, and many of these are under the direct influence of the activity. PMIs need to be better crafted to define and recognize these steps, which would make projects more accountable. Adoption is the final step and should remain an indicator, perhaps with a “bonus” if finally achieved. • The Activity’s reporting of PMIs related to training was flawed with respect to satisfaction. In constructing its ratings of satisfaction, PARE counted a single post-event expression of satisfaction letter from the director of an agency whose staff had attended a training as a full endorsement, giving it a “5” weight for that entire training event in its 1-to-5 satisfaction scale, rather than obtaining ratings from each participant and averaging them over the group. Nevertheless, according to our independent survey of trainees, strong satisfaction with the trainings was in fact indicated. The Activity’s reporting of PMIs related to number of persons trained, however, is quite misleading if interpreted as unique trainees – the normal reading – which were far less than the simple sum of the number of attendees at all training events, due to multiple trainings for most individuals. Question 2: Deposit Insurance (a) To what extent has PARE contributed to improving the overall capacity of the BiH deposit insurance agency? (b) To what extent has the PARE activity contributed to the BiH deposit insurance agency’s compliance with International Association of Deposit Insurers (IADI) standards? Background to Activity Objectives in Banking Supervision The wholesale loss of citizens’ foreign currency denominated bank deposits during the late war left the public extremely distrustful of banks in its wake. Deposit insurance did not exist. With World Bank encouragement, USAID and KfW provided initial capital for a deposit insurance fund, initially with a maximum individual threshold of BAM 5,000. The establishment of this fund by donors, and the recognition of the great value of deposit insurance by all parties, regardless of politics, enabled the establishment of the Deposit Insurance Agency (DIA) at the State level, and ensured its continuing independence. The insurance threshold has been progressively raised and is now BAM 35,000. Meanwhile, bank deposits have grown from some BAM 400 million in the late 1990s, and about BAM 3 billion in 2003, to over BAM 15 billion today. Nevertheless, as reviewed above, there were some significant deficiencies in the deposit insurance at the beginning of the Activity. The following specific results were stipulated in the PARE contract and its extension: • Revision of the DIA’s investment strategy to incorporate best EU practices • Implementation of an updated model for eligible insured deposit thresholds, incorporating EU standards • Incorporating institutional risk into the calculation of individual bank deposit insurance premia, and • Achieving compliance with IADI core principles for deposit insurance. - - 20 USAID PARE Evaluation Report The achievement of these results would require organizational and procedural changes within the DIA, amendments to relevant legislation, and the promulgation of new regulations. Deposit Insurance: Findings Activity Focus PARE initiated discussion and planning meetings at its outset with DIA to determine demand-driven priorities. The discussion resulted in a formal list of priorities received from the DIA to: (1) Improve DIA investment policy (Deposit Insurance Fund Asset Management) (2) Assess and improve the existing Deposit Insurance Fund Model (3) Revise procedure for the deposit insurance claims settlement (4) Develop a risk-based premium system Findings – Accomplishments The following are positive developments in the area of deposit insurance that took place over the course of the PARE activity and to which Activity activities contributed. • Investment policies and practices for the DIA Reserve Fund were greatly strengthened. Formal investment guidelines were prepared and an Investment Committee established to provide much more proactive oversight of outside asset management. The committee added investment experts from outside the Agency. DIA now has management staff dedicated to Reserve Fund investment oversight on a fulltime basis, which was not the case at the beginning of the Activity. • The Deposit Insurance Fund Model was updated and improved. This model had had not been updated since DIA’s launch in 2002. It is critical for orderly analysis in establishing the principal financial parameters of the system: the threshold for eligible insured deposits, the required size of the Deposit Insurance Reserve Fund, and the “flat rate” deposit insurance premium. PARE provided key technical assistance in updating the model, including the addition of modules to measure overall banking system risk and individual bank risk. Reporting formats were also improved, to assist decision-making by DIA’s management board. • Staff analytical capabilities were improved through 17 capacity-building events – trainings, working group meetings, study tours – related to deposit insurance, in analysis and reporting of risk and in implementing the improved model and understanding the information that went into it. (See Annex 1, Table 3, for a listing). Training and workshops also built management and staff knowledge of IADI core principles. A survey of participants in various capacity-building events (total eligible persons from the DIA=10, responses=7; overall response rate 70%) showed high levels of satisfaction with the technical expertise of facilitators/trainers, and the relevance of the capacity building events to their work. Opinions with regards to quality of capacity building events and a self-perceived perception of knowledge pre-PARE and post-PARE, by respondents also were positive. Most respondents reported some level of awareness of topics such as calculating premiums and IADI standards prior to the PARE activity and an increased understanding and depth of knowledge after exposure to the PARE events and work with the Activity. • A risk-based premium analytical framework and model were built. PARE provided the technical support for developing and testing the model, and assisted DIA in testing the model on real data. Risk￾based premia were not, however, actually implemented (see Issues section below). PARE provided a complete information system design specification for the eventual software application required to fully implement risk-based premia. • In coordination with the USAID PFS project, PARE assisted in conducting an analysis of compliance with IADI core principles, of which three were identified as needing to be addressed - #1, - - 21 USAID PARE Evaluation Report covering deposit insurance policy; #16, covering resolution of failed banks; and #17, covering access of depositors to their funds in the event of a failure. The Law on Deposit Insurance was amended to address the overarching policy issue. PARE provided technical assistance in developing standard operating procedures for insured deposit payout in the event of a bank failure were adopted, including related software, and were tested using a provisional bank administrator. Finally, procedures in bank resolution were developed with PARE assistance, while some issues await associated changes in banking law in order to be implemented, one important one – the deadline for duration of provisional administration after bank failure – was substantially advanced with the preparation of amendments to the two entity banking laws that were approved and on the verge of final passage by the end of the Activity. Findings - Issues • Despite the desire and Activity objective, it was not possible actually to implement risk-based premium setting for deposit insurance, for two reasons: (i) Incomplete application across all BiH banks of unified international accounting standards, especially those covering the valuation of financial assets, provisioning for NPLs, and valuation of collateral. This means that that banks cannot be compared on a fully parallel basis as to their risk of insolvency, which could result in one bank having an excessively high risk-based premium while another had one that is too low. (ii) Lack of legal clarity in the process of bank resolution and recovery in the event of bank failure. The adequacy of the Reserve Fund and the deposit insurance premium depend in part on DIA’s legal ability to recover proceeds from the assets of a failed bank. While its preference position in a liquidation is clear – third, after wages due to employees and payments to secured creditors – the definition of a “secured creditor” is not legally certain. There are also ambiguities as to the process of asset/loan sales in a recovery. Amendments to both entity banking laws are required to clarify these issues. PARE did organize seminars with DIA, BSA, CBBH, and other counterpart agencies to educate and inform them of these problems and develop plans for addressing them. The IMF has agreed to provide technical assistance in drafting entirely new entity banking laws. Deposit Insurance: Conclusions • PARE was instrumental in the substantial improvements in DIA investment policies and practices that took place over the period, which are now sound. Severe weaknesses in these practices at the beginning of the Activity were corrected in time to avoid losses that could have been incurred during the global financial crisis. • PARE contributed strongly to the development of professional capacities within the DIA. A majority of DIA staff received applicable of training and reported improvement of job skills. PARE helped upgrade the analytical capacities of the staff, and the quality of decision-making of the DIA management board. • PARE served as a critical outside resource for DIA, given its intensive day-to-day responsibilities, in managing the process of developing the legal-regulatory structure and tools required for establishing and implementing the deposit insurance system’s key financial parameters, and in assessing and addressing compliance with IADI core standards for deposit insurance. This assistance enabled improvements across a range of administrative regulations, standard operating procedures, and internal and external reporting. • PARE facilitated collaboration among all BiH government agencies at the State and entity levels responsible for financial sector supervision, including the DIA. - - 22 USAID PARE Evaluation Report Question 3: Obstacles What were the main obstacles to implementing PARE and how did the activity address them? Obstacles to Implementation: Findings Obstacle 1: Lack of will According to Activity and other international organization documents, there has been hesitancy from within the government to make changes that PARE and the IMF have advocated for the finance sector. This has resulted in slow progress toward achieving EU standards. All except one of the Key Informants (KIs) from government agencies and banks attributed this to at least one of the following: (i) lack of political will, (ii) lack of understanding and related fear of making changes, and/or (iii) feelings of insecurity about making changes related to the ongoing financial crisis in Europe. Only one KI thought that the Activity would have had the influence to make change happen more rapidly if it had pushed harder for change. At least 3 different KIs, including KIs from government agencies, proposed that the political independence of agencies had deteriorated over the past two years, which also influenced decisions about change. How PARE responded: The activity made an effort to focus on technical issues, at the technician level, and on the objective value of meeting international standards, to decrease political or vested-interest pressures against change. In order to improve the environment for change, PARE worked with the Ministry of Finance and Treasury (MoFT) to form a Financial Coordination Unit (FCU) as a logistics unit/secretariat for a Financial Development Council (FDC) that would include all financial regulatory agencies and the private sector. This strategy was aimed at providing a forum where the different bank supervisory agencies at entity and state levels of government would meet, discuss issues and coordinate activities toward increasing institutional financial supervision convergence. “This proposal was recognized and acknowledged by the European Commission in the EU Progress Report for BiH for 2011” (PARE self assessment). This activity was not prioritized under the newly-appointed Minister of Finance and Treasury in 2013 and the focus shifted toward producing a Financial Development Report. Key Informants (KIs) from all sectors indicated that the FDC/FCU is not functioning as a coordinating body. Various capacity-building events (workshops, seminars, study tours) that were sponsored or facilitated by PARE were structured to include participants from the BSAs, DIA, and CBBH. Additional events (roundtables and technical meetings) also served as capacity-building, as PARE consultants facilitated or made presentations related to understanding EU standards and issues related to specific draft decrees/laws. Among the KIs from government agencies and banks, all but one mentioned the particular value of study tours to neighboring countries and technical assistance from PARE consultants with experience in implementing regional and EU standards, in giving officials confidence to move forward with EU standards in BiH. KIs from the agencies also noted that these events helped them to better understand what the proposed changes meant and the rationale for what was included and not. As KI noted “we can’t just pass a decree without understanding what it means”. Annex A provides a list of capacity building events related to EU standards, banking supervision, and deposit insurance and Annex B the distribution of participants, by agency and gender. All but one Key Informant from government agencies and two from banking/international organizations were also very emphatic that the “external/independent” status of PARE was key to allowing the agencies to move forward, and that this provided agencies with protection against pressures from vested interests. - - 23 USAID PARE Evaluation Report Obstacle 2: Slowness in forward movement toward EU standards. In addition to the issues raised in Obstacle 1, KIs from government agencies, banks, and International Organizations all mentioned issues related to the workload of the government agency staff, and the lack of dedicated persons to work on the EU standards, with comments suggesting that working toward EU standards was work added on to already full time jobs. How PARE responded: PARE developed draft documents that were useful for beginning discussions about decrees or laws with the government agencies. PARE was specifically asked to help with drafting some documents (e.g., Confidentiality Law to conform with EU requirements for cross-border sharing of information, and 3 decrees related to management boards: banking remuneration, board member qualification, and diligence of the board). PARE initiated others (e.g., 13 risk-based supervision decrees). All but two of the key informants specifically acknowledged the role of PARE in producing initial drafts as important for moving an agenda forward. PARE also advocated that agencies assign individuals to one of three key topics (market risk, credit risk, operational risk), with the explanation that one person cannot learn about all of the new issues. With PARE encouragement three different technical teams were formed by each entity. The teams became part of joint agency technical working groups (TWGs) that included both BSAs and the CBBH. Obstacle 3: PARE had no authority to adopt decrees or to propose legislation to the Parliament. How PARE responded: PARE initiated activities with the Association of Banks (UBBiH), to involve the private banks that will be affected by changes resulting from EU standards more closely in initial discussions regarding decrees and legislation. The practice already existed for final draft decrees and laws to be sent to the UBBiH for comment; however, the PARE effort focused on including the UBBiH earlier in the drafting process. A Letter of Intent (LOI) with the UBBiH formalized the agreement of the UBBiH to participate in PARE-sponsored capacity-building events, and to include PARE in UBBiH events. It reaffirmed that the UBBiH would form a Committee for Banking Reforms (comprised of representatives from 6 banks). PARE was instrumental in the formation of the BiH Association of Risk Managers, to increase local professional capacities for international risk assessment and risk management standards, and to advocate for reforms. KIs from the banking industry were positive about the value of this new association, whose opening session was in September 2013, toward advancing risk-based supervision. Both the UBBiH-sponsored Committee for Banking Reforms and the Association of Risk Managers are meant to formally provide feedback to the BSAs about banking supervision reform strategies. Although not asked specifically about consumer education, KIs from one of the government agencies and from banks brought up their increased awareness of the obligations of banks to consumers, attributing this to their interactions with PARE. PARE entered into a Letter of Intent with the CBBH to develop the content of a consumer education section on its website. This website focuses on explaining terminology and issues related to different types of loans and interest rates relevant to the individual consumer. CBBH KIs were not familiar with the content (referring the team to an Information Technology staff member), and most KIs regarded it as having some utility but falling short of its potential. Obstacle 4: Weak coordination among government agencies. All Key Informants noted that there has always been good coordination between the Federation and RS BSAs. Coordination across the CBBH, BSAs, and DIA, however, has not consistently been implemented. Key Informants from the agencies indicated that the existing coordination forum chaired by the CBBH does not include the private sector, except by special invitation, and focuses more on current issues rather than forward-looking financial policy changes to meet EU/International standards. There was no will to address a state-level banking - - 24 USAID PARE Evaluation Report supervision agency. Several KIs noted that the CBBH had been unwilling to take on leadership and coordination in areas where it should have legal authority to do so – a prominent example being the establishment of a reference interest rate, How PARE responded: In addition to the attempt to develop the FCU within the FDC at the MoFT, PARE capacity￾building events were almost all structured to include representatives from the BSAs, DIA, and CBBH, and often the private banking sector, which ensured that all were receiving the same information and were exposed to the same discussions about issues related to regulation and supervision. Many of these events focused on harmonizing legislation/regulation between the RS and FBiH. PARE facilitated the drafting and acceptance of a strategy and workplan for implementing EU standards (2008). The Activity also was instrumental in updating the strategy in 2012/13. These strategies were harmonized between the two entity BSAs. Key Informants from government agencies acknowledged the importance of this unified strategy. PARE developed the banking supervision reform website www.bankarskasupervizija.ba as a “platform and transparent communication tool for all reform participants (BSAs, CBBH, Bank Association of BiH) to share information, lessons learned, and experiences” (PARE self￾assessment). Obstacles to Implementation: Conclusions • PARE adapted its strategies to address these obstacles and was effective. o Capacity-building events were critical to helping government agencies to understand, and feel more comfortable with changes required by international standards. o International study tours, particularly those to neighboring countries, were critical to strengthening the confidence of agency personnel in working toward EU standards. o Participation of the government agencies (BSAs, DIA, CBBH) and often the private banks in the same trainings/capacity building events was a positive step in strengthening technical support for moving forward with EU standards and toward increasing the rationale and content of various EU standards. o Development of subject-specific teams within the BSAs, that then joined technical working groups across the entities, strengthened harmonization of draft decrees/ laws between the BSAs. o Strategies to increase the active participation of the UBBiH in providing early input into decrees and legislation, and the Association of Risk Managers in supporting implementation, has contributed to generalized support (and potentially pressure) to support adopting changes toward EU standards. The PARE role in improving awareness of obligations of banks to consumers may also be contributing to support for adopting international standards. o The efforts with regards to the FDC and FCU followed reasonable steps and were not duplicative of existing financial sector coordination forums. This remains a potential means to provide strength to technical leadership in the financial sector and to decrease political/vested interest influence. • The PARE “external and independent” presence provided a degree of political screen to protect government agencies and allowed issues to be brought forward that otherwise might be stopped due to fear of political or legal retribution from those against the changes. This presence also - - 25 USAID PARE Evaluation Report created pressure on the BSAs to move more rapidly in adopting reforms than they might have chosen to do without it. • There is potentially more scope for using a “social marketing” strategy to develop public awareness and subsequent support for moving toward EU standards. This would require identifying the benefits to target sub-groups (small business? individuals?) that application of EU standards potentially bring (increased investment resulting in increased jobs? security of private savings?). • The CBBH consumer education section is informative but not particularly consumer friendly and lacking in breadth, based on KI feedback and a review by evaluators. For this site to achieve objectives toward increasing demand and support for international standards as well as improving consumer literacy in using bank services, significantly greater focus on maintaining and updating it, along with a consumer awareness strategy, is needed. Question 4: Donor coordination How well was PARE’s implementation coordinated/synchronized with other donors’ activities aimed at helping BiH more effectively supervise and maintain the stability of the financial system in BiH? Donor Coordination: Findings There are few organizations working in the financial sector in BiH, and no other on-the-ground projects to support the banking supervision and deposit insurance agencies in meeting Basel and EU financial standards. The focus of other international donors has been on financial stability, and in other financial areas such as insurance and financial accountability. PARE technical assistance was aligned with the recommendations and policies set forth in IMF and World Bank (WB) documents including the IMF/WB Financial Sector Assessment Program (FSAP) 2006 report and periodic IMF reviews that give recommendations tied to continuing Stand-By Agreements for funding. PARE consultant reports consistently reference IMF and WB documents used for background or to support the consultant recommendations. There was little overlap in on-the-ground work related to banking supervision and deposit insurance. Where topics overlapped, IMF and WB KIs reported that consultants did meet with PARE, and there was general awareness of each other’s work. IMF consultants have provided technical assistance to the government focusing primarily addressing financial stability and other current issues rather than addressing shifting the financial system toward EU financial standards or strengthening supervision. The SBA Letters of Intent / conditionalities in fact have not referenced the specific goals of PARE for movement toward Basel II/III or creating a unified State-level BSA. In at least one of its SBA review mission reports, however, the IMF specifically recognized the work of USAID PARE in advancing banking supervision and deposit insurance standards. Partners for Financial Stability (PFS), a USAID regional project, did provide what was identified by PARE and PFS as complementary work with the DIA, helping it carry out a self-assessment and making recommendations regarding its status in terms of the International Association of Deposit Insurers (IADI) “core principles for effective deposit insurance systems” adopted by the EU in 2009. PARE activities in support of the DIA were aligned with these recommendations. PARE and PFS cooperated with DIA in the organization of the International Conference of Deposit Insurers that was held in Sarajevo in 2011. Most collaboration with other partners took the form of participation in seminars and meetings, and jointly funding/organizing capacity building events. Some specific examples include the KfW development bank participating in a conference organized by PARE on banking supervision (2010) and - - 26 USAID PARE Evaluation Report PARE and KfW co-organizing a study tour to Germany related to EU regulations and banking supervision. Annex 1, Table 6 provides a list of related various information-sharing conferences and meetings in which PARE participated. PARE also developed relations with regional consultants who were hired to provide technical assistance. KIs from the government and banks noted that the regional and hands-on experience implementing international standards provided credible support to BiH in moving toward EU standards. PARE also linked with the Center of Excellence in Finance (CEF) in Slovenia, to provide hands-on case-study experience with the Bank of Slovenia and Slovenian commercial banks during a study tour. Donor Coordination: Conclusions • PARE had contacts and information-sharing with all identified donors or other external groups relevant to supporting BiH movement toward EU standards, particularly related to banking supervision and deposit insurance issues. • For the most part there was no overlap or duplication of the activities of various parties working on EU financial standards, banking supervision, and deposit insurance issues. • PARE work was totally harmonized with the recommendations of the IMF and World Bank. • PARE participation in various workshops and conferences provided forums for sharing information on the collaboration between government agencies and PARE in moving BiH toward Basel and EU financial supervisory and regulatory standards. Question 5: Recommendations Based upon the expert opinions of PARE counterparts and key stakeholders, what priority reforms in the BiH financial services sector (specifically banking) need to be addressed? This question is focused on banking, but based on the responses of KIs and our findings and conclusions for the previous questions, we also address other related areas. 1. Probably the most immediate question is whether a follow-on PARE-like activity is needed. We have concluded above that PARE provided critical support for the significant advances in the banking supervision and deposit insurance systems of BiH that occurred over its course, and that much less progress would have been made without it. While counterparts regularly cite the value of USAID’s “presence” behind the activity – shielding them from political pressure and similar generic issues – we believe that the main reason that progress was achieved is quite practical: the fact that the BSAs and DIA, preoccupied with their demanding day-to-day supervisory responsibilities, had through PARE an outside resource focused exclusively on systemic advancement, which was able to serve them in moving this program forward and providing essential expertise for it. There is no doubt that these needs will continue, because there is still a lot to be done, as is evident from our findings above. However, we believe that because of the substantial advances in professional capabilities in the counterpart agencies developed over the PARE activity, and the many building blocks of the advanced systems it helped put in place, another U.S. contractor-procured support activity is not necessary. In fact, PARE has helped prepared the counterparts for direct assistance, in keeping with USAID FORWARD. We recommend that USAID consider a direct contract or grant with the CBBH to manage technical assistance to the BSAs and DIA to help carry through implementation the advanced regulatory system in a timely manner. Essentially, the CBBH would engage a small, special-purpse project management team that would organize activities and required consulting support. USAID’s contribution to the cost - - 27 USAID PARE Evaluation Report (perhaps $1-1.5MM/year) would be complemented by the counterparts – the BSAs, DIA, and CBBH – who do have the financial resources to share in it. USAID would condition the contract/grant on approval of the team leader and workplan versus clear milestones. 2. When asked about their views as to gaps and needs in banking and financial sector development, many KIs quickly pointed to the very small size and influence of the nonbank financial sector in BiH, where banks hold some 85% of all financial assets. In turn, over 80% of those assets are held by the major foreign-owned banks, which ever since the financial crisis have essentially capped the funding of their BiH subsidiaries. These banks also own the main leasing companies, which represent most of the present nonbank financial sector. This leaves businesses in BiH, especially SMEs, with entirely too narrow a range of financing options. The BiH private sector credit / GDP ratio in the 60-70% range, compared to 125-250% in advanced economies, is the best indicator of the absence of financing options outside the commercial banking sector. Even though access to finance certainly is also constrained on the “demand side” by lack of collateral, small size, lack of owner equity, lack of capacity in SMEs to professionally present themselves to financial institutions, and their competitiveness problems. However, these are problems confronted by SMEs in developed economies as well, and many nonbank “supply side” options are designed to get around them – such as equipment leasing, purchase order finance, inventory and receivables finance (including factoring), as well as development credit. The emergence of a vibrant nonbank sector actually directly relates to the banking sector, because it enables banks both their sources of funding as well as diversify their asset portfolios. We were impressed that even the Association of Banks emphasized this need, even though a nonbank sector would compete with banks. USAID should consider an activity focused on SME access to finance through nonbank financial sector development. This project would be “activist”, possibly employing seed funding, innovative DCA guarantees, and public-private partnerships, to stimulate the emergence of equipment leasing, factoring, nonbank finance companies, targeted investment pools, and bond markets – corporate, mortgage, SME finance fund securitizations – aimed primarily at financing SMEs. The activity would also encourage the emergence of financial sector supporting services that are critical to both bank and nonbank finance, such as individual and business credit rating services, collections services, and collateral liquidation services. Finally, given the past investments that USAID and other donors have made in establishing a strong legal and institutional system for securities markets in BiH, the activity would carry forward initial work done by PARE in insurance and pensions, which are vital to expanding the supply of domestic savings to feed capital markets. 3. To build broader political support for EU-related reforms, future activities toward meeting EU standards might focus more on social marketing the advantages of a financial sector that adheres to international standards, and any local investment and job creation aspects that might be linked to this. 4. The fact that PARE did not achieve its specific objectives with respect to the actual adoption of laws and regulations, even though it greatly advanced their preparedness, calls into question the adequacy of this indicator itself. While adoption/implementation can and should be final objective of a business environment reform project, PMIs need to be better crafted to define and recognize the significant steps required prior to actual adoption. Eventually the Activity and USAID agreed to develop and measure achievement according to a set of milestones/benchmarks, which was a constructive decision. Future activities should be designed and PMIs and workplans developed in this manner, with well-defined and agreed benchmarks. This would in fact make activities more accountable, and would avoid possibly unproductive pressure from a project for actual adoption before proper review and discussion by all relevant local parties, including industry and the public. In addition, PMIs for number of - - 28 USAID PARE Evaluation Report trainees should be more clearly defined to distinguish participations (sum of attendees) from unique individual participants, who may attend multiple trainings. - - 29 USAID PARE Evaluation Report U.S. Agency for International Development Bosnia and Herzegovina