July 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez England & Associates. MID-TERM PERFORMANCE EVALUATION OF USAID/SERBIA BUSINESS ENABLING PROJECT (BEP) MID-TERM PERFORMANCE EVALUATION OF THE USAID SERBIA BUSINESS ENABLING PROJECT Final Report Prepared under Contract Number AID-169-C-00-11-00001 Submitted to: USAID/Serbia Submitted by: Andrew Schafernaker, Team Leader Dr. Ana S. Trbovich, Business Environment Specialist Contractor: Mendez England & Associates 4300 Montgomery Avenue, Suite 103 Bethesda, MD 20814 Tel: 301- 652 -4334 www.mendezengland.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS EXECUTIVE SUMMARY...................................................................................................... ………i 1.0 EVALUATION PURPOSE AND PRIORITY QUESTIONS…………………………………...1 1.1 Evaluation Purpose……………………………………………………………………………....1 1.2 Evaluation Questions……………………………………………………………………………1 2.0 PROJECT BACKGROUND…………………………………………………………………………1 3.0 EVALUATION METHODS & LIMITATIONS…………………………………………………..2 4.0 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS…………………………….......5 4.1 Question 1: How successful was the project in reducing the regulatory burdens to businesses in Serbia?...............................................................................................................................................................5 4.2 Question 2: How successful was the project in increasing the GoS capacity and transparency in public financial management? ............................................................................................ 13 4.3 Question 3: How successful was the project in creating conditions for increased finance for the private sector? ..................................................................................................................................... 17 4.4 Question 4 & 5: How successfully did the project gather the support of relevant groups for reform? To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? ...................................................................... 21 ANNEXES Annex I: Evaluation Statement of Work Annex II: Work Plan and Evaluation Research Design Annex III: Sources of Information  List of Persons Interviews  Bibliography of Documents Reviewed  Summary of Online Survey Findings Annex IV: Performance Analysis Table i ACRONYMS Acquis Acquis communautaire (body of EU legislation and standards) AmCham American Chamber of Commerce BA Business Associations BEP Business Enabling Project BIRN Balkan Investigative Reporting Network CeSID Center for Free Elections and Democracy (Serbian NGO) CSO Civil Society Organization EU European Union IFC International Financial Corporation FY Fiscal Year GCI Global Competitiveness Indicators GDP Gross Domestic Project GoS Government of Serbia GOV Government GS General Secretariat ILO International Labor Organization IMF International Monetary Fund IFI International Financial Institution IOSCO International Organization of Securities Commissions I.R. Intermediate Result (USAID Indicator) FIC Foreign Investors Council LOE Level of Effort LT Long-term MoFE Ministry of Finance and Economy MoIFTT Ministry of Internal and Foreign Trade and Telecommunications MoJPA Ministry of Justice and Public Administration MSO Media Support Organizations MT Medium-term OECD Organization for Economic Cooperation and Development NALED National Alliance for Local Economic Development NBFI Non Banking Financial Institutions NBS National Bank of Serbia NGO Non-Governmental Organization PDA Public Debt Agency PEO Public Enforcement Officer PMP Performance Monitoring Plan PPP Public-Private Partnership RIA Regulatory Impact Assessment RRO Regulatory Reform Office SC Securities Commission SCM Standard Cost Model SME Small and Medium Enterprise(s) SOW Statement of Work SUK Government of Serbia Human Resource Management Office USAID U.S. Agency for International Development USAID SLD USAID Serbia Sustainable Local Development Project USG U.S. Government ii USD U.S. Dollars VfMM Value for Money Methodology WB World Bank WBDB World Bank Doing Business Report WG Working Group WEF World Economic Forum 1 EXECUTIVE SUMMARY EVALUATION PURPOSE This is a report on the mid-term performance evaluation of the Business Enabling Project (BEP) funded by the United States Agency for International Development (USAID) Mission in Serbia, and implemented by Cardno Emerging Markets. The evaluation covered the period January 2011 - June 2013, and was conducted by a team assembled by Mendez, England & Associates located in Bethesda, MD. The team consisted of one international and one local expert – both with experience in evaluating and/or working on economic growth projects in the region. The purpose of the evaluation was to provide USAID with: 1) a measurement of change in development outcomes attributable to the defined interventions of BEP, based on models of a broad range of qualitative, quantitative, or mixed methods; and 2) a rigorous, evidence-based analysis of BEP’s mid-term performance. Specifically, the evaluation was intended to: 1. Examine the effectiveness of BEP’s activities implemented to date 2. Determine whether BEP has achieved planned results 3. Identify gaps in BEP’s performance against targets 4. Provide recommendations on closing these gaps in the final years of the project EVALUATION QUESTIONS The evaluation questions, per the Scope of Work (SOW), were: 1. How successful was the project in reducing the regulatory burdens to businesses in Serbia? 2. How successful was the project in increasing the Government of Serbia’s (GoS) capacity and transparency in public financial management? 3. How successful was the project in creating conditions for increased finance for the private sector? 4. How successfully did the project gather the support of relevant groups for reform? 5. To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? PROJECT BACKGROUND BEP is a 5-year, $15.48 million initiative launched in January 2011. As stated in the SOW, the purpose of BEP “is to help the GoS improve the competitiveness of the Serbian economy and its private sector businesses.” The project supports Assistance Objective 1: More Competitive Market Economy, IR 1.1 Business Enabling Environment Improved, and IR 1.4 Macroeconomic Environment Improved. BEP aims to improve the ability of all businesses in Serbia to compete in global, European Union (EU), and domestic markets. BEP consists of three major components:  Component 1: Business Regulation and Economic Governance. Streamlines laws, regulations, and institutions, improves regulators’ capacity to implement laws, and improves analysis and public￾private dialogue.  Component 2: Macroeconomic Policy and Public Financial Management. Helps GoS implement new budget and fiscal responsibility laws, improve budgeting, increase public disclosure and transparency, strengthen public debt management, and improve economic analysis and forecasting.  Component 3: Financial Market Development. Improves regulatory frameworks for non-bank institutions, develops capacity to strengthen and expand capital markets, and improves conditions for debt market development. 2 The project’s approach to achieving reforms is to help GoS work closely with the private sector and outside experts to make reforms that improve business competitiveness. The reforms reduce the time and money spent by Serbian businesses to comply with laws and regulations and help GoS to be more efficient. The reforms also enable businesses to attract necessary investment at an appropriate cost. As a result, Serbian businesses have a better chance to be competitive in the global economy, leading to export-led economic growth and increased employment. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS To conduct BEP’s evaluation, the Evaluation Team used a combination of different qualitative and quantitative methods. Qualitative Research and Analysis The Team interviewed 50 people, including representatives from the GoS, international/donor organizations, business associations, academic and research institutions, USAID Serbia, and the BEP team (see Annex III). In addition to direct one-on-one interviews, the Team also conducted focus group interviews – one with experts from the National Alliance for Local Economic Development (NALED), and one with company members of the Exporters Association of Serbia. Given the short time allocated for the fieldwork and the number of stakeholders that needed to be interviewed, the Team developed a standardized questionnaire for interviews (See Annex II). The same interview framework was used for the anonymous, online survey, which was administered via Survey Monkey, with the goal of cross-referencing information obtained in direct interviews, particularly with regard to respondents’ prioritization of the problematic areas in Serbia’s business climate. The online survey was administered in Serbian to ensure a higher response rate by company representatives and it reached a total of 92 individuals, mostly company representatives (See Annex III for a summary translated into English). Quantitative Research and Analysis Quantitative data was sourced from BEP’s annual work plans and other project-related periodic reports, and cross-referenced with international reports on Serbia’s business-enabling environment published by the EU, World Bank, WEF, and OECD, among other donors. Particular attention was paid to the project’s annual Performance Monitoring Plans (PMPs), as shown in Annex III of this report. Furthermore, a quantitative approach was employed, whenever possible, in the survey analysis to score responses and present findings in a clear and comprehensive fashion. The Evaluation Team analyzed the collected qualitative and quantitative information with the aim of answering the five main evaluation questions outlined in the SOW, as well as to provide information on cross-cutting issues that needed to be addressed by the evaluation, such as gender. When evaluating achieved results, the Evaluation Team considered a number of factors in seeking a balanced judgment, namely:  Relevance to USAID and GoS’ goals, which was based on data collected from primary and secondary sources on key problems affecting the Serbian business environment, and feedback from stakeholders on the importance of the issues being addressed.  Effectiveness of BEP’s strategy in achieving the intended results, which was evaluated by reviewing the outputs of specific activities and their contribution to achieving BEP’s objectives, as well as BEP’s reports, other secondary data, and stakeholder feedback.  Efficiency in the use of BEP’s own resources (e.g. experts, grants) and in leveraging partners’ support and resources (expertise, reform groups), mainly assessed on the basis of feedback from stakeholders and known limitations in terms of time and financial resources. The Evaluation Team also looked at sustainability and impact, although the Team considers it premature to expect measures of impact and indications of long-term sustainability, given the complexity of problems and time frame required to generate momentum for achieving deep change. In this respect, sustainability is whether the processes and mechanisms put in place are sufficiently comprehensive to enable an improved 3 public service (e.g. inspections), and ensure sustainable improvement in the mid and long-term, whereas impact is measured by the activity’s effect on improvement of the business environment and the economy (e.g. impact of construction permitting reform on new investment, employment generation, national income, etc.). The Team has use used a balanced judgment on a simple scale of there being: a) no or limited success; b) moderate success (whereby some important results have been achieved while others are expected from work-in￾progress); and c) significant success in achieving or exceeding the majority of objectives. EVALUATION LIMITATIONS  Low Response. The evaluation was conducted during the months of June and July, which carries a risk of low response since several stakeholders were away on vacation for one or two weeks. To mitigate this risk, the Evaluation Team scheduled the interviews at the beginning of the evaluation, interviewed in parallel, and used the Survey Monkey online tool to reach companies and gauge their opinion on the business climate, BEP and its impact.  Ability to Reach all Stakeholders. The Evaluation Team aimed to reach stakeholders in government institutions who tend to be in demanding positions and who are extremely occupied. Good time management in interview scheduling and previous professional acquaintance of many stakeholders ensured that all stakeholders were reached as planned.  Availability of Data. BEP has initiated a number of different activities, such as inspections reform or public administration management trainings, for which it is too early to have data to analyze their impact.  Pessimism. The Evaluation Team was concerned that the prevailing pessimism due to a difficult economic transition in Serbia - further aggravated by the world economic crisis and recent political sensitivities in the EU accession process - would negatively affect the responses. Aware of this possible bias, the Team asked the interviewees to try to assess the situation as objectively as possible and cross-referenced comments with different opinions and factual data. FINDINGS AND CONCLUSIONS The Evaluation Team’s findings confirm that BEP has tackled the most burning issues in Serbia’s business climate, although some key impediments such as corruption or public administration inefficiency require time and resources that are beyond the project’s scope. There is wide agreement among stakeholders, confirmed by available data, that this is an excellently managed project, even exemplary in providing not just highly appreciated know-how and expertise in improving operational processes, but also sincere dedication to achieve effective and sustainable results. In particular, stakeholders value the demonstrated flexibility of the BEP team in responding to the needs of beneficiaries and partner organizations. Several interviewed stakeholders further remarked that they were especially impressed by the support provided by the entire USAID team and the US Embassy in Belgrade, including the Ambassador’s role in encouraging the reform process. However, while acknowledging this, stakeholders and surveyed businesses found that, with some exceptions, relatively little improvement has been produced in the past two years of the project’s operations in the key policy areas. This implies that select activities with higher prospects of success would need to be completed with greater support during the implementation phase, and this would mean dropping or carefully limiting those activities that are less promising in order to increase the project’s effectiveness. BEP’s performance against key indicators is summarized in Annex IV to this report. The project currently has 46 targets covering the 3 components (8 IR, and 38 BEP indicators that are specific to Components 1-3, though the latter figure includes two indicators that are repeated in the PMP for Components 2 and 3, respectively). From the total number of indicators, 23 are attained, 12 are below/off-target, and 9 are yet to be measured. The project is on target, with the following exceptions, where it is not achieving or under achieving (see Result v. Target, for end 2012 or current if available, given below):  IR.1.4.1: Improved Open Budget Index (54 v. 39); IR.1.4.2: Improved Government Balance as percentage of Gross Domestic Product (GDP) (-6.6% v. 4.5%); Improved Score in WEF Global Competitiveness Indicators (GCI) Financial Market Development (3.7 in 2012 v. 3.8 target). 4  Component 1: Number of Public Enforcement Officers (PEOs) trained, examined and appointed (121 v. 300 planned in 2012); Days to enforce judgment (635 v. 530 targeted in 2012); Realization of new labor legislation (not achieved). Business perception of transparency and predictability of regulations (23.5% v. 25% of respondents of the 2012 BEP Business Survey perceiving transparency).  Component 2: Operational status of debt management system (not functioning as planned); Number of studies supporting changes in macroeconomic policy (3 v. 5 planned in 2012).  Component 3: Status of legal and market framework for Commodities Exchange (not on track for establishment); Improved access to finance (23.2% v. 30% of respondents see improvement in BEP Business Survey 2012); Increase in average value of small, medium enterprise (SME) loan (59,927 EUR in 2012 v. 89,475 EUR in 2011, therefore a reduction). In summary, the most important indicators on target include:  IR 1.1 Business environment improved (WEF GCI); IR 1.4.2 Financial markets developed (credit to private sector); Number of hours of training for journalists.  Component 1: Numbers of commercial laws and regulations simplified; Numbers of Regulatory Impact Assessments (RIAs) carried out; Average time spent by management working with inspections (34 hours in 2012 v. 59 hours in 2011); Improved perception of reducing burden of regulations; Decrease in businesses reporting negative impact of Labor Law; Increase in business participation in public￾private dialogue (from BEP Business Survey reports).  Component 2: Improvement of quality of medium-term budgeting and prioritization; Increase in value of dinar denominated debt; Reduction in public debt held by banks; Awareness of GoS’ economic growth strategy (from BEP Business Survey); Number of hours training GoS staff on public financial management topics.  Component 3: Value of dinar debt (as above); Reduction in public debt held by banks (as above). The most important indicators, yet to be measured, related to past activities include:  IR 1.1.1 Decrease in costs for business to comply with the regulatory framework (Standard Cost Model); Improvements in the Open Budget Index (OBI) survey.  Component 1: Improvement in the “doing business strength of investor protection index” (WBDB); Completion of regulatory framework for PPP (due 2013).  Component 3: Value of credit financed from non-banking financial institutions (NBFIs) to business; Compliance with International Organization of Securities Commissions (IOSCO) standards in areas of security intermediaries and equity issues. The following paragraphs provide a summary of the findings and conclusions as they relate to the key evaluation questions, whereas the subsequent section provides recommendations. How successful was the project in reducing the regulatory burdens to businesses in Serbia? The Evaluation Team found that BEP’s activities have been successful in thoroughly investigating regulatory issues, putting them on the political agenda and, subsequently, delivering needed support in collaboration with reform￾oriented stakeholders in the public and private sectors. However, the rate of progress in achieving the final outcomes is slower than desired by the business community. Concrete success has been achieved in the reduction of para-fiscal charges, although interviewed stakeholders considered the impact on business to be relatively limited. Final outcomes are yet to be achieved in other areas, such as significant improvement in construction permitting, introduction of new Labor Law, or better enforcement of judgments. However, most stakeholders agreed that the rate of progress is slow for reasons outside BEP’s control (e.g. public body inefficiencies and lack of political decision). Moreover, the GoS-declared intentions and actions, such as the reduction of para-fiscal charges, have contributed to a perception of reducing burden to businesses, which was reflected in the findings of the BEP’s Business Survey 2012. Thus, direct public and private sector beneficiaries praised the project on the effectiveness and efficiency of its methods, in part due to initial success with reduction of para-fiscal charges, and in part due to expectation of further successes in the numerous activities already in progress. In particular, BEP was praised for its achievement in bringing together key stakeholders and involving them in the reform process. 5 To summarize, the regulatory reform activities are highly relevant and have been implemented very efficiently with very good use of internal and external resources. However, effectiveness, in terms of achieving specific results other than reduction of para-fiscal charges, has yet to be achieved. Nevertheless, the outlook is promising (e.g. the new Labor Law is planned for the end of 2013, as confirmed by the Deputy Prime Minister of Serbia in July 2013). Thus the Evaluation Team concluded that the project has achieved moderate success in the area of regulatory reform. How successful was the project in increasing the GoS’s capacity and transparency in public financial management? The Evaluation Team found that BEP is contributing to increasing GoS’ capacity in public finance management through its work in introducing methodology, coupled with provision of short-term expertise and training. Non-specialists interviewed by the Team were skeptical about the increase in capacity and greater transparency; this demonstrates the difficulty of working in this area, and further elevates the importance of progress made by BEP (even though small) since improved management in this area means both improved governance and significant savings that could be transformed into additional economic development. Specifically, the Business Survey 2012, which although supported by BEP was conducted professionally and thus considered an objective source, showed that businesses are more aware of the GoS’ economic plans. More importantly, informed stakeholders have remarked upon good progress and the Fiscal Council provides good quality oversight. Therefore, the Evaluation Team concluded that BEP’s success in increasing GoS’ capacity and transparency in public financial management has been moderate, with good prospects of higher achievement in the remaining period of project duration. How successful was the project in creating conditions for increased finance for the private sector? Compared to other areas of activity, the Evaluation Team found that BEP has been less effective in capital markets development. Since the onset of the project, the capital trade volume in Serbia has been further reduced as a consequence of low economic activity in the wake of the global financial crisis and the fact that most foreign-owned banks and large public enterprises are not publicly listed, and hence, not traded. However, there is one achievement, namely BEP’s support in the development of a new Factoring Law, which has rendered this financial service better regulated and thus less risky. The project has refocused this component to concentrate predominantly on SME access to finance; this is commendable since BEP will dedicate its resources to tackle aspects of a major problem for Serbian businesses. Based on the above, the Evaluation Team concluded that BEP has achieved limited success in this area; however, its refocus on SME finance should lead to higher effectiveness in the second half of the project’s term. How successfully did the project gather the support of relevant groups for reform? To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? The Evaluation Team found that BEP has been very effective in achieving the intermediate results of its strategy to tackle key policy issues. BEP’s strategy is based on building reform groups focused on specific policy issues and catalyzing change through forged partnership. BEP has been successful in identifying stakeholders and focusing their attention on key issues in reform groups, and there are good examples where it has molded them into effective partnerships for reform. The project has also built support through outreach programs to inform stakeholders and the public. At the operational level, BEP has worked very effectively with a number of grantees to gain insight into the nature of problems. It has also provided support to its partners in public bodies through training programs and the Internship Program, both designed to bolster their capacity. Finally, BEP has paid some attention to building relationships with women’s organizations and ensuring that they are represented in its initiatives. The Evaluation Team concluded that BEP has achieved significant success in creating reform groups, but only moderate success in catalyzing reform, given that many issues remain work-in-progress. KEY RECOMMENDATIONS 1. Develop better activity-linked indicators: BEP should develop indicators with strong linkages to outputs and outcomes of activities, updated targets, and with reference to achievement of EU accession requirements (outlined in annual EU progress reports). 2. Further improve communications and develop strategies for unblocking hold-ups. BEP needs to clearly communicate intentions and plans on activities so stakeholders not only understand the reasons 6 for hold-ups but also jointly plan to address them. Activity planning and progress evaluation should include an analysis of how an initiative might stall and develop methods for accelerating progress (e.g. focusing public relations with partners on job creation to allay fears of the new Labor Law and resulting work force restructuring). BEP needs to follow-up on its activities and complete them so stakeholders do not have the impression that activities are left partially finished. 3. Eliminate or highly limit less productive activities and focus on high-impact activities that could be completed within the project duration. Relatively unproductive activities, as well as those enjoying or expecting significant support from other donors, should be eliminated. BEP needs to concentrate on fewer high-quality focused activities that have high impact potential and are implemented with stronger partnerships, rather than tackling a very comprehensive and wide range of problems. This will ensure effective and efficient use of resources. The following could be considered: 3.1. Activities to be closed:  Close activities such as state aid reform that cannot be completed within the scope of project duration.  Investigate whether to continue with PPP, given the wide scope and involvement of other donors. 3.2. Activities to be highly limited in scope:  Limit and focus work with Fiscal Council and National Assembly to build their capacities by transferring technical expertise at an operational level but in a limited fashion; the Fiscal Council is already at a high operational level, providing quality reports, while the National Assembly requires overarching reform beyond the scope of BEP.  Complete capital markets activities in the areas in which BEP can add value without significant resource utilization, with the exception of work related to public issuance of long-term government debt.  Confine the Commodity Exchange activity to draft law and limited adaptation of existing regulatory mechanism (Belgrade Stock Exchange and Central Registry of Securities), or, alternatively, identify other donor support, if a decision is made to upgrade the Novi Sad Product Exchange market instead of using the existing mechanism. 3.3. Activities to be pursued comprehensively and with further impetus:  Engage strongly with public sector representatives who champion reform in public finance management. This is an area where BEP could achieve significant impact by providing additional support to the public debt agency and Treasury with technical transfer of know-how and acquisition of software supporting innovation to improve operations (public debt management and payments’ management).  Undertake a comprehensive reform relating to para-fiscal charges, building on initial success.  Complete initiated reforms of labor legislation, construction permits, and inspections. 4. For further effectiveness, partnerships for reform advocacy should be intensified by doing the following:  Continue to use grantees effectively to provide thorough but more specific research and analysis, organize trainings, and raise public awareness to support advocacy, as well as augment the grant resources by including more funding.  Continue the Internship Program.  Coordinate business-demand side activities through groups representing small businesses, clusters, and the Exporters Association, and build their advocacy skills (smaller businesses have been less represented thus far in project activities).  Continue to seek improvement of the quality of economic journalism and develop new initiatives (e.g. a competition and prize for journalism on labor reform). 1 1.0 EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE This is a report on the mid-term performance evaluation of the Business Enabling Project (BEP) funded by the United States Agency for International Development (USAID) Mission in Serbia, and implemented by Cardno Emerging Markets. The evaluation covered the period January 2011 - June 2013, and was conducted by a team assembled by Mendez, England & Associates. The team consisted of one international and one local expert – both with experience in evaluating and/or working on economic growth projects in the region. The purpose of the evaluation was to provide USAID with: 1) a measurement of change in development outcomes attributable to the defined interventions of BEP, based on models of a broad range of qualitative, quantitative, or mixed methods; and 2) a rigorous, evidence-based analysis of BEP’s mid-term performance. Specifically, the evaluation was intended to: 1. Examine the effectiveness of BEP’s activities implemented to date 2. Determine whether BEP has achieved planned results 3. Identify gaps in BEP’s performance against targets 4. Provide recommendations on closing these gaps in the final years of the project 1.2 EVALUATION QUESTIONS The evaluation questions, per the Scope of Work (SOW), were: 1. How successful was the project in reducing the regulatory burdens to businesses in Serbia? 2. How successful was the project in increasing the Government of Serbia’s (GoS) capacity and transparency in public financial management? 3. How successful was the project in creating conditions for increased finance for the private sector? 4. How successfully did the project gather the support of relevant groups for reform? 5. To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? 2.0 PROJECT BACKGROUND BEP is a 5-year, $15.48 million initiative launched in January 2011. As stated in the SOW, the purpose of BEP “is to help the GoS improve the competitiveness of the Serbian economy and its private sector businesses.” The project supports Assistance Objective 1: More Competitive Market Economy, IR 1.1 Business Enabling Environment Improved, and IR 1.4 Macroeconomic Environment Improved. It aims to improve the ability of all businesses in Serbia to compete in global, European Union (EU), and domestic markets. BEP’s performance management plan (PMP) captures the impacts that the project will have, including the following Mission’s indicators (with source):  Decrease in the “Burden of Government Regulations” (World Economic Forum (WEF) Global Competitiveness Indicators (GCI));  Decrease in costs for businesses to comply with the regulatory framework (Standard Cost Model);  Increase in foreign direct investment (National Bank of Serbia (NBS) Statistics); 2  Improvement in the Open Budget Index (International Budget Partnership);  Improved government budget balance as percentage of Gross Domestic Product (GDP) (Ministry of Finance and Economy Report);  Improved “Financial Market Development” (WEF GCI Indicators)  Credit to private sector as a percentage of GDP (NBS Statistics) BEP also impacts several Mission indicators for Assistance Objective 2: Democratic Structures Strengthened. The project captures the impact on the following Mission indicators: a) Increase in number of US Government (USG) assisted civil society organizations (CSOs) that engage in advocacy and watchdog functions; b) Increase in number of draft laws changed by input from citizens/CSOs/Media Support Organizations (MSOs); c) Increase in number of policies that have been influenced by CSOs/MSOs; and d) Increase in number of public forums resulting from USG’s assistance, in which government officials and members of the public interact. BEP consists of three major components:  Component 1: Business Regulation and Economic Governance. Streamlines laws, regulations, and institutions, improve regulators’ capacity to implement laws, and improves analysis and public￾private dialogue.  Component 2: Macroeconomic Policy and Public Financial Management. Helps GoS implement new budget and fiscal responsibility laws, improve budgeting, increase public disclosure and transparency, strengthen public debt management, and improve economic analysis and forecasting.  Component 3: Financial Market Development. Improves regulatory framework for non-bank institutions, develops capacity to strengthen and expand capital markets, and improves conditions for debt market development. The project’s approach to achieving reforms is to help the GoS work closely with the private sector and outside experts to make reforms that improve business competitiveness. The reforms reduce the time and money spent by Serbian businesses to comply with laws and regulations and help GoS be more efficient. The reforms also enable businesses to attract necessary investment at an appropriate cost. As a result, Serbian businesses have a better chance to compete in the global economy, leading to export-led economic growth and increased employment. BEP uses an 18-person staff (16 Serbians, two expatriates), substantial Serbian and expatriate short-term advisors, and a grants fund to provide technical assistance to GoS and other counterparts. To date, the contract has utilized 9,460.56 days of LOE (3123.69 in 2011, 4338.56 in 2012, and 1998.31 in 2013), and expended around $8.1million in project funds, of which $796,900 was disbursed in grants. 3.0 EVALUATION METHODS & LIMITATIONS The Evaluation Team used a combination of different qualitative and quantitative methods in the evaluation process. Qualitative Research and Analysis Given the nature of the BEP’s activities (policy reforms are difficult to evaluate in quantitative terms), and time available to the Evaluation Team to conduct its inquiries, the evaluation of BEP was mainly qualitative. The Team’s approach was to identify, locate, and meet with as representative a body as possible with direct or indirect knowledge and/or experience of the project. By the end of the evaluation, the Team had interviewed 50 people, including the most relevant government representatives, international/donor organization 3 representatives, and non-government sector representatives. These were namely representatives of business associations and academic and research institutions, as well as representatives from USAID Serbia and the BEP team (see Annex III). In addition to direct, one-on-one interviews, the Team also conducted focus group interviews – one with experts from the National Alliance for Local Economic Development (NALED), and the other with company members of the Exporters Association of Serbia. To conduct interviews and ensure consistency across the evaluation, the Team developed a standardized questionnaire (see Annex II). The questionnaire was structured so that it first assessed the most pressing problems in Serbia’s business environment, which enabled the Team to evaluate whether BEP is involved in the most pertinent policy issues. It then requested an opinion with regard to progress in the issues that BEP is engaged in - without mentioning the project - to ensure objectivity. Only in the second part of the questionnaire were interviewees asked directly about BEP’s engagement in these areas (and hence pertaining only to interviewees acquainted with BEP). The same interview framework was used for the anonymous, online survey, administered via Survey Monkey, to cross-reference information obtained in direct interviews, especially with regard to selection of activities in priority problematic areas in Serbia’s business climate. The online survey was administered in Serbian to ensure a higher response rate. The survey reached a total of 92 individuals, mostly company representatives (summary, translated into English, has been provided in Annex III). The Evaluation Team specifically invited members of the Serbian Association of Businesswomen to participate in the online survey to ensure that the opinion of women was adequately represented. This effort led to 57% female participation in online survey, and 52% participation in direct interviews. The Evaluation Team also asked the Association of Corporate Directors of Serbia, the Exporters Association of Serbia, and several industry clusters to invite members to respond to the survey and, therefore, reached a relatively high response rate in the limited time available. This can be considered a representative survey since surveyed company representatives are members of active business associations and their opinions generally concur with the direct interviews, reinforcing the overall findings. One marked difference relates to companies’ greater interest in access to finance as an issue, which is unsurprising and it is discussed in the report. Quantitative Research and Analysis Quantitative data was sourced from BEP’s annual work plans and other project-related periodic reports, and cross-referenced with international reports on Serbia’s business-enabling environment published by the EU (Annual Progress Reports), World Bank (Doing Business), WEF (Competitiveness Report), OECD (Small Business Act Implementation Reports), and other donors. Particular attention was paid to the project’s annual Performance Monitoring Plans (PMPs) designed to provide ongoing analysis and commentary on actual results achieved against the plan, including actions and strategies that were taken to address anticipated deviation from the plan (see Annex III). In addition to specific analysis presented in the report per component, a table in Annex IV summarizes the Team’s assessment of the mid-term performance of BEP, relative to the planned objectives as envisaged in the original SOW and elaborated and/or revised in subsequent BEP annual work plans. Furthermore, whenever possible, the Evaluation Team employed a quantitative approach in survey analysis to present findings in a more clear and comprehensive fashion. Data Collection The Evaluation Team collected data by using a number of methods, including:  A critical desktop review of materials related to BEP and other relevant projects, as well as documents provided by USAID such as project reports and annual work plans, PMPs, performance indicators, training evaluations, and studies, among other sources. These documents provided a useful insight into the planning and operating activities of BEP, and were an important source of reference for data and information related to project’s objectives, activities, outcomes, indicators, and targets.  Interviews with USAID/Serbia, particularly the Economic Growth Office staff, and BEP’s implementers.  In depth, semi-structured interviews with selected stakeholders and beneficiaries, such as the Ministry of Finance and Economy (MoFE), the Ministry of Internal and Foreign Trade and Telecommunications (MoIFTT), the Ministry of Labor, Employment and Social Policy, the Serbian European Integration 4 Office, the Regulatory Reform Office (RRO)of the GoS, the Serbian Fiscal Council, the Securities Commission, etc.  Focus groups: The use of focus groups provided both quantitative and qualitative data to ensure context and background on the data obtained through the individual interviews. Focus groups were conducted with experts from NALED and with members of Exporters Association of Serbia.  Interviews with a number of non-government organizations (NGOs), research institutions, such as the Center for Liberal Democratic Studies (CLDS), Belgrade Open School (BOS), Economics Institute, and business associations such as the Association of Businesswomen of Serbia, the Serbian Association of Managers, the Foreign Investors Council, the Association of Corporate Directors of Serbia, the American Chamber of Commerce in Serbia (AmCham), etc.  Participation in USAID-sponsored event on economic impact of Serbia’s EU integration, held on June 27, 2013.  Direct observation to cross-check information and identification of factors not previously recognized.  Review of program outputs against objectives and performance indicators.  Interviews with other donors, including the EU, the World Bank, German aid agency GIZ, and others. The Evaluation Team analyzed the collected qualitative and quantitative information with the aim of answering the five main evaluation questions outlined in the SOW, as well as to provide information on cross-cutting issues that must be addressed by the evaluation, such as gender. The Team also reviewed the current indicators for performance measurement and proposed possible changes to the indicators. Findings have been tabulated both within the body of the final report and in annexes, summarizing the qualitative aspect of the evaluation, and addressing its quantitative elements. In evaluating results, the Team considered a number of factors in seeking a balanced judgment, namely:  Relevance to USAID and GoS’ goals, which was based on data collected from primary and secondary sources on key problems affecting the Serbian business environment, and feedback from stakeholders on the importance of the issues being addressed.  Effectiveness of BEP’s strategy in achieving the intended results, which was evaluated by reviewing the outputs of specific activities and their contribution to achieving BEP’s objectives, as well as BEP’s reports, other secondary data, and stakeholder feedback.  Efficiency in the use of BEP’s own resources (e.g. experts, grants) and in leveraging partners’ support and resources (expertise, reform groups), mainly assessed on the basis of feedback from stakeholders and known limitations in terms of time and financial resources. The Evaluation Team also looked at sustainability and impact, although the Team considers it premature to expect measures of impact and indications of long-term sustainability, given the complexity of problems and time frame required to generate momentum for achieving deep change. In this respect, sustainability is whether the processes and mechanisms put in place are sufficiently comprehensive to enable an improved public service (e.g. inspections), and ensure sustainable improvement in the mid and long-term, whereas impact is measured by the activity’s effect on improvement of the business environment and the economy (e.g. impact of construction permitting reform on new investment, employment generation, national income, etc.). The Team has use used a balanced judgment on a simple scale of there being: a) no or limited success; b) moderate success (whereby some important results have been achieved while others are expected from work-in￾progress); and c) significant success in achieving or exceeding the majority of objectives. EVALUATION LIMITATIONS  Low Response. The evaluation was conducted during the months of June and July, which carries a risk of low response since several stakeholders were away on vacation for one or two weeks. To mitigate this risk, the Evaluation Team scheduled the interviews at the beginning of the evaluation, interviewed in parallel, and used the Survey Monkey online tool to reach companies and gauge their opinion on the business climate, BEP and its impact. 5  Ability to Reach all Stakeholders. The Evaluation Team aimed to reach stakeholders in government institutions who tend to be in demanding positions and who are extremely occupied. Good time management in interview scheduling and previous professional acquaintance of many stakeholders ensured that all stakeholders were reached as planned.  Availability of Data. BEP has initiated a number of different activities, such as inspections reform or public administration management trainings, for which it is too early to have data to analyze their impact.  Pessimism. The Evaluation Team was concerned that the prevailing pessimism due to a difficult economic transition in Serbia - further aggravated by the world economic crisis and recent political sensitivities in the EU accession process - would negatively affect the responses. Aware of this possible bias, the Team asked the interviewees to try to assess the situation as objectively as possible and cross-referenced comments with different opinions and factual data. 4.0 FINDINGS, CONCLUSIONS & RECOMMENDATIONS 4.1 QUESTION 1: HOW SUCCESSFUL WAS THE PROJECT IN REDUCING THE REGULATORY BURDENS TO BUSINESSES IN SERBIA? 4.1.1 Key Activities Key BEP activities from January 2011 to June 2013 in reducing the regulatory burdens to businesses in Serbia include the following:  Improving Inspections: BEP supported the previous government’s working group (WG) on Inspection Reform and, following elections, the BEP’s organized Inspections Reform Leadership Group leading to the drafting of a new Law on Inspections, which will create a new legal framework for inspections. BEP has two staff on the four-person core drafting group established by the Ministry of Justice and Public Administration (MoJPA).  Reducing Para-fiscal Charges: In partnership with NALED, BEP completed a study on para-fiscal charges and provided support to the MoFE WG, leading to the elimination of 138 charges from a total of 370. The government also introduced legislation mandating that no new para-fiscal charges be independently introduced. However, as noted in the latest NALED Quarterly Report on Status of Reforms, published on July 15, 2013, several public institutions disregarded this legislation and introduced taxes whose level has not been defined by law and were not approved by the MoFE. Not a single institution complied with stipulation to review charged fees or obtain approval of MoFE for proposed revisions. BEP continues to provide support to the MoFE in organizing public discussions that should streamline the administration of and structure the remaining of para-fiscal charges.  Enforcement of Judgments: BEP supported MoJPA in drafting 16 ancillary bylaws that were adopted, thus completing the legal framework for enforcement and training of Public Enforcement Officers (PEOs). The training was, delivered in collaboration with GIZ, which will continue this support under an EU project. As a result, more than 80 PEOs are already appointed and operational.  Company Law & Corporate Governance: BEP trained judges in the new Company Law and developed the Codex of Corporate Governance, which was published in the Official Gazette and distributed to businesses by the Serbian Chamber of Commerce.  Construction Permitting: BEP completed a study of constraints to construction permits followed by a media campaign on the need for reform. Subsequently, it developed detailed recommendations for improving permitting and an implementation roadmap, as well as provided support to GoS in drafting a reformed Law on Construction. 6  Labor Reform: BEP prepared a study on policies to increase employment and workforce competitiveness, and worked with public-private stakeholders to engage them in dialogue and in reforming the Labor Law, which the GoS has pledged to complete by the end of 2013.  Regulatory Reform: BEP undertook a Standard Cost Model (SCM) study calculating the costs of administrative burdens and potential savings for business; It provided support to RRO in drafting the Strategy for Regulatory Reform; Drafted amendments to the now adopted Rules of Procedure requiring Regulatory Impact Assessment (RIA) for bylaws and financial burdens on business. Subsequently, in collaboration with GoS Human Resources Office (SUK), BEP trained public officials in RIA and laws implementation. The project provided technical support, which resulted in simplification of administrative burdens on employers and employees related to pregnancy and long-term medical leave, leading to savings of almost $10 million annually.  Enabling Public Private Partnership (PPP): BEP facilitated a public-private dialogue on the draft law on PPP, and supported the Commission for PPP in the development of a draft Value for Money Methodology (VfMM) for use by public officials in assessing initiatives. BEP also commenced training of local government and stakeholders on VfMM and the PPP model.  BEP supported public-private dialogue of businesses with GoS (covered in Section 4.4). 4.1.2 Analysis of Performance Data and Project Reports BEP’s progress is measured periodically with reference to the indicators in the PMP (see Annex IV). The PMP consists of Mission-level indicators for Creating a More Competitive Market Economy. (Note: The Evaluation Team understands that although BEP aims to achieve the indicators agreed upon with USAID, the emphasis on the project is to provide support in addressing key issues agreed upon with the GoS that are consistent with Mission’s goals and can be realistically achieved within the project’s timeframe and resources available.) The burden of government regulation is measured with reference to WEF GCI Score 1.09 “Burden of Government Regulation,” with a baseline of 2.34 (2011), and a slightly improved value of 2.4 (2012). BEP, in partnership with NALED, generated considerable momentum in its regulatory initiatives, and encouraged politicians to make regulatory issues part of their political campaigning in the period leading to the Serbian elections in May 2012. The perception of an improving environment was also noted in a marked difference between the BEP Business Survey results for 2011 and 2012, with 7% and 26% of respondents, respectively, perceiving an improving environment. Although the survey reported continued problems with the permitting process and inefficient enforcement, it registered progress in inspections. Thus, it is not unreasonable to say that there was an overall perception of improvement resulting from some limited change but also expectations in the wake of political campaigning. The second Mission level indicator is “Cost to Business in Complying with the Regulatory Framework,” with a baseline of 4.2% GDP (2011) calculated by using SCM, but the result is not yet available (due in the end of 2013). However, the reduction in charges reported from lower inspections costs is an encouraging signal. Preliminary NALED analysis (to be published with BEP’s support in the Quarterly Report on Status of Reforms on October 12, 2013) calculated annual savings of about 20 million Euros to Serbian businesses as a result of elimination of the “forestry fee.” Elimination of the “company name fee” is more difficult to calculate but it has created important savings for small businesses operating in central municipalities (based on company interviews, this amounts to several hundred Euros annually for a company registered in downtown Belgrade). A full review of the impact of removing para-fiscal charges should be undertaken later this year, after a full year has lapsed since this reform. The PMP (which BEP revised in Feb 2012) Component 1 indicators cover six objectives, although currently the project’s activities are driven by mainly four objectives after: a) the Competition Policy related activities following consultations with USAID’s agreement after the Commission for Protection of Competition declined BEP’s assistance; and b) the Corporate Governance and Company Law related activities were graduated after BEP completed trainings for judges on the implementation of the new Company Law and developed a Codex of Corporate Governance. The project is considering doing further work in the area of Corporate Governance since a project initiative partner, the Serbian Chamber of Commerce, is interested in providing training to companies on the application of the Codex. 7 Progress on the five (4 current plus Corporate Governance) objectives driving current activities is measured by a total of 16 indicators, of which 9 are achieving/exceeding targets and 4 are under-target. The outcomes for the remaining 3 indicators are expected to be measured this year (these include 1indicator for introduction of normative framework and bylaws for PPP, and 2 indicators related to corporate governance). The under￾target indicators relate to: judgment enforcement days, as well as the number of judges trained; construction permit processing measured by World Bank Doing Business (WBDB) Report; introduction of new labor legislation; and only a marginal off-target improvement in business’ perception of transparency and predictability of doing business (BEP Business Survey). BEP accepts that the above is a work-in-progress, and that many obstacles outside its control had been encountered. For example, the draft labor legislation is ready but there is a need to garner full support of the social partners (government, employers, trade unions) before the new law is enacted. However, at a conference hosted by AmCham and organized with BEP’s support, the government indicated this would be done by the year’s end, later noting that the Law on Strikes needs to first be adopted (the deadline for this is end of year, as confirmed by the Deputy Prime Minister of Serbia in July 2013). With regards to delays in inspections reform, this involves the MoJPA, which is currently focused on the overall reform of public administration, and the issues being dealt with by BEP are just a small part and not being accorded the same priority. Moreover, inspections reform requires changes to sector legislation (e.g. Trade Law), following reform of the Law on Inspections to facilitate practical change in inspection operations, thus calling for a comprehensive approach involving several line ministries. Despite the off-target results, the 2012 BEP Business Survey showed positive results contributing to the on￾target indicators. The percentage of businesses perceiving a reduction in the regulatory burden from 2011 to 2012 increased from 18% to 26%. The 2012 survey confirmed the poor performance of judgments and construction permitting but measured considerable progress in inspection time (52% reduction) and cost (22%). However, this measure is not included in the PMP, possibly because BEP has not yet initiated policy reform in this area. The latter also demonstrates the difficulty in using perception surveys to monitor progress. Interestingly, despite no new Labor Law, businesses’ perception of the negative impact of the Labor Law reduced from 75% in 2011 to 66% in 2012. Over the same period, BEP achieved good results in its work with the RRO and MoFE, which brought the implementation of RIAs and simplification of commercial laws and regulations. Furthermore, following a study on para-fiscal charges, BEP, in partnership with NALED, played a key role in putting this on the GoS agenda, resulting in the removal of a significant number of charges, as noted above. While only some of the charges directly affected business, the psychological effect of this reform should not be underestimated, since it created trust within the business community that the GoS was prepared to take steps to improve the business climate. There are several donors supporting PPPs (e.g. USAID, SLD, IFC); to date, BEP has made a useful contribution in supporting the development of the VfMM for assessing PPPs. It is notable that the development of the VfMM fulfills a recommendation (no. 8) in the 2012 Serbia EU integration progress report. The BEP Business Survey confirms that by working in the targeted areas, the project is addressing key issues. Both the 2011 and 2012 surveys showed that the factors viewed as having the most negative impact on the operation of Serbian businesses are wage taxes and contributions, VAT, quasi-fiscal charges, and administrative procedures. Of all the factors assessed (including inspection, Labor Law procedures, construction permits, court procedure, customs and trade), there are virtually no factors with a positive impact. The same is also confirmed by the Evaluation Team’s survey (see below). 4.1.3 Analysis of Stakeholder and Expert Opinions The Evaluation Team interviewed numerous stakeholders from the public and private sectors, government officials from counterpart ministries and agencies (referred to as GOV below), business associations (BAs), NGOs (such as economic think tanks), multilateral and bilateral international organizations (EU, international financing institutions (IFIs), donor agencies and projects), and found they were consistent in their view of the project’s effectiveness. In general, BEP is considered to be a highly flexible project that is willing to respond to the beneficiaries’ needs, and tackles solutions in partnership with them in an objective manner. This aspect of BEP’s operations is given specific consideration in Section 4.4. Interviewees acknowledged the project’s 8 success in putting key issues on the agenda and in catalyzing behavioral or attitude change and subsequent action regarding key regulatory issues. Notably, several business representatives and international donor representatives have highly praised the unified role of USAID and the US Embassy in Belgrade, including the vocal support of the US Ambassador that enabled both the BEP team and partners to raise awareness about key business enabling environment issues, such as Labor Law reform or suppression of grey economy. The World Bank, in particular, strongly values its cooperation with USAID and BEP specifically, since they have different types of assistance that, when combined, effectively enables greater reform progress. However, while there have been recognized successes (para-fiscal charges), in the other areas (e.g. inspections, construction permits, Labor Law, etc.) change has been slower than anticipated due to GoS’ inertia or dawdling, which is often beyond BEP’s control. The following is a synthesis of some expert views and opinions obtained from the above mentioned groups. When asked about whether the project was successful in reducing regulatory burdens, responses included:  The project is highly effective. It has achieved a great deal in relation to what was realistically feasible. BEP is an excellent facilitator and highly flexible. The support of the US Embassy and the Ambassador has been highly appreciated.  BEP has put regulation on the political agenda. This is a breakthrough but the results are yet to be achieved because there has been insufficient change to date in the areas of labor, inspections, and construction permits although there has been good progress on para-fiscal reform. Furthermore, regulation impact analysis has improved with BEP’s support.  BEP’s support has helped to raise expectations of change and this was reflected in the 2012 BEP Business Survey, which recorded a perception of a reduced regulatory burden.  Grants have been used very effectively in researching key issues and creating a basis for forming partnerships for reform with public and private sector stakeholders. The following is a synthesis and combination of remarks made with respect to specific activities:  Key issues Identified: BEP identified the most urgent problems for businesses and this was confirmed by the Business Surveys. It is important to work closely with all donors and agree who takes on which task with reference to Serbia needs and the acquis communautaire (acquis) requirements.  The Shadow Economy: This issue is the biggest problem because it puts a burden on legitimate businesses and drives corruption. Businesses in the shadows do not abide by laws and regulations and may not even know them. Few businesses operate entirely in the open.  Regulatory Reform: BEP commenced its support to RRO at the right time and helped the outgoing and, subsequently, the new government take brave steps in reform. However, civil servants find it difficult to carry out cost-benefit and RIA and, in some ministries, it is very difficult to get progress.  Inspections: The draft law is ready for implementation thanks to BEP but there are no final yet because of delays resulting from the slow process of public administration reform, of which inspections is a constituent part. A key reason for this is that inspection is just a small part of public administration reform and is not a top priority for MoJSA.  Para-fiscal Charges: BEP carried out a well-timed study in para-fiscal charges, after para-fiscal charge “flooding” in Serbia, resulting in significant changes. Now there is better predictability and the system for removal has been put in place. However, it is important to undertake a study to determine impact of such reforms once a full year has lapsed, building on preliminary analysis undertaken by NALED in October 2012.  Construction Permits: BEP put construction permits on the political agenda; however there has been slow progress. This is a complex problem and requires extensive changes to legislation and by-laws, as well as tackling other issues (e.g. converting usage to ownership, another major inhibitor of investment.) 9  Labor Law: Until the new law is in place, success cannot be claimed but BEP has met all expectations. The lack of political will and trade unions are the main obstacles to progress. Nevertheless, despite the existing Labor Law, companies find ways of skirting around it.  Enforcement: This continues to be a major issue and the main problem lies at the weak administrative support provided to judges.  PPPs: BEP has raised awareness about PPPs and without its support, all the relevant laws and methodologies would not be in place. However, the Committee on PPPs and Concessions has weak capacity and needs continued assistance. Since other projects are also supporting PPPs, perhaps the BEP project could withdraw to focus more on other areas. 4.1.4 On-line Survey, with Comparison to Direct Interview Findings The Evaluation Team conducted an online survey targeting companies on the key issues addressed by the project. 92 respondents answered the survey, of which 79.1% were businesses, 9.9% business associations and NGOs, and 11% other. When asked about obstacles to an improved business environment, respondents listed corruption as the main one (53.8%). Other obstacles are indicated in Fig. 1. Fig. 1: Main Obstacles to an Improved Business Environment Inspections’ efficacy (4.4%) and labor legislation (18.7%) were ranked much lower than the other obstacles but one could argue that gray economy is the cause of inefficient inspections and labor legislation, and hence that these are extremely relevant reform areas. In direct interviews, government and international organizations representatives noted corruption, shadow economy, and State Administration inefficiency as key problems in Serbia’s business climate, while business associations and research organizations noted corruption, construction permitting, and enforcement of contracts as their top three obstacles. Many stakeholders have also noted “Other”; in most cases this was judiciary and unpredictable policy-making. In the online survey, 49.4% of respondents believe that there has been no change in key issue areas, and only 20.7% have seen some improvement. A total of 63.2% believe that the overall business environment has worsened or remains the same. In direct interviews, government and international organization representatives expressed a similar critical attitude, while business associations and think tanks tended to value changes in the form of removal of para-fiscal charges or some improvements in public debt management. However, they also stressed that the improvement has been indeed very small, although it should be acknowledged. More specifically, surveyed companies perceived no improvement in construction permits, inspections, labor legislation, and para-fiscal charges (see Figure 2, next page). 10 Figure 2: Lack of Improvement These percentages are even higher for stakeholders, except in the area of para-fiscal charges, where the majority noted improvement, albeit with a repeated caveat that this is just the beginning and that more needs to be done in this policy area. Only about a third of online respondents had been involved in the project, and when assessing BEP’s impact on the business regulatory environment on a scale of 1-5 (1 lowest), 53.3% gave 3 and 26.7% gave 4, while more informed interviewed stakeholders granted higher marks (52% gave 4, 28% gave 3 and 20% gave 2). Graphic presentation of these results is available in Annex III to this report. 4.1.5 Analysis of Secondary Data (Other Reports) “The National Plan for the Adoption of the Acquis (2013-2016),” the “Serbia Progress Report 2012,” and the “Action Plan for Fulfillment of the European Commission’s Recommendations Under the Serbian 2012 Progress Report in the European Integration Process,” are important reference documents in relation to BEP’s activities, which should feed into the action plan. Moreover, they are sources of information on progress. The Progress Report for 2012 included the following statements, which are directly or indirectly related to BEP’s work:  Economic Rights: There has been little conversion of the right of usage into ownership of construction land.  Construction Permits: The setting up of new businesses is still hampered by difficulties in obtaining construction permit. The process remains lengthy and costly.  Informal Economy: Remains strong and is a main hindrance to fair competition and business development.  Labor Law: The report states there has been “no progress in the legislative activities in the field of Labor Law, social protection, and equal opportunities between men and women.”  Inspections: The report states that tax inspections have declined.  PPP: “There has been progress in the field of public procurement, particularly in the area of PPP” in the section on the Public Procurement Chapter, and the Action Plan refers to progress on the development of VfMM. However, it points to the need for implementation of the Law on PPP and training of staff of the MoFE on the award of PPP contracts. The “OECD Western Balkans Small and Medium Enterprises’ Policy Index, 2012” provides information on the implementation of the Small Business Act (SBA) for Europe. The report states that Serbia has notably carried out a comprehensive review of legislation and has undertaken measures to simplify it, and reports that RIA is in place in the six leading economies in the region, which includes Serbia. A comparison with the Western Balkans group shows that Serbia is ranked second, behind Turkey, for the regulatory reform framework for SME policy-making (SBA Dimension 3). However, regarding the operational environment (SBA Dimension 5), Serbia lags behind other regional economies in areas related to business start-up days and costs, the availability of one-stop shops for services to the business community, and e-based services. 11 The WEF GCI shows that inefficient government bureaucracy, corruption, and access to financing have been the top three problems identified in the 2011 and 2012 surveys. In both surveys, tax rates and regulations also show in the top 10. Restrictive labor regulations were 10th and 12th in the 2011 and 2012 surveys, respectively. Construction permitting puts Serbia at the bottom of the rankings, standing out in a negative way. CONCLUSIONS BEP is tackling issues which are in alignment with USAID’s country support strategy and EU acquis. Specifically, the project is dealing with regulatory issues recognized by the business community as obstacles to a better regulatory environment. This was confirmed by the BEP-funded and independently conducted survey, the views of interviewed public and private sector experts, and the online company survey conducted by the Evaluation Team. Mission objective intermediate indicators, and project component indicators, are generally on track, indicating that the project has played a beneficial role in creating the perception of an improving business environment that is less burdened by regulations (WBDB and BEP Survey measure). The impact on cost to business will be measured through a recalculation of the SCM, planned for later this year. Component 1 progress indicators reflect BEP’s effectiveness through such measures as the number of regulations simplified, and average time on inspections. However, some indicators are under-target (e.g. judgment enforcement days and construction permit processing), although it is generally accepted by stakeholders that this is for reasons outside the project’s control and due to hold-ups or inefficiency in the responsible public bodies. At the same time, improved inspection time target cannot be clearly attributed to BEP since activities in this area are still developing. Notably, PMP targets do not refer to Serbia’s EU requirements and this could be re-considered. Gender cross-component, although not regularly reported, it is partly tracked (e.g. trainings participation). Although the 2012 Business Survey showed inspections as improving, the Evaluation Team observed skepticism about improvement in these areas, as shown by the mini-survey carried out for this evaluation, and the opinions of public and private sector stakeholders. Perceptions and opinions were more positive about the para-fiscal charges initiative, which is not included as a direct measure amongst the PMP indicators. Information reviewed and interviews reveal that BEP, by partnering with NALED, has succeeded in putting these issues on the political agenda, building consensus on change, and facilitating a number of improvements (e.g. renewing initiative to draft a new Labor Law, improving the RIA process, etc). Limited progress has been made in the PPP’s framework and corporate governance. the success of para-fiscal charge reduction is considered to be important for psychological reasons, since it demonstrates GoS readiness to undertake reform, rather than for its impact, which is somewhat limited although still important for small companies. Public and private stakeholders applauded the efforts that BEP has made but there are some concerns about whether and when the final outcomes will be achieved, namely the introduction of the new Labor Law, improved construction permits, inspections’ reform, and pursuit of removal of administrative burdens. Although it is acknowledged that this is due to a combination of lack of political decision and/or weak public bodies, there is an underlying concern that the project will not follow the final objectives to achievement, leaving some initiatives semi-finished. This apprehension is influenced by experience with past donor projects. While considerable effort has been undertaken, with processes analyzed and documents generated, this has to be translated into policies, objectives and operational activities at the public body level. Interviewed experts believed that success at the operational level is dependent on political determination and championing at the highest echelons of government and its bodies. However, they all accept that the capacity of public institutions is also an obstacle. The success of the para-fiscal initiative is given as an example. Lack of progress in developing rules and procedures is largely due to the lack of political will and decision-making. Some public and private stakeholders believe that BEP activities in the regulatory domain will have limited impact on the business environment, unless they are part of an integrated approach addressing the shadow economy and corruption. The Evaluation Team heard observations from public and private interviewees that businesses learn to live with some of these burdens and find ways around them. At the same time, it is accepted that BEP cannot tackle everything and already has a very large portfolio of objectives and activities. 12 Moreover, since shadow economy and corruption are consequences of inefficient inspections or permitting process, reform in these areas will reduce these wider problems. BEP’s strategy of research and analysis, outreach, consensus building on reform, formation of partnerships, and targeted activities, is well conceived and works. As described in Section 4.4 related to evaluation questions 4 and 5, BEP has excelled in working with stakeholders. Research underpinning the activities has been thorough and carried out through grant-funded expert organizations. This has been an important component of the project’s success to date. In summary, the regulatory reform activities are highly relevant and have been carried out very efficiently, with very good use of internal and external resources. However, the effectiveness in terms of achieving specific results, other than reduction of para-fiscal charges, has yet to be achieved although the outlook is promising (e.g. with new Labor Law planned for the end of 2013). For this reason, the Evaluation Team concluded that, at the time of the evaluation, the project has achieved moderate success in the area of regulatory reform. RECOMMENDATIONS  Develop indicators with strong linkages to outputs and outcomes of activities. The PMP indicators should be linked as much as possible to the outcomes and impact of specific initiatives so that a more clear link is established between cause and effect. For example, the para-fiscal initiative could have had specific targets, particularly related to cost-savings, as well as quantity of charges. BEP could use grantee research to set realistic targets, which also need updating. For example, the enforcement target in the PMP relates to numbers trained but the ultimate effect sought is an improvement in enforcements. Thus, if this activity is to be continued, then better measures need to be developed. Incidentally, in this case, the number trained is below target but this might not matter if targets on enforcement were achieved. A more clear link should also be made with regard to EU progress targets outlined in the annual EU Progress Reports for Serbia, at least in the form of explanation in quarterly reports. Finally, BEP should report on the gender component, which is in part tracked (e.g. participation in trainings) but not regularly reported.  The project needs to follow-up on its activities and complete them. A case-in-point is the para-fiscal charges initiative, where follow up is needed to ensure sustainability of procedures in place to control para-fiscals and to assess the impact on business. Some activities require time for follow-up but stakeholders should be fully informed of work-in-progress. Impact assessment of reforms such as this one also helps build stronger partnerships with stakeholders and reinforces advocacy efforts for further reforms.  Develop strategies for unblocking. Activity planning and progress evaluation should include an analysis of how an initiative might stall and develop ways of accelerating progress. This is far from simple. BEP should subsequently find ways to apply its own resources in conjunction with those of its local stakeholders, USAID, EU, donors, and international organizations, to try to unblock progress. Criteria or scenarios should be developed to help decide whether and when to withdraw. A case-in￾point is the Labor Law, which is allegedly in limbo, although the GoS announced it would be introduced at the year’s end. It would be useful to find a way how to provide GoS with the courage and assistance in public relations to go down this difficult road. This would likely be easier with public opinion largely behind the issue. Therefore, there should be an emphasis on the job creation aspects of the new Labor Law to allay fears of greater unemployment and the negative effects of workforce restructuring. Linkages between this imperative, for example, and others need to be made and established. For example, resolution of conversion and ownership issues, which also requires political courage, will increase the availability of industrial infrastructure, which in turn will increase the urgency for (re)construction permitting, which in turn leads to new jobs. Thus, outreach and public information should be an important element of BEP’s strategy for tackling blockages.  Focusing on fewer activities with greater impact would likely enable a better application of project resources. BEP should consider eliminating activities which cannot be progressed further 13 or unblocked, or which are already the target of other donors, (e.g. PPP). Corporate governance activities could also be eliminated to enable focus on key activities, or pursued through partnership with a more credible business association than the Serbian Chamber of Commerce, which, while still useful for some local initiatives outside Belgrade, does not enjoy a reputation with businesses, particularly not in corporate governance. Given the above, the Evaluation Team acknowledges that BEP has in the past dropped activities that had no promise e.g. Competition Policy. Whenever possible, BEP should continue to identify other donors to take on initiatives, as it has successfully done with contract enforcement activity where GIZ will continue the endeavor.  Focusing on fewer activities will also enable BEP to engage resources to implement comprehensive reform. A case in point is inspections reform where sector laws, such as Trade Law, should be amended, following adoption of new Law on Inspections and inspectors trained. This calls for concerted, well-planned undertaking.  The use of grantees to do the research and analysis should be continued, but with a focus on impact analysis (evaluation of undertaken reform) and drafting more specific policy recommendations in select focus areas, as well as on organizing trainings to enable a comprehensive approach to reforms, and public awareness activities to support advocacy. This is given further consideration in Section 4.4 on BEP’s partnership building. 4.2 QUESTION 2: HOW SUCCESSFUL WAS THE PROJECT IN INCREASING THE GOVERNMENT OF SERBIA CAPACITY AND TRANSPARENCY IN PUBLIC FINANCIAL MANAGEMENT? 4.2.1 Key Activities Key BEP activities from January 2011 to June 2013 include the following:  Fiscal Council effectiveness: BEP provided technical assistance that helped the Council become one of the best ranked independent budgetary watchdogs in the region by the International Monetary Fund (IMF). It also provided advisory assistance on various key government proposals and helped expand its analytical capacity in fiscal decentralization, economic forecasting, and debt sustainability analysis.  Strengthened Parliamentary oversight: BEP increased capacity of parliamentary finance committee in budgetary oversight and knowledge of business reform legislation. It provided various trainings and outreach events in the National Assembly (e.g. a panel during the National Budget Forum on program budgeting), which helped increase awareness of Parliament’s role in budget oversight and promote fiscal transparency.  Adoption of program budgeting: BEP is the lead donor helping the MoFE adopt program budgeting in line with 2015 requirements. The project is playing a key role in securing ministry commitment and has been helping the MoFE roll out a program budgeting framework to budget users. BEP is assisting the MoFE to hold initial consultations and trainings with over 20 line ministries and independent budget users.  Improved transparency and public education of budgeting: BEP supported the development of a citizen’s guide to the budget that will be published by the MoFE.  Support to the General Secretariat (GS): The project carried out high-level work with the GS to review performance indicators by different line ministries, and supported a study entitled “Effective Use of Performance Indicators in Public Sector Budgeting and Planning.”  Donor coalition on public finance reform: BEP initiated an effort among donors to develop a unified position on public financial management reform. This effort eventually resulted in a joint position paper that was presented to GoS, citing priority areas for reform.  Program budgeting training for ministries: BEP helped SUK develop and deliver its module on performance-based budgeting to representatives of all ministries. 14  Structural deficit projections: BEP helped the Macro Fiscal Department of the MoFE develop a model and make the first structural deficit projections, as required to achieve EU candidacy. BEP’s support for improving economic forecasting now focuses on general equilibrium modeling.  Shadow economy: BEP sponsored and designed a comprehensive study, which combined macro￾level analysis with an enterprise-level survey, to understand sector-specific causes and consequences of shadow economy activity. The study received national attention and has been addressed by policymakers.  Improved risk management of public debt: BEP provided over 20 training sessions on risk management to the Public Debt Agency (PDA), including training on the development of a model for analyzing debt sustainability.  Improved reporting on public debt: BEP assisted PDA to improve its information systems for public debt management and reporting, including debt stock analysis and reporting on debt sustainability.  Management of state aid: The project assisted the MoFE to harmonize EU regulations for managing state aid and to develop the criteria and framework for assessing state aid regulation compliance.  Media capacity for economic reporting: BEP hosted various special briefings and a formal four weekend-long training program for journalists to improve their understanding of key economic issues.  Economic research and outreach: BEP sponsored various research and outreach events on fiscal policy topics and public finance management reform (annual economic conference in Kopaonik, the Vilvaldi Forum in Mokra Gora, Quarterly Monitor Presentations by FREN, etc.) on topics including budgetary reforms, the impact of fiscal decentralization, performance measurement in government, and national savings and public debt management. 4.2.2 Analysis of Performance Data and Project Reports The PMP includes a number of indicators that are linked to the objectives of Component 2, namely IR 1.4.1 Improved Fiscal Policy (OBI survey) and Improved Government Balance (%GDP), which are off-target. Of the 13 Component 2-specific indicators, 8 are reported as on target, 2 as off-target, and the remaining 3 will be determined during the course of the year. BEP believes that the success of 2 of the 8 on-track indicators (increase in government dinar denominated debt and reduction of GoS debt owed to banks) is partly attributable to improved methods and procedures adopted by PDA. According to the PDA website, the level of public debt totaled 59.3% in 2012 and, contrary to Fiscal Council numbers, has not exceeded 60%. Furthermore, PDA openly disagreed with the method of debt calculation. Serbian Dinar (RSD) denominated debt currently stands at 19.16%, so GoS has maintained a lower exposure to foreign currencies (currently about 50% Euro, 25% USD) compared to the baseline (14% RSD denominated debt in 2010). Other on-target indicators include targets that are easier to achieve, such as numbers of articles on fiscal policy published, and numbers of activities on improving public-private dialogue on macroeconomic matters. These include training GoS staff on macroeconomic policy and public financial management, which has greatly exceeded targets. It is interesting to note that the BEP Business Survey 2012 reported a significant increase in business awareness of economic growth strategy (38% in 2012 to 20% in 2011). Off-target indicators include the operation status of the public debt management system, which faced a vendor software problem, and a shortfall in the BEP supported publications on needed changes in macroeconomic policy. 4.2.3 Analysis of Stakeholder and Expert Opinions The Evaluation Team interviewed public and private sector stakeholders regarding BEP’s work assisting GoS to increase transparency and effectiveness of public financial management. Although most respondents were critical, they acknowledged some progress and insisted that this work is an important policy area where BEP should continue its involvement, focusing future efforts more on public debt and treasury financial management. Some respondents expressed doubts regarding the project’s budgeting success because of the 15 lack of political will in this area. The following is a short selection of comments from respondents with regards to public financial management issues:  Parliamentary Economy and Finance Committee: It is more active since working with BEP. Proposals for amendments are submitted. Reports from regulatory bodies have been received. Particular attention can be given to the guide for MPs on “how to read the budget” prepared with BEP’s help. This was very useful in improving the work of MPs.  Public awareness about public financing: There is more awareness about the need for improved public financial management; however, more needs to be done. The budget is still not linked to strategic priorities and line items are not respected. PDA improved evidencing but still needs a better framework and software to have data readily accessible. Treasury has a new program but can only monitor what is reported and, therefore, still does not monitor all payments.  Support to public bodies: BEP assisted the Fiscal Council with initial capacity building after it was established, by providing experts and methodology tools. In the future, the Fiscal Council could benefit from additional know-how, for example by having experts from the Congressional Budgeting Office deliver training and discuss best practices. BEP could help by providing IT solutions for public debt management and Treasury. 4.2.4 On-Line Survey, with Comparison to Direct Interview Findings The online survey included questions related to public financial management. Figure 3 below demonstrates the results. As the Figure indicates, only 19% of respondents have seen improvement. Figure 3: Capacity and Transparency in Public Financial Management This discrepancy can be explained by differences in perception on the issue, with some respondents reporting increased public debt, and others reporting increased transparency. This was corroborated by stakeholder interviews, with more informed interviewees noting specific areas of improvement while others provided information relating more to public awareness of the problem. Stakeholders assessed BEP’s activity in this area by an almost evenly distributed grade of 2, 3, or 4, with 3 prevailing, while the majority of companies (which are probably less informed about BEP’s activities but consider themselves to be informed) gave a grade of 2. Graphic presentation of these results is available in Annex III to this report. 4.2.5 Analysis of Secondary Data (Other Reports) The Fiscal Council May 2013 Report states that “public finances in Serbia are in a very bad position. The fiscal deficit will exceed 5.5% of GDP in 2013 instead of the planned 3.6% of GDP, even with the latest Government measures for deficit reduction. The public debt, which has already reached more than 60% of GDP, will continue growing in both 2013 and 2014. The key reason for lower public revenues lies in optimistic budget planning, not in the change in macroeconomic environment or the growth of tax evasion. Out of the total amount of RSD 80 billion deviating from the planned republic revenues, around RSD 55 billion is a result of bad planning. The Fiscal Council believes that the tax increase should not be even considered since the public finances problem in Serbia is not a low level of public 16 revenues but a high level of public expenditures.” This overview, which is in line with IMF reports on Serbia, stresses the magnitude of the current problem and the demand for major public sector restructuring and pension reform to reduce public expenditure, which is outside the scope of BEP. The EU Serbia 2012 Progress Report on Financial Control says, inter alia: “There was limited progress in the area of public internal financial control (PIFC). Legislation on internal audit and financial management and control (FMC) was further aligned with international standards in December 2011, but the PIFC policy paper needs updating because the action plan is outdated. Also, the Budget Law needs to be amended regarding provisions on FMC. Progress in external audit and the State Audit Institution continued to build up capacities but is still in the institution building phase.” On financial and budgetary provisions, the report states that a fully operating coordination structure is required to ensure the correct calculation, accounting, forecasting, collection, pay and control of resources: “There has been no progress regarding financial and budgetary provisions. Overall, the preparations in this area are at an early stage.” This information acknowledges some of BEP’s work but also points out that BEP may need to focus further on some of the identified problems in the absence of other donor projects at this time. CONCLUSIONS BEP has made an important contribution to building the capacity of the Serbian Fiscal Council and the Economy and Finance Committee of the National Assembly, especially in introducing methodology and short￾term expertise that enabled transfer of best practices at an operational level. Program budgeting is a particularly challenging activity which has, until now, absorbed a high level of donor funds with insufficient progress as a consequence of weak political will. However, with political will, this type of reform would lead to the momentous improvement of public finance management. This is an example of where it is necessary to engage with GoS at the highest level in order to assess political will and agree upon a strategy for moving forward or dropping this activity in favor of another. An important factor in such discussions is public finance reform as part of the EU accession process. Public debt management and Treasury payments management are areas where the beneficiary has demonstrated strong interest in advancing reform, and where progress has already been achieved, facilitating further reform steps. BEP provided know-how in public debt management, which has been highly valued but included fewer activities involving Treasury. It would be vital to support PDA further because this is an area of activity where BEP can have the most significant impact on Serbia’s business climate by enhancing public finance management. According to the World Bank, GoS has to collect 4.5 billion Euros each year and pays about 960 million Euros in interest expenses, meaning that even small enhancements in public debt management could lead to millions of Euros in savings and improve Serbia’s international financial credibility and hence investment security. Economic research and public outreach generated a basis for further reforms targeting the causes of the shadow economy, such as inspections, which are undertaken under Component 1. A more sophisticated approach is required to achieve the same in policies directly impacting public financial management. BEP’s technical assistance in improving state aid legislation directly contributed to Serbia’s achieving EU accession requirements. Nonetheless, even though state aid is one of the principle EU policies, BEP ended related activities because this is a wide policy area that requires resources beyond the scope of the project. Apart from the technical support to the public bodies involved in public sector financial management, BEP aims to improve business awareness of GoS’ economic growth strategy through outreach events and training, and cooperation with the media. As a result, business awareness has increased from 20% (2011) to 38% (2012) with firms reporting familiarity with GoS strategy. Though there were skeptical remarks from some non-specialist stakeholders about meager results, the Evaluation Team believes that on balance, BEP is making progress and there are prospects of higher effectiveness in the future. Thus, BEP’s success in Component 2 is moderate at present. RECOMMENDATIONS  Limit capacity building to Serbian Fiscal Council and the Economy and Finance Committee of the National Assembly. BEP should continue to build capacity of the Serbian Fiscal Council and the Economy and Finance Committee of the National Assembly by transferring technical 17 expertise at the operational level, but in a limited fashion, since the Fiscal Council is already at a high operational level providing quality reports and the National Assembly requires overarching reform beyond the scope of BEP. These capacity building efforts should be focused on building momentum to implement program budgeting and should be supported by public advocacy campaigns implemented by relevant NGOs through a grants program. Such efforts should include participation in high-level economic forums such as the Kopaonik business forum. However, the program budgeting activity should be revisited in six to nine months and eliminated, should the political will continue to be meager, in order to provide more support to public debt and Treasury management.  Engage strongly with public sector representatives who champion reform in public finance management and where significant impact could be achieved. The PDA and Treasury should be additionally supported by transferring technical knowledge and acquiring software to support innovation to improve operations (public debt and payments management). Economic research and analysis should be continued only in a concentrated, activity-specific manner, and to support public advocacy initiatives to forge wider partnerships pushing for a higher degree of reforms. Similarly, future trainings for economic journalists should be linked to these activities and should take the form of workshops rather than the classical training format.  Close activities that cannot be completed within the scope of project duration, such as state aid reform, which is also an area where the EU will pay particular attention, and provide funding (some is already provided). 4.3 QUESTION 3: HOW SUCCESSFUL WAS THE PROJECT IN CREATING CONDITIONS FOR INCREASED FINANCE FOR THE PRIVATE SECTOR? 4.3.1 Key Activities Key BEP activities from January 2011to June 2013, excluding activities aimed at strengthening public-private dialogue and which are covered in Section 4.4, include the following:  Capacity building of the Securities Commission (SC): BEP provided risk-based supervision training to the SC to build its capacity to regulate capital markets.  Improving disclosure: BEP supported the SC in meeting the requirements of the new Capital Markets Law to strengthen reporting and disclosure, and in identifying options for creating an electronic information register on intermediaries and listed companies, including grant support to develop software for this system.  Capital market legislation: BEP supported the development of 18 bylaws for a new Capital Markets Law. Participated in WG on Harmonization of Securities Regulations aimed at assuring consistency of numerous laws affecting capital markets. BEP participated in drafting amendments to the Capital Market Law, Takeover Law, Investment Law, and Company Law, as well as ensuring that changes in the regulatory framework are consistent with EU and international practices.  Legal framework for the Commodity Exchange: BEP supported GoS counterparts in developing a concept for modern Commodity Exchange mainly focused on agricultural commodities. BEP provided input for a draft law, training on the drafting process under the previous government, and following elections, continued support to GoS counterparts led by the MoIFTT in improving this draft, given that the original had a number of deficiencies. Participated in WG for finalizing the new Commodity Exchange Law, including RIA and stakeholder discussions, and assessed implementation models which include limited upgrading of the existing Belgrade Stock Exchange versus development of Product Exchange (Produktna Berza) in Novi Sad.  GoS access to finance through long-term (LT) public debt: BEP identified useful information on the limitations of and opportunities for the development of LT public debt securities through a round-table with market participants and policymakers. It supported the PDA in assessing viable options for a government retail debt program. 18  Access to SME finance: BEP completed a ground-breaking document on financing growth of SMEs, which included over 50 recommendations that created a basis for reform that was accepted by the MoFE and business community. Recommendations included: a) liberalization of value added tax payments for SMEs through the introduction of cash-based payment rules and reduction of waiting time, which were subsequently adopted by the Parliament; and b) supporting the MoFE in developing legislation, namely the Factoring Law, including RIA and stakeholder discussions1 for the Law on Late Payment (in line with EU practice), which was adopted at the end of 2012. The drafted Factoring Law is in line with BEP’s work to increase capital availability from non-banking financial institutions (NBFI) as well as ongoing work to improve the leasing regulatory framework, develop a regulatory framework for other non-bank credit institutions, and planned advocacy and assistance to the MoFE to amend collateral regulations (laws on mortgage and pledge) in order to improve the out-of-court and court enforcement mechanism in case of debtor default. BEP developed a grant scheme for equipment purchase for the MoFE. It will continue to provide technical advice to the MoFE and the new SME Council, including assistance to the Council's Access to Finance Expert Group to develop an inter-governmental action plan for improving access to finance based on the findings of BEP’s White Paper and to develop an inter-governmental Business Enabling Action Plan. BEP will also assist the MoFE in drafting the national SME Development Strategy, 2014 – 2020 in the areas of SME access to finance and business regulation (construction permits and inspections, in particular). Finally, BEP plans to strengthen the capacity of business associations and business groups to advocate for access to finance reforms and assist selected business associations and SME groups to develop dedicated financing programs with banks and other lenders. 4.3.2 Analysis of Performance Data and Project Reports The two Mission-level indicators directly related to Question 3 are the WEF GCI Financial Markets Development Indicator, which was under-target for 2012 and remains static, and the Credit to the Private Sector as a percentage of GDP Indicator, which NBS reported to have grown from 43.7% to 54% during the period 2011 to 2012. BEP agrees that this increase cannot be related to its activities and explains that it was the combined result of a stagnant GDP and funds that had previously been taken out of the banking sector flowing back in. The nine Component 3 PMP indicators relate to strengthening capital markets, improving GoS’ access to finance through a LT liquid debt market, and improving SME access to finance. The SME financing objective is now the main thrust of this component since it is identified as a major obstacle (eg. BEP business survey and WEF GCI report). Three out of the nine indicators related to GoS’ access to finance are reported to be on target, whereas the remainder are either off-target or undetermined. Of the undetermined targets:  NBFI financing has yet to be estimated from a survey of leasing and factoring, which would set a baseline.  The establishment of a Commodity Exchange is targeted for this year but BEP reports that it is unlikely to be achieved since the legal framework is being revised and there is ongoing discussion on where it would be institutionally located. Progress on improving GoS’ access to LT public debt is measured by the value of dinar debt obtained from the Statistics Office. This value has increased from the baseline of 14% (2010) to over 18% (2012), whereas the public debt owned by banks has reduced from 87.1% (March, 2011) to 80.38% (November 2012). BEP is confident that these two results are a direct consequence of its work with PDA, which has improved its risk 1 Note that factoring already existed as a finance option in Serbia but that prior to the new law adopted in July 2013, it was based on a factoring contract rather than a specific law. Therefore, the new law reduces risk and hence may also lead to reduced cost of factoring service. 19 management procedures and has subsequently increased dinar debt. BEP attributes the reduction in government debt held by banks to the increased activity of investment funds. The perception of improved public finances management by professionals could also explain this movement. Regarding SME finance, the 2012 Business Survey revealed a worsening situation in terms of perceptions about the ease of access and the size of average loans, which is understandable given the state of the Serbian economy and the increases in the cost of borrowing. This information is corroborated by the findings of the BEP Business Surveys on SME financing. Although borrowing has reduced, it is in no way a reflection on BEP since SME finance-related actions have mainly focused on NBFI and the new Factoring Law, which just took effect on July 16, 2013. Furthermore, the emphasis on tackling SME financing under this component is new and a range of activities is currently under development. This will be completed with reference to the project’s in￾depth research into SME finance in Serbia. 4.3.3 Analysis of Stakeholder and Expert Opinions The view of business associations and government officials is that BEP has done as much as it can at this stage for capital markets. They generally value contributions to the development of a legal framework for the Commodity Exchange, as well as work performed on the Factoring Law. However, the impact of this work has yet to be seen. The following is a compilation of comments:  Capital markets reform: BEP assisted the regulator in drafting bylaws and procuring software for data management. New laws were introduced and BEP cannot do more. The new Capital Markets Law is a good piece of legislation but must still be fully adjusted to the Serbian level of development and more support is needed.  Commodity Exchange: BEP has made important contributions to the Commodity Exchange Law but project progress and results remain to be seen. Questions were raised about the reason for creating a new Commodity Exchange when the Belgrade Stock Exchange could be adapted for this purpose. Since the processes are the same as with securities, rather than developing new institutions, the new Commodity Exchange could be organized within existing infrastructure, using the already developed Belgrade Stock Exchange and the Central Securities Depository and Clearing House.  Access to SME finance: The SME finance research and report was good and businesses are now more aware of the challenges. There has been progress on the Law of Factoring but access to finance is still a major issue. Small companies, with 10 or fewer employees, face SME financing problems because banks are not interested in working with them due to their general inability to satisfy the bank requirements and interest rates. Although big banks have been very successful in protecting their interests and in serving big businesses, they will hopefully now adapt to SMEs. Delayed payments are a main cause of financial problems, because they cause companies to deteriorate, thus making credit more difficult to secure. Companies that are involved in the shadow economy have difficulty presenting the real situation of their businesses. Before creating new instruments, the existing ones that are not working properly need to be fixed. 4.3.4 On-line Survey, with Comparison to Direct Interview Findings The online survey included questions related to the development of financial markets. Approximately 32.1% believe there is no change, and 37% believe there has been some deterioration. Notably, the online survey also provided a platform for companies to voice their complaints regarding the tax burden level, rather than complexity, reinforcing the need to increase access to finance while also improving public finance management. Graphic presentation of these results is available in Annex III to this report. 4.3.5 Analysis of Secondary Data (Other Reports) The EU Serbia Progress Report 2012 states that there has been little progress in aligning financial market infrastructure with the acquis. However, progress has been noted with the Securities Commission adopting legislation to regulate the capital market framework. The report also states that the SME Council had not been established, and that further efforts are required regarding SME access to finance. 20 The OECD SME Policy Index Report has access to finance Dimension 6, in which Serbia scores the highest in the region, although all economies fall significantly short of the standard set by the SBA. CONCLUSIONS BEP has been less effective in the area of capital markets development than in helping to reform legislation on Commodity Exchange and supporting SME access to credit. The capital trade volume has been further reduced as a consequence of low economic activity in the wake of the global financial crisis and the fact that most foreign-owned banks and large public enterprises are not publicly listed and, therefore, not publicly traded. However, there is one notable achievement, namely BEP’s support in the development of a new Factoring Law, which enabled improved regulation of this financial service. The project has re-focused this component to concentrate predominantly on SME access to finance; this is commendable since BEP will dedicate its resources to tackle aspects of a major problem for Serbian businesses. Development of capital markets has stalled in the eyes of many stakeholders. However, they acknowledge the usefulness of work done on capital markets’ legislation, as well as the support provided in improving data management technology. Although the MoFE has proposed that further support be provided to better adapt capital markets’ legislation to local conditions, it is questionable how effective it is to continue the focus on this activity, given other project priorities. It may be more consequential to continue building a market for public debt issuance since this is related to public sector financial management. Stakeholders appreciated BEP’s support in drafting legislation enabling Commodity Exchange. However, they disagreed on what institution will host the exchange, citing their preference for the Belgrade Stock Exchange. The prevailing opinion is to entrust Securities Commission and Central Registry for Securities and Clearance House regulate exchange of commodities instead of expending resources on the Novi Sad Product Exchange and additional modes of regulation. BEP has made an important contribution to the problems of NBFI financing through its work on the Factoring Law. This needs to be followed through to its conclusion along with other initiatives, such as the development of SME leasing, a popular source of financing in other transition European countries. Special attention should be paid to export financing, while micro-financing initiatives require more robust government support. The project has re-focused Component 3 to SME finance; this is a major problem given that the majority of Serbian businesses are small and outside the zone of interest of banks, which prefer to focus on big businesses. BEP conducted a study on SME finance, which explains the multi-faceted nature of the problem on both the supply and demand sides, warranting additional donor attention. The project’s support to the MoFE to date has involved, inter alia, drafting of Factoring Law, and the development of a small new grant supported loan scheme for equipment purchase. The project has recently selected several activities with focus on SME finance (outlined above) and these may need to be revisited in six months to ensure that they can be implemented in the remaining years of the project. Based on the above, the Evaluation Team concludes that BEP has achieved limited success in this area but its refocus on SME finance should lead to higher effectiveness in the second half of the project’s term. RECOMMENDATIONS  Capital markets activities should be concluded in the areas in which BEP can add value without significant resource utilization, with the exception of work related to public issuance of LT government debt.  The Commodity Exchange activity should be confined to draft law, limited adaptation of existing regulatory mechanism, and Belgrade Stock Exchange. Alternatively, if a decision is made to upgrade the Novi Sad Product Exchange market instead of using the Belgrade Stock Exchange, other donor support must be identified.  On SME finance, BEP has recently selected areas of focus and should revisit these in six months to ensure timely implementation or transfer activity to other donors. Collaboration with IFIs, such as the European Investment Bank or the European Bank for Reconstruction and Development on SME lending, is essential since these organizations have been 21 effective in increasing access to finance in neighboring countries. When working with the GoS it is necessary to consider where to target support in order for it to reach business sectors with the best prospects of development. Demand side activities should be coordinated through groups representing small businesses and clusters representing small companies. Gender component should also be considered since research and the OECD SME Policy Index demonstrate specific problems related to access to finance by women-owned businesses. 4.4 QUESTIONS 4 & 5: HOW SUCCESSFULLY DID THE PROJECT GATHER THE SUPPORT OF RELEVANT GROUPS FOR REFORM? TO WHAT EXTENT HAS THE PROJECT BEEN EFFECTIVE IN DEVELOPING PARTNERSHIPS THAT HAVE CATALYZED REFORM IN AREAS IN WHICH THE PROJECT OPERATES? 4.4.1 Key Activities These two questions have been grouped together in this section because they are linked. Question 4 relates to the project’s success in gathering relevant support groups, whereas Question 5 relates to its effectiveness in partnering with stakeholders in catalyzing the reform process. The Evaluation Team has expanded this topic to include those aspects of BEP’s operations, which lend themselves to building support from the public and stakeholders connected with specific issues, including the use of grants, introduction of internships, application of trainings, gender approach, and outreach and use of social media. The BEP Business Survey is also a successful and highly appreciated (see section on stakeholder interviews) instrument for reaching out to the business community to obtain views for identifying needs and directing project focus. BEP has worked with local and public stakeholders and cooperated with international organizations in all three components. It has used its grant schemes to create a basis for launching activities, which have been reinforced through partnership with local stakeholders. The capacity of public institutions has been strengthened in partnership with BEP through the introduction of the GoS Internship and Public Administrators Training Program. The following includes a summary of stakeholders and groups with which BEP collaborated in the aforementioned areas2.  Component 1 - Business Regulation and Economic Governance: BEP collaborated with the following institutions and organizations in pursuit of its Component 1 objectives: MoFE, MoJPA, Ministry of Health, Energy and Environment, Ministry of Labor and Social Policy, Ministry of Construction and Urbanism, MoIFTT; Economic and Social Council; GS and SUK; RRE; Inspections Leadership Group (initiated by BEP); business associations including the Serbian Chamber of Commerce, the Foreign Investors Council, the Serbian Managers Association; NALED; Employers Union; Trade Unions; PPP Commission; Chamber of Engineers; Standing Conference of Towns and Municipalities; Fire Protection Association; Association of Construction Engineers of Serbia; and City of Belgrade.  Component 2 - Macroeconomic Policy and Public Financial Management: MoFE; Macro Fiscal Department; Fiscal Council; National Assembly.  Component 3 - Financial Market Development: MoFE; MoIFTT; Securities Commission; PDA; NBS; National Agency for Regional Development; National Association of Valuers and Surveyors (NAVS); Leasing Association; Factoring Association; business associations (as above); and clusters.  International Organizations/Other Projects: EU, GIZ, ILO, USAID, World Bank.  Grantee (examples): MoFE; Centre for Liberal Democratic Studies; Economics Institute; Foundation for the Advancement of Economics; Balkan Centre for Regulatory Reform; Serbian Association of 2 (Note: Some ministries were restructured following the election and do not in all cases correspond to the names of government partners at the project’s start). 22 Managers; NALED; BIRN; AmCham; Association of Leasing; Centre for Follow-Up Data Analysis; BOS; Serbian Management Centre; Belgrade Stock Exchange.  Internship and Management Training: MoFE; Ministry of Labor; MoIFTT; Office for Regulatory Reform; Securities Commission; National Assembly.  Gender aspect: Throughout its work, BEP attempted to ensure that the views of both men and women were heard, and that special attention was provided to women in business. The project cooperates with women associations (Serbian Chamber of Commerce’s Women-in-Business Group, Employers Association’s Women-in-Business Group, UN Women, Network of Women in Parliament, Etno Mreza, etc.).  Outreach and Social Media: ‐ Since the beginning, BEP had a total of 1,338 media reports, all with positive or neutral tone. ‐ Strategic communications and advocacy efforts have led to advances on several reforms, prodding the government to move forward with: ‐ Inspections reform ‐ Labor Law reform ‐ Removal of para-fiscal charges ‐ Streamlining the construction permitting process ‐ Increasing access to finance ‐ Addressing the shadow economy ‐ Journalists’ trainings, combined with systematic and day-to-day contact with both local and national journalists, have contributed to more sophisticated reporting on the economic challenges that Serbia is facing. ‐ BEP YouTube channel features more than 90 videos. BEP has two twitter accounts (@USAID_BEP and @konkretnoSrbija), a Facebook fan page http://www.facebook.com/KonkretnoSrbija?fref=ts, and three websites (www.bep.rs, www.policycafe.rs, and www.konkretno.rs). So far, the Facebook fan page has been actively spreading relevant content from the project, resulting in more than 3,000 people liking (following) the page. News (including videos, photos, relevant articles, studies, analyses) was posted on daily basis and resulted in an average of 100 to 300 likes and shares. The Twitter account has attracted 200 followers. ‐ Business Surveys: Conducted by grantee Center for Free Elections and Democracy (CeSID) in 2011 and 2012. 4.4.2 Analysis of Performance Data and Project Reports The PMP includes a number of indicators that are specifically linked to the objectives of bringing reform groups together, and developing partnerships for catalyzing change, as well as targets for generating public awareness and support, since this too is part of the process of generating widely based support for change. In this regard, there is a target for training journalists for the purpose of improving the quality of reporting on economics and business at the Mission level. The delivered training hours exceeded targets significantly. Component 1 includes activities and two corresponding indicators for measuring business: GoS dialogue and partnership in the reform process. BEP has exceeded both targets. The BEP Business Survey has been used to measure the business sector’s participation in the reform process (19% to 25% over 2011-12), and the number of BEP dialogue mechanisms/events on regulation are well above target. Component 3 also includes a target related to the strengthening of public-private dialogue on financial sector issues, and awareness, measured by the BEP Business Survey, has grown from a baseline of 25% in 2011 to almost 84% in 2012. Notwithstanding the dedicated objectives and corresponding activities, all of the three components’ objectives could have not been achieved without BEP facilitating inclusion of all relevant stakeholders, close collaboration between government institutions and, in many cases, dialogue and cooperation between the public and private sectors. The list of stakeholders provided in the preceding section shows the breadth of BEP’s engagement with them. There is no doubt, and this was confirmed by the interviews that the Evaluation Team conducted, that BEP has succeeded in identifying and approaching all the relevant parties involved in the specific reform areas that it has targeted (see following section). While an exhaustive list of examples is outside the scope of 23 this report, the following is a selection of how the project has used roundtables, workshops, outreach initiatives, grantees, and other measures to build and cement the relationship between itself and stakeholders, and to catalyze results:  Regional roundtables on “Business Speak Inspections Reform,” involving NALED, Foreign Investors Council (FIC), AmCham, and the Employers’ Union (Q2 – Q3, 2011), resulting in a GoS WG on Inspection Reform. After the elections, BEP organized the Inspections Leadership Group, comprising inspectorate, business associations, and trade union representatives. The Leadership Group was formed to continue to work off the WG formed by the previous government, and has developed a Strategy on Inspections.  Roundtables with NALED on Streamlining Construction Permits, involving private and public sector stakeholders (Q2-Q3, 2011) leading to cooperation and activities with the Republic Geodetic Authority and the responsible ministry (currently Ministry of Construction and Urban Planning), which has announced law amendments and plans to simplify the process.  The AmCham conference on business regulations (Q4, 2012), involving the Prime Minister and key government officials, during which the GoS committed to labor reform.  SCM and Business Survey presentation to media (Q4, 2011).  Finance roundtables with SMEs to understand needs (Q4, 2011, Q1, 2012).  NALED grant to conduct an assessment on quasi-fiscal charges (Q4, 2011) presented to the Advisory Group for Business Initiatives, ultimately leading to the implementation of measures by the next government and abolition of a group of para-fiscal charges.  CLDS grant on Policies to Increase Employment and Workforce Competitiveness.  Study on shadow economy in Serbia.  WG formed to develop a new law on Commodities Exchange.  Meetings of Labor Policy Reform Group involving business associations leading to agreement on priorities for reform. Consensus building with unions at retreat in Arandjelovac (started Q1, 2012).  Third annual PPP, that BEP supported, involving local stakeholders and international experts.  WG Access to Finance formed in 2013.  Media breakfast events, started in Q3, 2011. The project’s quarterly reports provide exhaustive lists of media reporting related to BEP’s activities. Training to journalists on economics and business to improve reporting quality.  Social media. Apart from its own website, BEP also has the Policy Café website where anyone interested in policy issues can find information and connect with other interested parties. There is also a grant-funded website, Konkretno, focused on Construction Permits, Labor Law Reform, and SME finance. The site is also a platform for interested parties to get information, connect, and engage in discussion through Facebook and other social media. According to BEP, Policy Café is not used as extensively as had been expected.  “Ask When” TV campaign, in cooperation with NALED, to advocate for removing red tape related to pregnancy leave. Campaign resulted in new procedure and led to abolition of 138 para-fiscal levies.  Collaboration with SUK on management training of civil servants.  GoS Internship Program launched in Q1 of 2013, in collaboration with BOS, provides support to 5 GoS bodies and the National Assembly. The above list is not exhaustive but provides a snapshot of the considerable efforts that BEP has made to generate public awareness, bring stakeholders together, create reform groups, and catalyze change. These activities have produced some immediate results (e.g. para-fiscal charges), while other initiatives are works-in￾progress, even though progress is slow in some areas. These activities are in line with the project’s strategy to: a) carry out studies and analysis with the support of expert grantees to create a basis for reform efforts; b) use media, round-tables, conferences to mobilize coalitions and motivate GoS to reform; and c) form WGs with key public and private stakeholders to pursue reform objectives. The Evaluation Team looked into how BEP would leverage the small business community, given that it was fragmented and arguably not well-represented among stakeholder groups. BEP staff replied that they planned 24 to strengthen the capacity of business associations and business groups to advocate for access to finance reforms, and assist selected business associations and SME groups to develop dedicated financing programs with banks and other lenders. BEP’s findings are that SMEs’ collective voice is not sufficiently articulate, or strong, for it to pressure the GoS to understand how much the Serbian economic growth depends on SME development. Given that GoS has not provided sufficient attention to the necessary reforms, the aim is to strengthen the advocacy and outreach skills of business associations whose members include SMEs. More effective advocacy will help business associations identify a pilot joint financing program for the business and financial sectors, develop a practical guide for SMEs on financing issues, and explore possibilities to increase the transparency of information that SMEs need in order to make informed decisions. The Evaluation Team discussed the grants program with BEP to determine the extent to which it supports its activities, notwithstanding the fact that there are some well-known examples of basic research and analysis that provide a basis for launching activities. BEP informed the Evaluation Team that the majority of the grants were directly linked to the project’s objectives and provided inputs (e.g. studies, surveys, events) to the project’s activities with some exceptions (e.g. Foundation for the Advancement of Economics’ (FREN) Quarterly Macroeconomic Reviews and the CLDS’s Study on Savings). BEP is in the process of monitoring grantee projects to look at the outputs and outcomes, and to assess impact. The Evaluation Team looked at training scores for the range of trainings provided by BEP, including those delivered by grantees, and in cooperation with SUK, and found them to be satisfactory. However, the training evaluation does not guarantee effectiveness of application in the work place. The Evaluation Team discussed the quality of economic journalism, in the light of views heard during stakeholder interviews about its low quality. BEP staff responded by saying that it maintained regular contact with a number of high-quality economic journalists but accepted that this did not apply broadly. 4.4.3 Analysis of Stakeholder and Expert Opinions The Evaluation Team interviewed public and private sector stakeholders to discuss the project’s effectiveness in gathering support and catalyzing change. When asked how successful the project was in gathering the support of relevant groups for reform, all interviewees proclaimed that this was one of BEP’s major strengths, namely its ability to win the attention of stakeholders, to be objective, to demonstrate expertise, and to facilitate discussion on change and solutions to key issues. When talking about the Labor Law initiative, some remarked that it is very difficult to get the trade unions involved. One interviewee noted that, based on experience with other projects, BEP was the only one that had been successful in bringing businesses, government, and trade unions together. However, there were some negative remarks, too, including: a) there was insufficient cooperation from a public sector institution in relation to SME finance; b) not all representatives of reform groups were at the “appropriate level,” made in relation to the Labor Law work; and c) members of parliament were not engaged to the extent required in the work of the Parliamentary Committee on Finance, State Budget and Control of Spending. On the question of partnership, the views were more guarded, particularly those of the state institutions representatives. Several interviewees observed that although stakeholders had been brought together, the partnerships required further strengthening to be effective. One senior ministry representative said that it was difficult to motivate (civil servants) without political will backing the overall reform process. The following is a short selection of comments from the above stakeholders (with an indication of the sources): ‐ BEP is the only project that has been successful in bringing together business associations, unions, and government to discuss reform issues (A business association). ‐ Things will get stuck at the operational level and the only way to get them moving is by having commitment from the very top (A government body). ‐ All relevant parties were involved in the Parliamentary Committee work but it is a pity that MPs could not find more time (Parliamentary Committee). ‐ USAID needs to carefully choose its partners on SME finance for effectiveness because the public sector has not been good at this (A government body). ‐ Not all participants in the Labor Law discussions were at the appropriate level (A government body). 25 ‐ BEP was very much at the lead of the partnership for para-fiscal reform and catalyzed reform. BEP is the first project that targeted all priorities in the order of our priorities, and in the manner in which we address government (A business association). ‐ There has been little noticeable improvement in the quality of economic and business reporting in the media (Business association, NGO, government body). Interviewees were also asked about how familiar they were with the BEP’s objectives, approach and work plan. The responses are reported in the section below. 4.4.4 Online Survey The online survey asked respondents whether they had heard of USAID and the project. The results indicated that 97.7% had heard of USAID and 43.3% knew of BEP. Online respondents, who are predominantly company managers, graded BEP efforts in gathering support for reforms relatively low, with most opting for a grade of 2, followed by a grade of 3, and about 10-14 % giving a grade of 1 or 4, thus resulting in an average grade of about 2.5. Interestingly, a somewhat higher grade is given to forging partnerships for reform, which contrasts with opinions of stakeholders who highly evaluated outreach (43.48% granted a grade 5 and 34.8% grade 4) but consider that partnerships have not been established in the true sense of the concept (27.27% granted a grade of 5, 45.45% granted a grade of 4, and 27.27% a grade of 3), and this is slightly more present among business and research organization representatives. These could also be considered as more informed responses, considering that all stakeholders had heard of USAID (100%), 96.55% had heard of BEP, and 73.33% had worked with BEP. In addition, a vast majority possessed high expertise in relevant areas. Stakeholder interviewees were additionally asked about how familiar they were with the BEP objectives, approach and work plan. 73.33% of NGO stakeholders and 45.45% of government and international organizations stated that they were “fully familiar,” with the rest being at least “somewhat familiar.” In sum, this is a notable result, considering the wide range of project activities. Almost all responded positively (95.83% in total, with 100% of NGO representatives) to the inquiry as to whether all relevant stakeholders were included in BEP’s consultations and activities, and affirmed that their organization had sufficient opportunity to provide views and inputs affecting the design and implementation of project activities (95.65% said yes). When asked about how they had benefited from their involvement, the following responses were received (see Figure. 4, below): Figure 4: Benefits from Involvement Finally, 64% of stakeholders evaluated their cooperation with BEP with a grade of 5 (76.92% of NGO representatives compared to 50% of government/international organizations representatives), while 28% graded the cooperation with a 4 (23% of NGO representatives), one person graded cooperation as a 2, and one as a 1 (both from the government). This is an extremely satisfactory result. 26 Graphic presentation of the above results is available in an Annex III to this report. 4.4.5 Analysis of Secondary Data (Other Reports) EU SBA Dimensions 1 and 6 of the SME Policy Index Report on Entrepreneurial Learning and Women’s Entrepreneurship and Regulatory Framework respectively, are relevant to the issue of reform groups and partnerships. The Regulatory Framework dimension includes a measure of opportunities for the private sector to comment on draft laws and regulations. The report states that most economies in the Western Balkans treat women’s entrepreneurship as an equity or poverty reduction issue rather than as a lever for increasing competitiveness. Women remain ill-informed about training and initiatives to help access financing. Serbia ranks 4th out of the 8 economies in this area (Croatia is in the lead). In the case of Dimension 6, as reported in a previous section, it ranks second. BEP attended a public media event hosted by six major Serbian clusters. The BEP financial component leader was also present at this meeting and invited the clusters to attend a meeting at the project’s office immediately after to discuss collaboration. Information obtained by the Evaluation Team from the clusters and BEP indicates that the clusters, which represent many enterprises and thousands of employees, welcome the project’s support in helping them negotiate better financing and financial products with the financing institutions. CONCLUSIONS BEP has been very effective in identifying stakeholders and focusing their attention on key issues in reform groups, and there are good examples where they have molded them into effective partnerships for reform. The project has also built support through outreach programs to inform stakeholders and public. On the operational level, BEP has worked very effectively with a number of its grantees to gain deep insight into the nature of problems. It has also provided support to its partners in public bodies through training programs and the Internship Program, which are designed to bolster their capacity. Finally, BEP has paid special focus on building relationships with women’s organizations and ensuring they are represented in its initiatives. A review of the list of project beneficiaries and partners shows that BEP has been successful in identifying all important stakeholders in the public and private sectors, and has integrated them into activities. Where this was not possible (e.g. Competition Commission), the project withdrew from the activity with USAID’s agreement. As many have pointed out, little can be done with reform involving public bodies without commitment from senior officials in GoS. BEP has succeeded in capturing the attention of both governments in place during the evaluation period. The ability to garner support from two administrations has been an important factor in BEP’s progress. It is notable that although the change of government would have been a challenge for the project, BEP managed to maintain the thrust of its activities. An example of its ability to capture the attention of senior politicians is the para-fiscal charge initiative, which had the personal attention and support of the Minister of Finance. Another is the Labor Law reform, which GoS intends to introduce at the year’s end, as announced by the Prime Minister at a BEP-supported conference at the end of 2012. In summary, BEP has succeeded in putting regulatory reform on the political agenda, as well as winning the support of senior officials in its activities, which include many activities that are part of the acquis adoption process. The project has succeeded in bringing together all players affected by the reform or development initiative. In doing so it has employed a number of techniques, ranging from conferences, to WGs, to retreats. Interviewed stakeholders have consistently remarked that the project’s ability to do this has been one of its main strengths, together with its flexibility in adapting itself to the situation and to stakeholders’ needs. The project is perceived as being objective and, as one interviewee remarked, never alienated a single stakeholder. One example that was given by many stakeholders was BEP’s ability to bring together the social partners to discuss labor reform, which has never been achieved by any other project. BEP faces the difficult task of facilitating the progress of WGs involving public and private sector actors, who may be at different levels of motivation and may work at different paces. BEP’s success depends on the issue being tackled and the diversity of the group and its members’ interests. Thus, the effectiveness of partnerships will vary and this was also noted in the key informant interviews that the Evaluation Team conducted with the 27 stakeholders. While all applauded the project’s efforts in bringing reform groups together, not all partnerships were seen as strong. Because of this, the Labor Law initiative, involving social partners, has moved forward more slowly than some other initiatives, such as BEP’s collaboration with the PDA and Fiscal Council. MoFE and RRO focus on areas such as public sector financial management reform and RIA, which mainly involve the interaction of BEP and public bodies. However, in the case of inspection reform, the slow-progress is attributed to the MoJPA being preoccupied with the wholesale reform of the public administration. The survey of stakeholders’ familiarity with BEP’s objectives, approach, and work plan indicates that there is still room for improvement in informing them and this could be an important element of partnership-building. New partnerships are being created and they will bring new challenges. For example, while the business associations are much aligned with BEP on the regulatory issues, there is less interest from players, such as Foreign Investors Council, on SME finance because of the nature of the companies that are represented. Thus, BEP is in the process of targeting new reform groups and forming new partnerships (e.g. involvement of new groups representing SMEs for developing and advancing the access to finance initiative). Another example is the Commodity Exchange, where a way ahead has yet to be established. The Evaluation Team notes that BEP is making efforts to establish links and initiate discussions with new groups such as business clusters. The Team attended an event hosted by 6 top business clusters at which BEP made contact and invited them to discussions for collaboration on SME finance. The grantees group is mentioned separately because it has been an integral part of BEP’s approach to forming partnerships. The grant program has been used very effectively and it has provided the underlying research and analysis that laid the foundations for BEP’s activities. There are several notable examples including studies on para-fiscal charges, the shadow economy, and construction permits, among others. These particular grants have been an excellent application of project resources. BEP has worked closely with the printed media and has sought to enhance the quality of information provided through trainings. Journalists have given the project good coverage while quarterly reports produce numerous references to reports and articles. However, stakeholders have not noticed a significant improvement in the quality of information. Moreover, stakeholders agree that more specific, higher-quality reporting is important to allow the public to discern the government’s actions and judge its efficiency in carrying out its duties. The project has also used websites (BEP, Policy Café) and its partners have used social media (Konkretno), although this last one has had limited penetration because it is for the more computer savvy individuals. Nevertheless, the BEP Business Survey 2012 noted an improvement in business awareness of GoS’ Economic Growth Strategy, although this could be attributed to a combination of sources of information from political campaigning, to events, to printed and social media. BEP provided training to strengthen the capacity of public bodies and to transfer know-how to stakeholders. For example, trainings to the RRO and ministries have played an important role in improving regulatory reform or introducing program budgeting. In this respect, the project has worked closely with SUK, grantee training providers, as well as short-term experts. An overview of the trainings shows a high average training evaluations score of 4.27 out of 5. The trainings have been attended by 1,246 participants. The Evaluation Team did not establish the impact of these trainings on civil servant performance in the workplace since SUK leaves collecting such information to the line ministries. SUK does not involve itself in trainings that are not horizontal (i.e. across government bodies), and is not involved in the Internship Program provided by GoS. SUK is currently awaiting the outcome of the public administration reform, as a result of which a new training institute may be created to which SUK would transfer the operational responsibility for trainings. The GoS Internship Program was launched this year and is reportedly successful, with interviewees from host public bodies remarking on how pleased they were with the knowledge, attitude, and quality of the outputs provided by the interns. The training is conducted in conjunction with BOS, a grantee. The Evaluation Team requested that BEP provide gender information. Although this information was provided, it was not included in the usual project reports. The information shows that the project has taken sufficient measure to ensure representation of both genders in its activities. 28 On balance, the Evaluation Team concluded that BEP has achieved significant success in engaging with public and private sector key stakeholders and in creating reform groups, which have been consistently involved in the reform processes and project activities. However, BEP has achieved only moderate success in catalyzing reform, given that many issues remain as work-in-progress and many are dependent on political will. RECOMMENDATIONS The recommendation to focus on working partnership is linked to those in the above sections, which recommend downsizing or withdrawing from initiatives that do not produce results despite significant effort and resources. Part of the selection or de-selection of activities involves assessing the effectiveness of partnerships and whether they can work. The project should focus its efforts on those partnerships that are functioning well, e.g. NALED’s efforts to remove administrative barriers.  Extend partnerships to include a wider base of stakeholders representing small businesses and exporters. In the case of SME finance, it will be necessary to ensure wider representation of the interests of small businesses given that some of the business associations represent larger companies that do not experience access to finance problems in the same way. The Evaluation Team notes that the project is already taking such measures. Attention should be paid to the development of exporters, given their importance to the development of the Serbian economy. One way of channeling this support would be through the existing, although relatively passive, Serbian Exporters Association.  BEP should assess what else could be done to ensure that all stakeholders are sufficiently familiar with the objectives. The Evaluation Team noted that the project has held bi-annual meetings and maintained regular contact with the staff of beneficiary institutions and organizations; however, the need to improve communications with them should be assessed.  BEP should continue to improve the quality of economic journalism and develop new initiatives. One possibility could be a competition, with a prize for journalism related to the benefits of specific reforms e.g. Labor reform.  Develop the GoS Internship Program and look for opportunities to cross-fertilization of experiences to SUK. Although the topic of capacity building for GoS ministries and bodies is a sub￾activity connected with specific activities under the three components, it is worth looking into how the project’s experiences in training and internship could be used to help SUK achieve the objective of improving the quality of HR in public bodies. The GoS Internship Program should be continued and developed.  Grants should be continued to provide essential inputs for BEP activities. As described above, grants appear to have been used effectively and consideration should be given to bolstering grant funding through transfer of resources from discontinued or downsized activities. The focus should be on outreach, partnership building, and the provision of essential inputs for BEP’s activities. Needless to say, the use of grants should be supported by an effective grant monitoring system.  Provide advocacy training to less experienced partners. While some of the business associations have developed methods of advocacy, and dedicated resources (e.g. Foreign Investors Council) have a well-defined advocacy approach, the smaller associations have less experience. Given the importance of business associations in the application of a two-pincer approach to catalyze reform, e.g. advocacy from business to politicians resulting in motivation and pressure on the public bodies to produce results, it is essential that these skills are transferred. Although advocacy training has not been part of the project’s activities other than through demonstrations during joint activities with BEP, the needs for this should be assessed and an approach commensurate with available resources should be designed and developed. Note: The Evaluation Team saw that this need has been identified by BEP and this recommendation supports that intention. 29 ANNEXES 30 ANNEX I: EVALUATION STATEMENT OF WORK 31 SECTION C – DESCRIPTION / SPECIFICATIONS/STATEMENT OF WORK C.1 SUMMARY The United States Agency for International Development (USAID) in Serbia seeks the services of one Contractor to the USAID-funded Business Enabling Project during the period of June 2013. C.2 BACKGROUND USAID has supported economic growth related programs since opening its Serbia and Montenegro Mission in 2001. In 2011, USAID started implementation of an amended Strategic Plan for Serbia, which covered the period FY 2011- FY 2015. Under this strategy, the Mission will continue to engage in areas of past investment, including economic growth, to help ensure sustainability of reforms and progress in these areas. USAID/Serbia’s overarching goal for the 2011-2015 strategy is “Supporting Serbia in its vision to be democratic, prosperous and moving toward Euro-Atlantic Integration.” In support of this goal, USAID has set two primary objectives, one of which is a More Competitive Market Economy. In compliance with the Automated Directives Systems (ADS) 203.3.2 and the Agency’s New Evaluation Policy, USAID/Serbia will conduct a number of performance evaluations to determine the real impact of the programs implemented in the final phase of its presence in the country. C.3 PURPOSE OF EVALUATION The purpose of the performance evaluation subject to this Statement of Work (SOW) is to provide USAID with a measurement of change in development outcomes attributable to the defined interventions in the project mentioned above, based on models of a broad range of qualitative, quantitative, or mixed methods. This is an umbrella SOW that covers two separate performance evaluations and provides overall guidance for the potential implementer. More specific statements of work providing the details about methodology, evaluation teams’ structure, schedule and logistics, and deliverables are provided for each individual evaluation in the following sections. C.4 PERFORMANCE EVALUATION STATEMENT OF WORK - BUSINESS ENABLING PROJECT I. PURPOSE (BEP) USAID/Serbia is planning to conduct a mid-term performance evaluation of the USAID Business Enabling Project (BEP), contract number 169-C-00-11-00001, implemented by Cardno Emerging Markets. BEP is a five-year, $15.57 million initiative launched in January 2011. The USAID Contracting Officer's Representative (COR) for this contract is Aleksandar Djureinovic. The Project helps the Government of Serbia (GOS) to increase the competitiveness of the Serbian economy and its private sector businesses by streamlining the business enabling environment, improving public financial management, and strengthening financial markets. Project activities are based on priorities identified by the private sector and the government. 32 II. BACKGROUND (BEP) BEP was designed in 2010 to provide continuation of assistance to Serbia in improving its business enabling environment. The Project is the result of a thorough and lengthy planning and analysis process, which culminated with the assessment of Serbia’s assistance needs in the area of economic policy for the period 2011 – 2015 and recommendations for a new activity design to support improvements in the business enabling environment and Serbia’s path to European Union (EU) membership. This report entitled “Private Sector Development and Business Enabling Environment, June 2009” responded to a need to: analyze USAID/Serbia’s achievements, identify specific assistance gaps, account for the activities and planned results of other donors, and determine the most effective possible USAID assistance interventions for the future strategy period. With this in mind, USAID launched BEP, as an integral part of USG efforts to achieve its goals by assisting the GOS to improve policies and capacity related to the business enabling environment and the macroeconomic environment, resulting in increased interest of investors and growth and expansion of business. The long-term impact of the project will contribute to making the market economy of Serbia more competitive with other regional, European, and global economies. BEP consists of three components: Component 1: Business Regulation and Economic Governance, streamlines laws, regulations, and institutions; improves regulators’ capacity to implement laws; and improves analysis and public-private dialogue. Component 2: Macroeconomic Policy and Public Financial Management, helps the Government implement new budget and fiscal responsibility laws, improves budgeting, increases public disclosure and transparency, strengthens public debt management, and improves economic analysis and forecasting. Component 3: Financial Market Development, improves regulatory frameworks for non-bank institutions, develops capacity to strengthen and expand capital markets, and improves conditions for debt market development. Relationship to the Mission Strategy The Project’s strategy, Work Plan and Performance Management Plan (PMP) are consistent with USAID Serbia’s Amended Country Strategy FY 2011 – 2015; and activities support IR 1.1 Business Enabling Environment Improved, and IR 1.4 Macroeconomic Environment Improved. The project improves the ability of all businesses in Serbia to compete in global, EU, and domestic markets. BEP’s PMP captures the impacts that the project will have, including the following Mission Indicators (with source): Decrease in the “Burden of Government Regulations” (World Economic Forum Global Competitiveness Indicators); Decrease in costs for businesses to comply with the regulatory framework (Standard Cost Model); Increase in foreign direct investment (NBS Statistics); Improvement in the Open Budget Index (International Budget Partnership); Improved Government budget balance as percentage of GDP (Ministry of Finance report); Improved “Financial Market Development” (World Economic Forum Global Competitiveness Indicators); and Credit to private sector as a percentage of GDP (Statistics from the National Bank of Serbia). III. OBJECTIVE (BEP) The evaluation will provide rigorous, evidence-based and independent analysis on BEP’s performance at the mid￾term of the project. Specifically, the objective of this mid-term evaluation is to conduct a full, evidence-based and independent review of BEP’s activities and results from January 2011 to the present. 33 The purpose of the evaluation is to examine the effectiveness of project activities implemented to date; to determine whether the project has achieved planned results; to identify gaps in performance against targets; and to provide recommendations on closing these gaps in the final years of the project. It is expected that the contractor will not simply provide an accounting of performance against targets but provide an independent analysis on why targets were realized, met, or significantly surpassed. Of particular interest to the Mission would be an analysis of the reasons behind any gaps between expected and actual performance, including an identification of the likely source(s) of these gaps (design, project management, changes in operating environment, success factors, etc.). This analysis will inform future decisions regarding the project's focus by the Mission, particularly by the Economic Growth Office. IV. DETAILED STATEMENT OF WORK (BEP) Evaluation Questions In examining BEP, the contractor shall provide detailed answers to the following questions: 1. How successful was the project in reducing the regulatory burdens to businesses in Serbia? 2. How successful was the project in increasing the Government of Serbia’s capacity and transparency in public financial management? 3. How successful was the project in creating conditions for increased finance for the private sector? 4. How successfully did the project gather the support of relevant groups for reform? 5. To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? The final report will summarize the findings from interviews, discussions, and evaluations of relevant BEP reports, studies, and assessments. Pitfalls and gaps, if any, should be justified and addressed by recommendations. The evaluators are encouraged to be as specific as possible in its recommendations, so as to best inform our Mission for the sake of prompt reaction if needed. Recommendations must be action-oriented, practical and specific, with defined responsibility for the action. Methodology The evaluation team shall start its work with a paper review of all the documents cited in the “Sources" section below. The evaluation team will first complete a desk study that will be used to establish an understanding of BEP’s activities and environment before arrival in Serbia. Based on this desk study, the team shall prepare a work plan that will be sent to the Mission prior to arrival in-country. The evaluation team will then present this work plan at the in￾brief meeting at the Mission upon arrival. The workplan shall include a design matrix that demonstrates how the team plans to answer each evaluation question (data collection methods, sources, methods of analysis, limitations, etc). The methodology will include a mix of tools appropriate to evaluation’s research questions. Key informant interviews are suggested as a secondary data source for this evaluation. In addition to these approaches, the Mission is looking for additional suggestions for assessing the performance of the contractor. The contractor will provide a detailed explanation of the proposed methodology for carrying out the work. The methodology will be comprised of a mix of tools appropriate to the evaluation’s research questions. These tools may include a combination of the following: Review BEP documentation (e.g., SOW, approved work plans, and quarterly reports); Review of the business enabling environment in Serbia and its constraints; Organize focus group discussions with project beneficiaries; Conduct stakeholder interviews (business associations, GOS representatives); Identify case studies of successful initiatives. USAID reserves the right to approve/suggest additional questions. The evaluation shall be backed up by the relevant data and information gathered from meetings with program partners and all the relevant recommendations and data available. 34 Sources Potential sources the evaluation team may reference or contact to prepare the deliverables include: § All relevant project reports, and annual work plans; § The project’s studies and assessments; § The project’s performance management plan (PMP); § USAID/Serbia’s new 2011-2015 Amended Strategy; § USAID/Serbia’s Mission Director, Economic Growth Office Director, and the project’s COR; § Appropriate Government of Serbia department beneficiaries; § Other donors and international institutions, including the EU, International Monetary Fund, and World Bank; § Selected business associations. To support the team’s initial literature review, USAID/Serbia will provide electronic copies to the contractor of all documents to be reviewed (listed above) in preparation for the desk study. USAID prefers to share these documents via Google drive. Deliverables: The evaluation team will produce the following deliverables: a. Work Plan and Design: A draft Work Plan and Evaluation Design shall be completed by the lead evaluator before departing the US and presented to the COR. The evaluation design will include a detailed evaluation design matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, and a dissemination plan. The final design requires COR approval. Unless exempted from doing so by the COR, the design will be shared with country-level stakeholders as well as with the implementing partners for comment before being finalized. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. The work plan shall at least include the following: § The overall methodology for carrying out the evaluation; § Design matrix broken down by question; § A draft schedule of targeted meetings and list of potential interviewees; § Data collection and analysis plan; and § Identification of potential major constraints. b. Draft Evaluation Report: A draft report of the findings and recommendations shall be submitted to the USAID COR prior to the team’s debrief with the Mission. The written report must clearly describe findings, conclusions, and recommendations. USAID will provide comments on the draft report within two weeks of submission. c. PowerPoint Presentation and Debriefing with USAID: The team will present the major findings of the evaluation to USAID/Serbia through a PowerPoint presentation after submission of the draft report and before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project approaches, results, or activities. The team will consider USAID comments and revise the draft report accordingly, as appropriate, based on the content of the Debriefing. d. Final Report: The team will submit a final report that incorporates the team's responses to Mission comments and suggestions no later than 15 days after USAID/Serbia provides written comments on the team’s draft evaluation report (see above). The report will be submitted in English, electronically. The report will be disseminated internally by USAID for comments and shared with the implementing partner being evaluated. 35 Agency criteria to ensure quality in the evaluation final report are presented in Attachment I. The contractor shall use all of these criteria in the preparation of the final report. The evaluation final report must include: Introduction and executive summary (2-5 pages in length summarizing the purpose and background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned, as applicable); Background (the local context and the activities being evaluated, the main evaluation questions); Methodology and the limitations to the evaluation (explained in detail, with particular attention to the limitations associated with the evaluation methodology - e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.); Findings and conclusions; and Recommendations and lessons learned (if applicable). Report shall not exceed 25 pages (not counting annexes). The annexes to the report shall include: The Evaluation Scope of Work; Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team; All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; Sources of information, properly identified and listed; and Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Other requirements: a. Weekly Meetings: The evaluation team will be in regular contact with the Mission’s evaluation COR throughout the period of performance. In addition, the team will be required to schedule formal weekly meetings with the Mission’s evaluation COR and Economic Growth Office staff to update them on the progress of the evaluation and to ask any clarifying questions on the project as the evaluation progresses. These meetings can be held by phone, if agreed upon by the Mission. b. All the records from the evaluation (e.g. interview transcripts or summaries) must be provided to the evaluation COR. All quantitative data collected by the evaluation team must be provided in an electronic file in easily readable format agreed upon with the COR. The data must be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID/Serbia & Montenegro will retain ownership of the survey and all datasets developed. The evaluation report and summaries shall be submitted to the Development Experience Clearinghouse (DEC) within three months of completion by the COR at http://dec.usaid.gov. Level of Effort and Timeline Task / Deliverables Duration / LOE (Days) Team Lead Technical Specialist Translator / Admin Assistant Preparation and Research (desk study prior to arrival in country, includes draft of the work plan) 5 5 Travel to Serbia 2 2 36 Team planning, work plan review, and meeting with USAID in Belgrade 1 1 1 Evaluation of BEP 12 12 5 Discussion, analysis, draft evaluation report in country, debrief meetings with USAID, and preliminary draft report presentation 2 2 1 Departure from Serbia 1 1 Team revises draft report and submits final to USAID (out of country) 10 10 Totals 33 33 7 V. EVALUATION TEAM STRUCTURE/COMPOSITION (BEP) The evaluation team shall consist of three members: two technical specialists and interpreter/administrative assistant. One of the two team members will be designated Team Lead. Although experience in conducting evaluations would be valuable for both analysts, extensive and documented experience in conducting performance evaluations is highly preferred for the Team Lead. The proposed Evaluation team must be familiar with USAID’s Evaluation Policy: (http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf) Evaluation Team Member/Economic and Trade Analyst (Senior Level) S/he will serve as the expert on business environment and reform. Must have significant business environment and reform-related experience; Must meet minimum level of academic and the work experience qualifications for Senior Level; Experience and knowledge related to similar reforms in countries in transition highly preferred. Evaluation Team Member/Program and Policy Analyst (Mid Level) S/he will serve as the expert on economic growth-related policy reform. Must have excellent knowledge regarding economic growth-related policy reform challenges in countries in transition, along with familiarity with transitional economies Must meet minimum level of academic and the work experience qualifications for Mid Level; Documented experience in conducting performance evaluations highly preferred. Interpreter/Administrative Assistant The Administrative Assistant will provide logistical, administrative, and clerical and translation support to the team throughout the evaluation. Must have experience in translating technical language related to economics and business. STATEMENT OF WORK – ATTACHMENT 1 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why. Evaluation reports shall address all evaluation questions included in the scope of work. The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. Evaluation findings will assess outcomes and impact on males and females. 37 Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. Sources of information need to be properly identified and listed in an annex. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. [END STATEMENT OF WORK – ATTACHMENT 1] 38 STATEMENT OF WORK – ATTACHMENT 2 SUGGESTED OUTLINE FOR AN EVALUATION REPORT Element Approximate Number of Pages Description and Tips for the Evaluation Team Title Page 1 (but no page number) Essential. Should include the words “U.S. Agency for International Development” with the acronym “USAID,” the USAID logo, and the project/contract number under which the evaluation was conducted. See USAID Branding and Marking Guidelines (http://www.usaid.gov/branding/) for logo and other specifics. Give the title of the evaluation; the name of the USAID office receiving the evaluation; the name(s), title(s), and organizational affiliation(s) of the author(s); and the date of the report. Contents As needed, and start with Roman numeral ii. Essential. Should list all the sections that follow, Including Annexes. For multi-page chapters, include chapter headings and first- and second-level headings. List (with page numbers) all figures, tables, boxes, and Other titled graphics. Foreword 1 Optional. An introductory note written by someone other than the author(s), if needed. For example, it might mention that this evaluation is one in a series of evaluations or special studies being sponsored by USAID. Acknowledge ments 1 Optional. The authors thank the various people who provided support during the evaluation. Preface 1 Optional. Introductory or incidental notes by the authors, but not material essential to understanding the text. Acknowledgements could be included here if desired. Executive 2-5 Essential Glossary 1 Optional. Is useful if the report uses technical or project-specific terminology that would be unfamiliar to some readers. Acronyms and Abbreviations 1 Essential, if they are used in the report. Include only those acronyms that are actually used. I. Introduction up to 5 pages, starting with Arabic numeral 1. Optional. The two sections listed under Introduction here could be separate, stand-alone chapters. If so, a separate introduction may not be needed. Description of the Project Essential. Describe the context in which the USAID project took place— e.g., relevant history, demography, political situation, etc. Describe the specific development problem that prompted USAID to implement the project, the theory underlying the project, and details of project implementation to date. The Evaluation Purpose and Methodology Essential. Describe who commissioned the evaluation, why they commissioned it, what information they want, and how they intend to use the information (and refer to the Annex that includes the Statement of Work). Provide the specific evaluation questions, and briefly describe the evaluation design and the analytical and data collection methods used to answer them. Describe the evaluation team (i.e., names, qualifications, and roles), what the team did (e.g., reviewed relevant documents, analyzed secondary data, interviewed key informants, conducted a survey, conducted site visits), and when and where they did it. Describe the major limitations encountered in data collection and analysis that have implications for reviewing the results of the evaluation. Finally, refer to the Annex that provides a fuller description of all of the above, including a list of documents/data sets reviewed, a list of individuals interviewed, copies of the data collection instruments used, and descriptions of sampling procedures (if any) and data analysis procedures. II. Findings, Conclusions, and Recommendations 20 Essential. Annexes Statement of Work Some are essential and Some are some are optional as noted Essential. Lets the reader see exactly what USAID initially expected in the evaluation Evaluation Design and Methodology Essential. Provides a more complete description of the evaluation questions, design, and methods used. Also includes copies of data collection instruments (e.g., interview guides, survey instruments, etc.) And describes the sampling and analysis procedures that were used. Interviewed List of Persons Essential. List of Documents Reviewed Essential. Includes written and electronic documents reviewed, background literature, secondary data sources, citations of websites consulted. Dissenting Views Reviewed If needed. Include if a team member or a major stakeholder does not agree with one or more findings, conclusions, or recommendations. [END OF ATTACHMENT 2] [END OF SECTION C] ANNEX II: WORK PLAN AND EVALUATION RESEARCH DESIGN WORK PLAN & EVALUATION RESEARCH DESIGN In response to: Mid-Term Performance Evaluation of the USAID/Business Enabling Project (BEP) June 21, 2013 Submitted by: Mendez England & Associates 4300 Montgomery Ave., Suite 103 Bethesda, MD 20814-4413 Contact Person: Thomas C. England, President Tel: (301) 652-4334  Fax: (301) 652-3733  Email: tengland@engl.com Table of Contents 1. Introduction 2. Evaluation Team 3. Evaluation Tasks and Sub-tasks 3.1 Pre-Mobilization Activities 3.2 In-Country Activities 3.3 End-of-Mission Activities 4. Evaluation Design 5. Evaluation Methodology Annex 1 – Statement of Work Annex 2 – Mission Schedule Annex 3 – List of Stakeholders for Interview Annex 4 – Key Evaluation Questions and Proposed Interview Framework Annex 5 – Evaluation Design & Methodology Matrix Annex 6 – BEP Overall Monitoring and Evaluation Matrix Annex 7 – Draft Outline of Final Evaluation Report (Table of Contents) 1. INTRODUCTION USAID/Serbia is planning to conduct a mid-term performance evaluation of the USAID Business Enabling Project (BEP), contract number 169-C-00-11-00001, implemented by Cardno Emerging Markets. BEP is a five-year, $15.57 million initiative launched in January 2011. The purpose of the performance evaluation is to provide USAID with a measurement of change in development outcomes attributable to the defined interventions in the project mentioned above, based on models of a broad range of qualitative, quantitative, or mixed methods. The BEP evaluation will cover the period January 2011 - June 2013. In the preparation of this Work Plan and Evaluation Research Design, the Evaluation Team has followed the guidelines outlined in USAID’s Request for Task Order Proposal (RFTOP) and accompanying Scope of Work (SOW) (Annex 1), together with ME&A’s Technical Proposal. The evaluation of BEP will provide USAID with a rigorous, evidence-based analysis of BEP’s mid-term performance. Specifically, the evaluation will: 1) examine the effectiveness of BEP’s activities implemented to date; 2) determine whether BEP has achieved planned results; 3) identify gaps in BEP’s performance against targets; and 4) provide recommendations on closing these gaps in the final years of the project. In pursuit of these findings, the Evaluation Team is further tasked with answering a specific set of evaluation questions posed by USAID/Serbia. These questions include: 6. How successful was the project in reducing the regulatory burdens to businesses in Serbia? 7. How successful was the project in increasing the Government of Serbia’s capacity and transparency in public financial management? 8. How successful was the project in creating conditions for increased finance for the private sector? 9. How successfully did the project gather the support of relevant groups for reform? 10. To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? (also listed in Annex 4 and broken down in proposed questionnaire in the same Annex). The conclusions reached and suggested recommendations by the Evaluation Team will assist USAID/Serbia to design future interventions in the field of business enabling. Below, we present the Evaluation Team. Annexes (1 to 7) include Statement of Work, a revised Mission Schedule, List of Stakeholders to Interview, Key Evaluation Questions and Proposed Interview Framework, Evaluation Design & Methodology Matrix, BEP Overall Monitoring & Evaluation Matrix, and Draft Outline of the Final Evaluation Report (Table of Contents). 2. EVALUATION TEAM The evaluation of BEP will be conducted by a team of two experts: international expert Mr. Andrzej Schafernaker (Team Leader) and local expert Dr Ana Trbovich (Business Enabling Environment Specialist). In addition, the team will be assisted by Ms. Dragica Todorovic, who will serve as interpreter and logistics specialist. Mr. Andrzej Schafernaker will assume responsibility for the overall management of the evaluation in collaboration with USAID/Serbia. In short, this encompasses all activities specified in the SOW related to the evaluation mission, including pre-mobilization, on￾site implementation and end-of-assignment deliverables. Dr Ana Trbovich will contribute to the evaluation mission by conducting stakeholder interviews, including interviews and focus group sessions (where deemed appropriate), as well as contributing towards the preparation of the work plan, draft and final evaluation reports. Finally, oversight of the evaluation mission will fall under the remit of Ms Mirela McDonald, Evaluation IQC Manager with ME&A and Ms Audra Stark, Project Manager with ME&A. In preparing this Work Plan recognition has been given to the number of days allocated for the in-country part of the assignment as this is clearly going to impact on the nature and extent of the contributions made by both members of the Evaluation Team. For this reason, the Team Leader and the local expert will conduct separate field visits as highlighted in the evaluation schedule. In adopting this strategy, the Evaluation Team will maximize the number of interviews and will enhance the veracity of its overall findings. 3. EVALUATION TASKS and SUB-TASKS 3.1 Pre-Mobilization Activities Conference Calls On 5 June, 2013, a conference call took place between Ms. Mirela McDonald and Ms. Ana Trbovich (on behalf of ME&A team), and Ms. Alison Case (COTR) and Mr. Miodrag Bogdanovic (M&E Specialist) from USAID/Serbia. A number of issues were discussed including the form of the interview contact list and timing for submission of interview questions and draft work plan. It was agreed that any outstanding issues would further be discussed at the in-briefing with USAID, scheduled for Monday, 17 June, 2013. Literature Review Both members of the Evaluation Team were provided with a selection of project-related documentation by USAID. These included BEP Work Plans, Annual Reports, Quarterly Reports, PMPs and White Papers among others. The documents provide a useful insight into the planning and operating activities of BEP and will be the source of reference for the Evaluation Team for data and information related to project objectives, indicators and targets. Initial review of the documents was a basis for drafting interview questions and work plan. Preparation of Draft Work Plan and Evaluation Design This activity will be completed prior to beginning fieldwork, in collaboration with the local expert and the ME&A Evaluation Project Manager. Any amendments, corrections or adjustments will be made following the in-briefing with USAID in Serbia. The Evaluation Team will then have 2 days to incorporate any requested amendments before final submission of the Work Plan on June 21. 3.2 In-Country Activities Initial Meetings On 17 June, 2013, a meeting will be held between the Evaluation Team members and USAID/Serbia. This will essentially be an in-briefing for both parties to review the evaluation’s objectives and the progress achieved by desk research and initial interviews. Issues regarding the proposed content of the Final Evaluation Report will be discussed. Data Gathering Meetings with stakeholders will begin as soon as the USAID Mission approves the work plan and the interview list. The meetings will be conducted with pre-identified stakeholders and will continue through 1 July, 2013. The interviewees will include not only BEP’s direct beneficiaries but also a cross-section of other relevant individuals and organizations with an interest in the project’s activities and outcomes – (see Annex 3 for suggested list of interviewees). In some instances, required information will be sourced via the adoption of quantitative research methods (review and analysis of facts that can be statistically verified from recorded data, e.g. BEP records such as PMP’s and Annual Reports). In other cases, qualitative research methods (opinions, observations and explanations gathered from participating interviewees) will be employed to act as a support mechanism, or otherwise. (See Section 5. below for a more in-depth explanation of the Evaluation Team’s planned methodology). Data Analysis This activity involves digesting and analyzing data and information gathered during the in-country mission. Findings will be tabulated both within the body of the final report (see Annex 5 – Evaluation Design & Methodology Matrix), which will summarize the qualitative aspect of the evaluation, and as an attachment to the main report (see Annex 6 – BEP Overall Monitoring & Evaluation Matrix), which will address the quantitative elements of the evaluation. Section 5 below further elaborates on both approaches. On Tuesday, 9 July 2103, the Evaluation Team will have an out-briefing with USAID/Serbia during which initial findings will be presented in summary format together with some of the key issues arising from the evaluation. At this stage, comments and/or suggestions offered by USAID/Serbia will be acknowledged and addressed in the Draft Final Report. 3.3 End-of-Mission Activities Completion and Submission of Draft Report to USAID A Draft Final Report will be prepared and submitted to USAID/Serbia on Saturday, 6 July 2013 for discussion at the out-briefing that will take place on the 9th July after which the Team Leader will leave for home base. Final Report submitted to USAID with integrated comments By Wednesday, 24 July 2013, the Team Leader will receive their comments for integration into the Final Report, which will subsequently be re-submitted to USAID/Serbia on Tuesday, 6 August 2013. 4. EVALUATION DESIGN PLAN In the Technical Proposal that ME&A submitted to USAID/Serbia for the BEP evaluation an illustrative Evaluation Design chart was included (Annex 5) which highlighted the approach and sequence of events that the Evaluation Team would undertake in implementing the assignment i.e. Development of Methodology – Data Collection – Results Analysis and Reporting. This remains our underlying approach for this evaluation. 5. EVALUATION METHODOLOGY The Evaluation Team recognizes that the credibility of an evaluation’s findings, conclusions and recommendations rests for the most part on the quality of the research design as well as data collection methods and analysis used. Given this, we believe that only a combination of both quantitative and qualitative research methods will suffice to provide a clear picture of BEP’s performance. The following approach is how the Evaluation Team envisages tackling the evaluation process. 5.1 Quantitative Research and Analysis Quantitative data (e.g. number of actions / activities to be accomplished against plan) will be sourced from BEP Annual Work Plans and other project-related periodic reports, and cross-referenced with international reports on Serbia’s business-enabling environment published by the European Union (annual progress reports), World Bank (Doing Business), World Economic Forum (Competitiveness Report), OECD (Small Business Act Implementation Reports), etc. In addition, particular reference will be made to the project’s annual Performance Monitoring Plans (PMPs) designed to provide ongoing analysis and commentary on actual results achieved against plan including actions and strategies that were taken to address anticipated deviation from plan. The collective outcome of this evaluation will be a thorough assessment of the performance of BEP in the middle of the project implementation relative to the planned objectives as envisaged in the original SOW and elaborated and/or revised in subsequent BEP Annual Work Plans. The findings here will also be cross-referenced with findings accumulated from our qualitative research approach to determine the extent to which evidence gathered contributes towards the Evaluation Team’s understanding of BEP’s impact on Serbia’s business climate. This in turn will enhance the team’s prospect of being able to provide satisfactory and meaningful answers to USAID’s pre-determined evaluation questions. 5.2 Qualitative Research and Analysis The required approach under this activity assumes particular importance given the nature of the activities (policy reforms are difficult to evaluate in quantitative terms) and time available to the Evaluation Team to conduct its enquiries. Here, the team’s approach will be to identify, locate and meet with as representative a body as possible with direct or indirect knowledge and/or experience of the project (see Annex 4 for a proposed list of potential interviewees). In so doing, the Evaluation Team will pay particular attention to avoiding subjective opinion and hearsay as these effectively add little or no value to understanding the facts. On the other hand, perceptions are valid and will be included in the team’s findings as they may enhance the quality of recommendations for any future planned interventions by USAID/Serbia. Given the short time allocated for the fieldwork and the number of stakeholders that will need to be interviewed, we have developed a standardized questionnaire for interviews (Annex 4). This questionnaire will serve as a guide to key informant interviews to ensure that there is consistency across participants. The proposed interview questionnaire has been structured so that it first assesses what are the most pressing problems in Serbia’s business environment, which will enable the Evaluation Team to evaluate whether the BEP project is involved in the most pertinent policy issues. It then requests an opinion with regard to progress in the issues that BEP is engaged in without mentioning the project to ensure objectivity. Only in the second part of the questionnaire are interviewees asked directly about BEP’s engagement in these areas. Therefore, the last part of the questionnaire therefore pertains only to those familiar with the BEP project. The same interview framework will be used for anonymous, online survey as for direct interviews, but information about the interviewee will be taken in direct and focus group interviews. The anonymous on-line survey will be offered via SurveyMonkey. The team will clearly assure respondents that their responses will be anonymous to encourage frank and open responses. The purpose of the online interview is to cross-reference information obtained in direct interviews, especially with regard to selection of activities (priority problematic areas in Serbia’s business climate) and not to obtain a representative sample that could serve for autonomous research since the time period of the evaluation is very limited. In direct interviews, the aim will be to obtain as many examples and comments as possible. The survey, developed in English, after being approved by USAID will be translated into Serbian language. This information will be analyzed and the results will be tailored to answer the five main evaluation questions outlined in the Scope of Work (SOW), as well as to provide information on cross-cutting issues that must be addressed by the evaluation, such as gender. The evaluation team will also review the current indicators for performance measurement and propose possible changes to the indicators. Data will be collected by using a number of methods including:  A critical desk-top review of materials related to BEP and other relevant projects, as well as any material that will be provided by USAID such as project reports and annual work plans, project performance management plan, performance indicators, etc.  Interviews with USAID/Serbia Mission, particularly the Economic Growth Office staff.  Interviews with BEP’s implementers.  In depth, semi-structured interviews with selected stakeholders such as the Ministry of Finance and Economy, Ministry of Internal and Foreign Trade and Telecommunications, Ministry of Labor, Employment and Social Policy, the Regulatory Reform Office of the Government of Serbia, the Serbian Fiscal Council, the Securities Commission, American Chamber of Commerce in Serbia, etc.  Focus groups: The use of focus groups will provide both quantitative and qualitative data to provide context and background on the data obtained through the individual interviews (focus groups planned with experts from CLDS, experts from NALED, exporting firms members of Exporters Association of Serbia and women-owned businesses who are members of Association of Businesswomen of Serbia).  Interviews with a number of NGOs/research institutions such as the National Alliance for Local Economic Development (NALED), Center for Liberal-Democratic Studies (CLDS), Belgrade Open School, Economics Institute, as well as business associations such as Exporters Association of Serbia, Association of Businesswomen of Serbia, Serbian Association of Managers, and Association of Corporate Directors of Serbia.  Direct observation to cross-check information (e.g. comparing statements to observed practice) and identification of factors not previously recognized.  Review of program outputs against objectives and performance indicators  Interviews with other donors, including European Union, the World Bank, German aid agency GIZ, etc. 11. EVALUATION LIMITATIONS a. Low response The evaluation will be conducted during the holiday season, which carries a risk of low response. To mitigate this risk, the Evaluation team will schedule the interview as soon as the interview list and questionnaire are approved by USAID/Serbia. Furthermore, interviews will be scheduled in parallel, with only select key interviews conducted jointly by the two-member Evaluation team. Finally, the SurveyMonkey online tool, as noted above, will be used with the aim of reaching as many companies and gauge their opinion of any changes in the business climate in Serbia and the BEP Project’s impact in this area. b. Ability to reach all stakeholders The Evaluation team aims to reach stakeholders in government institutions who tend to be at demanding positions and extremely occupied. The team also wishes to interview independent experts and business associations, all in a limited time period. Good time management in interview scheduling and previous professional acquaintance of many stakeholders will ensure that most stakeholders are reached. c. Availability of data BEP project has initiated a number of different activities, such as inspections reform or public administration management trainings, for which it is too early to have data to analyze the impact of activities. d. Pessimism The prevailing pessimism due to difficult economic transition in Serbia, further aggravated by the world economic crisis and recent political sensitivities in the EU accession process, may negatively affect the responses. The Evaluation Team will be aware of this possible bias and ask the interviewees to try to assess the situation as objectively as possible. STATEMENT OF WORK Performance Evaluation of Business Enabling Project 1 SUMMARY The United States Agency for International Development (USAID) in Serbia seeks the services of one Contractor to the USAID-funded Business Enabling Project during the period of June 2013. 2 BACKGROUND USAID has supported economic growth related programs since opening its Serbia and Montenegro Mission in 2001. In 2011, USAID started implementation of an amended Strategic Plan for Serbia, which covered the period FY 2011- FY 2015. Under this strategy, the Mission will continue to engage in areas of past investment, including economic growth, to help ensure sustainability of reforms and progress in these areas. USAID/Serbia’s overarching goal for the 2011-2015 strategy is “Supporting Serbia in its vision to be democratic, prosperous and moving toward Euro-Atlantic Integration.” In support of this goal, USAID has set two primary objectives, one of which is a More Competitive Market Economy. In compliance with the Automated Directives Systems (ADS) 203.3.2 and the Agency’s New Evaluation Policy, USAID/Serbia will conduct a number of performance evaluations to determine the real impact of the programs implemented in the final phase of its presence in the country. 3 PURPOSE OF EVALUATION The purpose of the performance evaluation subject to this Statement of Work (SOW) is to provide USAID with a measurement of change in development outcomes attributable to the defined interventions in the project mentioned above, based on models of a broad range of qualitative, quantitative, or mixed methods. This is an umbrella SOW that covers two separate performance evaluations and provides overall guidance for the potential implementer. More specific statements of work providing the details about methodology, evaluation teams’ structure, schedule and logistics, and deliverables are provided for each individual evaluation in the following sections. 4 PERFORMANCE EVALUATION STATEMENT OF WORK - BUSINESS ENABLING PROJECT I. PURPOSE (BEP) USAID/Serbia is planning to conduct a mid-term performance evaluation of the USAID Business Enabling Project (BEP), contract number 169-C-00-11-00001, implemented by Cardno Emerging Markets. BEP is a five-year, $15.57 million initiative launched in January 2011. The USAID Contracting Officer's Representative (COR) for this contract is Aleksandar Djureinovic. The Project helps the Government of Serbia (GOS) to increase the competitiveness of the Serbian economy and its private sector businesses by streamlining the business enabling environment, improving public financial management, and strengthening financial markets. Project activities are based on priorities identified by the private sector and the government. II. BACKGROUND (BEP) BEP was designed in 2010 to provide continuation of assistance to Serbia in improving its business enabling environment. The Project is the result of a thorough and lengthy planning and analysis process, which culminated with the assessment of Serbia’s assistance needs in the area of economic policy for the period 2011 – 2015 and recommendations for a new activity design to support improvements in the business enabling environment and Serbia’s path to European Union (EU) membership. This report entitled “Private Sector Development and Business Enabling Environment, June 2009” responded to a need to: analyze USAID/Serbia’s achievements, identify specific assistance gaps, account for the activities and planned results of other donors, and determine the most effective possible USAID assistance interventions for the future strategy period. With this in mind, USAID launched BEP, as an integral part of USG efforts to achieve its goals by assisting the GOS to improve policies and capacity related to the business enabling environment and the macroeconomic environment, resulting in increased interest of investors and growth and expansion of business. The long-term impact of the project will contribute to making the market economy of Serbia more competitive with other regional, European, and global economies. BEP consists of three components: Component 1: Business Regulation and Economic Governance, streamlines laws, regulations, and institutions; improves regulators’ capacity to implement laws; and improves analysis and public-private dialogue. Component 2: Macroeconomic Policy and Public Financial Management, helps the Government implement new budget and fiscal responsibility laws, improves budgeting, increases public disclosure and transparency, strengthens public debt management, and improves economic analysis and forecasting. Component 3: Financial Market Development, improves regulatory frameworks for non-bank institutions, develops capacity to strengthen and expand capital markets, and improves conditions for debt market development. Relationship to the Mission Strategy The Project’s strategy, Work Plan and Performance Management Plan (PMP) are consistent with USAID Serbia’s Amended Country Strategy FY 2011 – 2015; and activities support IR 1.1 Business Enabling Environment Improved, and IR 1.4 Macroeconomic Environment Improved. The project improves the ability of all businesses in Serbia to compete in global, EU, and domestic markets. BEP’s PMP captures the impacts that the project will have, including the following Mission Indicators (with source): Decrease in the “Burden of Government Regulations” (World Economic Forum Global Competitiveness Indicators); Decrease in costs for businesses to comply with the regulatory framework (Standard Cost Model); Increase in foreign direct investment (NBS Statistics); Improvement in the Open Budget Index (International Budget Partnership); Improved Government budget balance as percentage of GDP (Ministry of Finance report); Improved “Financial Market Development” (World Economic Forum Global Competitiveness Indicators); and Credit to private sector as a percentage of GDP (Statistics from the National Bank of Serbia). III. OBJECTIVE (BEP) The evaluation will provide rigorous, evidence-based and independent analysis on BEP’s performance at the mid-term of the project. Specifically, the objective of this mid-term evaluation is to conduct a full, evidence-based and independent review of BEP’s activities and results from January 2011 to the present. The purpose of the evaluation is to examine the effectiveness of project activities implemented to date; to determine whether the project has achieved planned results; to identify gaps in performance against targets; and to provide recommendations on closing these gaps in the final years of the project. It is expected that the contractor will not simply provide an accounting of performance against targets but provide an independent analysis on why targets were realized, met, or significantly surpassed. Of particular interest to the Mission would be an analysis of the reasons behind any gaps between expected and actual performance, including an identification of the likely source(s) of these gaps (design, project management, changes in operating environment, success factors, etc.). This analysis will inform future decisions regarding the project's focus by the Mission, particularly by the Economic Growth Office. IV. DETAILED STATEMENT OF WORK (BEP) Evaluation Questions In examining BEP, the contractor shall provide detailed answers to the following questions: 1. How successful was the project in reducing the regulatory burdens to businesses in Serbia? 2. How successful was the project in increasing the Government of Serbia’s capacity and transparency in public financial management? 3. How successful was the project in creating conditions for increased finance for the private sector? 4. How successfully did the project gather the support of relevant groups for reform? 5. To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? The final report will summarize the findings from interviews, discussions, and evaluations of relevant BEP reports, studies, and assessments. Pitfalls and gaps, if any, should be justified and addressed by recommendations. The evaluators are encouraged to be as specific as possible in its recommendations, so as to best inform our Mission for the sake of prompt reaction if needed. Recommendations must be action-oriented, practical and specific, with defined responsibility for the action. Methodology The evaluation team shall start its work with a paper review of all the documents cited in the “Sources" section below. The evaluation team will first complete a desk study that will be used to establish an understanding of BEP’s activities and environment before arrival in Serbia. Based on this desk study, the team shall prepare a work plan that will be sent to the Mission prior to arrival in-country. The evaluation team will then present this work plan at the in-brief meeting at the Mission upon arrival. The workplan shall include a design matrix that demonstrates how the team plans to answer each evaluation question (data There really should be a study to determine the impact of this. collection methods, sources, methods of analysis, limitations, etc). The methodology will include a mix of tools appropriate to evaluation’s research questions. Key informant interviews are suggested as a secondary data source for this evaluation. In addition to these approaches, the Mission is looking for additional suggestions for assessing the performance of the contractor. The contractor will provide a detailed explanation of the proposed methodology for carrying out the work. The methodology will be comprised of a mix of tools appropriate to the evaluation’s research questions. These tools may include a combination of the following: Review BEP documentation (e.g., SOW, approved work plans, and quarterly reports); Review of the business enabling environment in Serbia and its constraints ; Organize focus group discussions with project beneficiaries; Conduct stakeholder interviews (business associations, GOS representatives); Identify case studies of successful initiatives. USAID reserves the right to approve/suggest additional questions. The evaluation shall be backed up by the relevant data and information gathered from meetings with program partners and all the relevant recommendations and data available. Sources Potential sources the evaluation team may reference or contact to prepare the deliverables include: All relevant project reports, and annual work plans; The project’s studies and assessments; The project’s performance management plan (PMP); USAID/Serbia’s new 2011-2015 Amended Strategy; USAID/Serbia’s Mission Director, Economic Growth Office Director, and the project’s COR; Appropriate Government of Serbia department beneficiaries; Other donors and international institutions, including the EU, International Monetary Fund, and World Bank; Selected business associations. To support the team’s initial literature review, USAID/Serbia will provide electronic copies to the contractor of all documents to be reviewed (listed above) in preparation for the desk study. USAID prefers to share these documents via Google drive. Deliverables: The evaluation team will produce the following deliverables: a. Work Plan and Design: A draft Work Plan and Evaluation Design shall be completed by the lead evaluator before departing the US and presented to the COR. The evaluation design will include a detailed evaluation design matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, known limitations to the evaluation design, and a dissemination plan. The final design requires COR approval. Unless exempted from doing so by the COR, the design will be shared with country-level stakeholders as well as with the implementing partners for comment before being finalized. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. The work plan shall at least include the following: The overall methodology for carrying out the evaluation; Design matrix broken down by question; A draft schedule of targeted meetings and list of potential interviewees; Data collection and analysis plan; and Identification of potential major constraints. b. Draft Evaluation Report: A draft report of the findings and recommendations shall be submitted to the USAID COR prior to the team’s debrief with the Mission. The written report must clearly describe findings, conclusions, and recommendations. USAID will provide comments on the draft report within two weeks of submission. c. PowerPoint Presentation and Debriefing with USAID: The team will present the major findings of the evaluation to USAID/Serbia through a PowerPoint presentation after submission of the draft report and before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project approaches, results, or activities. The team will consider USAID comments and revise the draft report accordingly, as appropriate, based on the content of the Debriefing. d. Final Report: The team will submit a final report that incorporates the team's responses to Mission comments and suggestions no later than 15 days after USAID/Serbia provides written comments on the team’s draft evaluation report (see above). The report will be submitted in English, electronically. The report will be disseminated internally by USAID for comments and shared with the implementing partner being evaluated. Agency criteria to ensure quality in the evaluation final report are presented in Attachment I. The contractor shall use all of these criteria in the preparation of the final report. The evaluation final report must include: Introduction and executive summary (2-5 pages in length summarizing the purpose and background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned, as applicable); Background (the local context and the activities being evaluated, the main evaluation questions); Methodology and the limitations to the evaluation (explained in detail, with particular attention to the limitations associated with the evaluation methodology - e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.); Findings and conclusions; and Recommendations and lessons learned (if applicable). Report shall not exceed 25 pages (not counting annexes). The annexes to the report shall include: The Evaluation Scope of Work; Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team; All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; Sources of information, properly identified and listed; and Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Other requirements: a. Weekly Meetings: The evaluation team will be in regular contact with the Mission’s evaluation COR throughout the period of performance. In addition, the team will be required to schedule formal weekly meetings with the Mission’s evaluation COR and Economic Growth Office staff to update them on the progress of the evaluation and to ask any clarifying questions on the project as the evaluation progresses. These meetings can be held by phone, if agreed upon by the Mission. b. All the records from the evaluation (e.g. interview transcripts or summaries) must be provided to the evaluation COR. All quantitative data collected by the evaluation team must be provided in an electronic file in easily readable format agreed upon with the COR. The data must be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID/Serbia & Montenegro will retain ownership of the survey and all datasets developed. The evaluation report and summaries shall be submitted to the Development Experience Clearinghouse (DEC) within three months of completion by the COR at http://dec.usaid.gov. Level of Effort and Timeline Task / Deliverables Duration / LOE (Days) Team Lead Technical Specialist Translator / Admin Assistant Preparation and Research (desk study prior to arrival in country, includes draft of the work plan) 5 5 Travel to Serbia 2 2 Team planning, work plan review, and meeting with USAID in Belgrade 1 1 1 Evaluation of BEP 12 12 5 Discussion, analysis, draft evaluation report in country, debrief meetings with USAID, and preliminary draft report presentation 2 2 1 Departure from Serbia 1 1 Team revises draft report and submits final to USAID (out of country) 10 10 Totals 33 33 7 V. EVALUATION TEAM STRUCTURE/COMPOSITION (BEP) The evaluation team shall consist of three members: two technical specialists and interpreter/administrative assistant. One of the two team members will be designated Team Lead. Although experience in conducting evaluations would be valuable for both analysts, extensive and documented experience in conducting performance evaluations is highly preferred for the Team Lead. The proposed Evaluation team must be familiar with USAID’s Evaluation Policy: (http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf) Evaluation Team Member/Economic and Trade Analyst (Senior Level) S/he will serve as the expert on business environment and reform. Must have significant business environment and reform-related experience; Must meet minimum level of academic and the work experience qualifications for Senior Level; Experience and knowledge related to similar reforms in countries in transition highly preferred. Evaluation Team Member/Program and Policy Analyst (Mid Level) S/he will serve as the expert on economic growth-related policy reform. Must have excellent knowledge regarding economic growth-related policy reform challenges in countries in transition, along with familiarity with transitional economies Must meet minimum level of academic and the work experience qualifications for Mid Level; Documented experience in conducting performance evaluations highly preferred. Interpreter/Administrative Assistant The Administrative Assistant will provide logistical, administrative, and clerical and translation support to the team throughout the evaluation. Must have experience in translating technical language related to economics and business. Mission Schedule: ◄ May 2013 ~ June 2013 ~ July 2013 ► Sun Mon Tue Wed Thu Fri Sat 26 27 28 29 30 31 1 2 3 4 5 Phone Call w/USAID/Serbia Discuss early start for interviews to allow for higher participation in holiday season 6 Review Materials Draft Work Plan and Evaluation Methodology Home/Belgrade 7 Review Materials Draft Work Plan and Evaluation Methodology Home/Belgrade 8 9 10 Review Materials Draft Work Plan and Evaluation Methodology Home/Belgrade 11 Review Materials Draft Work Plan and Evaluation Methodology Submit Draft Work Plan and Evaluation Design Home/Belgrade 12 Review Materials Meetings and Interviews with Stakeholders Home/Belgrade 13 Review Materials Meetings and Interviews with Stakeholders Home/Belgrade 14 Meetings and Interviews with Stakeholders Belgrade 15 16 Travel to Serbia 17 In-Briefing with USAID Team Meeting Belgrade 18 Meetings and Interviews with Stakeholders Belgrade 19 Meetings and Interviews with Stakeholders Belgrade 20 Fieldwork Final editing of work plan Belgrade 21 Fieldwork Submit Final Work Plan and Evaluation Design Belgrade 22 Fieldwork Belgrade 23 Belgrade Belgrade 30 24 Fieldwork Belgrade 25 Fieldwork Belgrade 26 Fieldwork Belgrade 27 Fieldwork Belgrade 28 Fieldwork Belgrade 29 Fieldwork Belgrade ◄ June 2013 ~ July 2013 ~ August 2013 ► Sun Mon Tue Wed Thu Fri Sat 30 1 Fieldwork Belgrade 2 Analysis of Findings Draft Evaluation Report Belgrade 3 Analysis of Findings Draft Evaluation Report Belgrade 4 Analysis of Findings Draft Evaluation Report Belgrade 5 Analysis of Findings Draft Evaluation Report Belgrade 6 Analysis of Findings Submit Draft Report Belgrade 7 Belgrade 8 Prepare for Outbriefing Belgrade 9 Out-Briefing with USAID Team Leader Departs 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Receive Comments from USAID 25 Integration of USAID Comments 26 Integration of USAID Comments 27 28 29 Integration of USAID Comments 30 Integration of USAID Comments 31 Integration of USAID Comments 1 2 3 ◄ July 2013 ~ August 2013 ~ September 2013 ► Sun Mon Tue Wed Thu Fri Sat 28 29 30 31 1 2 3 4 5 Integration of USAID Comments 6 Submit Final Report to USAID 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Proposed List of BEP Stakeholders to be Interviewed Government 1. Ministry of Finance and Economy 2. Ministry of Internal and Foreign Trade and Telecommunications 3. Ministry of Agriculture, Forestry and Water Management 4. Ministry of Labor, Employment and Social Policy 5. Ministry of Justice 6. The Regulatory Reform Office of the Government of Serbia (GoS) 7. The Serbian Fiscal Council 8. Central Register for Securities and Clearance House 9. The Securities Commission 10. The National Assembly (Committee on Finance, State Budget and Control of Public Spending) 11. Government of Serbia Human Resources Office (SUK) Associations/NGOs/Research Institutions 1. American Chamber of Commerce in Serbia (AMCHAM) 2. Foreign Investors Council (FIC) 3. Serbian Chamber of Commerce 4. National Alliance for Local Economic Development (NALED) 5. Center for Liberal-Democratic Studies (CLDS) 6. Belgrade Open School (BOS) 7. FREN 8. Economics Institute 9. Exporters Association of Serbia 10. Association of Businesswomen of Serbia (EAS) 11. Serbian Association of Managers (SAM) 12. Association of Corporate Directors of Serbia (UKDS) Donor Organizations 1. European Union 2. The World Bank 3. The International Monetary Fund 4. GIZ *The Evaluation Team will also interview the USAID/Serbia Mission, particularly the Economic Growth Office staff and BEP’s implementers (Cardno Emerging Markets and The Balkan Center for Regulatory Reform - BCRR). Key Evaluation Questions and Proposed Interview Framework: A. KEY EVALUATION QUESTIONS In examining BEP, we shall aim to provide detailed answers to the following questions: 1. How successful was the project in reducing the regulatory burdens to businesses in Serbia? 2. How successful was the project in increasing the Government of Serbia’s capacity and transparency in public financial management? 3. How successful was the project in creating conditions for increased finance for the private sector? 4. How successfully did the project gather the support of relevant groups for reform? 5. To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? These key evaluation questions have been further broken down elaborated in the interview questionnaire, presented below. B.1. INTERVIEW FRAMEWORK – COMPANIES (DIRECT INTERVIEWS) Introduction: The following survey is performed with the purpose of evaluating some of the activities funded by United States Agency for International Development (USAID) with the goal of supporting Serbia’s economic development. We would appreciate if you could take 10-15 minutes to provide your opinion on Serbia’s business climate and relevant issues so that USAID could become more effective in its support to Serbia’s reforms. Organization: Name: Position: Contact Information (email): Gender: M/F 1. Please note whether you are: Company NGO/Academic institution (including business associations) Government institution (even if independent and advisory to executive government) 2. Please note 3 most important problems in Serbia’s business climate: Access to Finance Construction Permitting Consumer Protection Corruption Customs and trade regulations Enforcing Contracts Grey Economy Inspections Efficacy Labor Legislation Limited Investor Protection Limited Market Competition Non-transparent budget Public Enterprises Public Debt Management State Administration Efficacy (Ministries, Local Government) Taxes (complexity) Taxes and contributions (Level of financial burden on companies) Underdeveloped Financial Markets Other (please explain) Please add any other comments here: 3. Do you feel that these problems have improved or become worse in the last two years, on a scale of 1-5: 1 (much worse) 2 (worse) 3 (same – no change) 4 (somewhat improved) 5 (significantly improved) Please add any other comments here: 4. In your opinion, has Serbia’s business enabling environment improved in the last two years (from 2011)? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 5. More specifically, has construction permitting improved? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 6. Have inspections become more efficient? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 7. Has labor-related legislation become more effective? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 8. Has the removal of some non-tax and parafiscal charges improved doing business in Serbia? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 9. In your opinion, has public financial management improved in the last two years? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 10. In your opinion, have financial markets become stronger in the last two years? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 11. Have you heard of USAID before? YES NO 12. Have you heard of USAID Business Enabling Project? YES NO The following questions are to be answered only by those who answered “yes” to question 12: 13. If yes, how successful was the project in reducing the regulatory burdens to businesses in Serbia in your opinion, on a scale of 1-5, with 1 being not successful and 5 extremely successful? 1 2 3 4 5 Please add any other comments here: 14. How successful was the project in increasing the Government of Serbia’s capacity and transparency in public financial management? 1 2 3 4 5 Please add any other comments here: 15. How successful was the project in creating conditions for increased finance for the private sector? 1 2 3 4 5 Please add any other comments here: 16. How successfully did the project gather the support of relevant groups for reform? 1 2 3 4 5 Please add any other comments here: 17. To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? 1 2 3 4 5 Please add any other comments here: 18. How successful has the project been in creating business enabling environment for Public Private Partnerships? 1 2 3 4 5 Please add any other comments here: 19. How successfully has the project supported development of Capital Market in Serbia? 1 2 3 4 5 Please add any other comments here: 20. Have you worked with USAID Business Enabling Project? YES NO 21. If yes, have you participated in trainings provided by USAID Business Enabling Project? If yes, on a scale of 1 to 5, how would you rate the effectiveness of training on business enabling process in Serbia? 1 2 3 4 5 Please add any final comments here: 22. If yes, are you satisfied with your cooperation with USAID Business Enabling Project, on a scale of 1 to 5, with 1 being least and 5 most satisfied? 1 2 3 4 5 Please add any final comments here: THANK YOU FOR YOUR TIME AND EFFORT! B.1. INTERVIEW FRAMEWORK – ORGANIZATIONS (DIRECT INTERVIEWS) Introduction: The following survey is performed with the purpose of evaluating some of the activities funded by United States Agency for International Development (USAID) with the goal of supporting Serbia’s economic development. We would appreciate if you could take 10-15 minutes to provide your opinion on Serbia’s business climate and relevant issues so that USAID could become more effective in its support to Serbia’s reforms. Organization: Name: Position: Contact Information (email): Gender: M/F 1. Please note whether you are: Company NGO/Academic institution (including business associations) Government institution (even if independent and advisory to executive government) 2. Please note 3 most important problems in Serbia’s business climate: Access to Finance Construction Permitting Consumer Protection Corruption Customs and trade regulations Enforcing Contracts Grey Economy Inspections Efficacy Labor Legislation Limited Investor Protection Limited Market Competition Non-transparent budget Public Enterprises Public Debt Management State Administration Efficacy (Ministries, Local Government) Taxes (complexity) Taxes and contributions (Level of financial burden on companies) Underdeveloped Financial Markets Other (please explain) Please add any other comments here: 3. Do you feel that these problems have improved or become worse in the last two years, on a scale of 1-5: 1 (much worse) 2 (worse) 3 (same – no change) 4 (somewhat improved) 5 (significantly improved) Please add any other comments here: 4. In your opinion, has Serbia’s business enabling environment improved in the last two years (from 2011)? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 5. More specifically, has construction permitting improved? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 6. Have inspections become more efficient? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 7. Has labor-related legislation become more effective? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 8. Has the removal of some non-tax and parafiscal charges improved doing business in Serbia? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 9. In your opinion, has public financial management improved in the last two years? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 10. In your opinion, have financial markets become stronger in the last two years? Yes, significantly Yes, somewhat There is no change It has become somewhat worse It has become significantly worse Please add any other comments here: 11. Have you heard of USAID before? YES NO 12. Have you heard of USAID Business Enabling Project? YES NO The following questions are to be answered only by those who answered “yes” to question 12: 13. If yes, how successful was the project in reducing the regulatory burdens to businesses in Serbia in your opinion, on a scale of 1-5, with 1 being not successful and 5 extremely successful? 1 2 3 4 5 Please add any other comments here: 14. How successful was the project in increasing the Government of Serbia’s capacity and transparency in public financial management? 1 2 3 4 5 Please add any other comments here: 15. How successful was the project in creating conditions for increased finance for the private sector? 1 2 3 4 5 Please add any other comments here: 16. How successfully did the project gather the support of relevant groups for reform? 1 2 3 4 5 Please add any other comments here: 17. To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? 1 2 3 4 5 Please add any other comments here: 18. How successful has the project been in creating business enabling environment for Public Private Partnerships? 1 2 3 4 5 Please add any other comments here: 19. How successfully has the project supported development of Capital Market in Serbia? 1 2 3 4 5 Please add any other comments here: 20. Have you worked with USAID Business Enabling Project? YES NO 21. If yes, have you participated in trainings provided by USAID Business Enabling Project? If yes, on a scale of 1 to 5, how would you rate the effectiveness of training on business enabling process in Serbia? 1 2 3 4 5 22. How familiar are you with the BEP objectives, approach & work-plan? Fully familiar Somewhat familiar Not at all familiar 23. How have you individually and your organization been involved in the USAID BEP Project? Problem identification & prioritization Goal setting Solution development Partnership in implementation Progress monitoring & activity adjustment Please add any other comments here: 24. Is BEP is addressing the key issues/problems and give reasons for your answer? Yes No Reasons: 25. Do you think all relevant stakeholders were included in BEP’s consultations and activities and give reasons for your answer? Yes No Reasons: 26. Has your organization been involved in directly seeking and hearing the views and arguments of the business community and other end-beneficiaries of BEP’s activities? Was this done with BEP Involvement? Yes No Comments: 27. Do you feel your organization had sufficient opportunity to provide views and inputs affecting the design and implementation of project activities? Yes No Comments: 28. Which of the BEP activities involving your organization have been the most useful/successful? What has been their output and outcome? What is your view of their long-term impact and sustainability? Know-How Attitude Change Procedural/process improvement Innovation Comments: 29. What is your view of the greatest obstacles faced in implementing the activities and how were they overcome? How efficient were the methods applied? Comments: 30. Are there any outstanding issues or problems connected with this activity and why do they remain? Comments: 31. How do you rate BEP from the point of view of a) effectiveness, 1 2 3 4 5 and b) efficiency in implementing the work-plan and achieving goals? 1 2 3 4 5 32. Are you satisfied with your cooperation with USAID Business Enabling Project, on a scale of 1 to 5, with 1 being least and 5 most satisfied? 1 2 3 4 5 33. Do you have any remarks or ideas about what could be improved? Please add any final comments here: THANK YOU FOR YOUR TIME AND EFFORT! Evaluation Design and Methodology Matrix Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Interview Selection Process and Size of Sample Limitations / Concerns Q.1 How successful was the project in reducing the regulatory burdens to businesses in Serbia? Description – based on content analysis of expert opinions Key informant interviews with key personnel in BEP stakeholder’s organizations. Focus group sessions with selected beneficiaries; Interviewees identified and agreed by Evaluation Team, BEP personnel and USAID Description – based on content analysis of secondary data Review of relevant country reports, studies, white papers etc. Evaluation team will visit all BEP beneficiaries and stakeholders in this subject area. Effects of BEP initiatives not yet registered Description based on content analysis of primary data Review of BEP M & E records and relevant reports Relevant BEP documentation provided to Evaluation Team Low response rate/ under￾reporting/ discrepancies Q.2 How successful was the project in increasing the Government of Serbia’s capacity and transparency in public financial management? Description – based on content analysis of expert opinions Key informant interviews with key personnel in BEP’s stakeholder’s organizations. Focus group sessions with selected beneficiaries Interviewees identified and agreed by Evaluation Team, BEP personnel and USAID Description – based on content analysis of secondary data Review of relevant country reports, studies, white papers etc. Evaluation team will visit all BEP beneficiaries and stakeholders in this subject area. Effects of BEP initiatives not yet registered Description based on content analysis of primary data Review of BEP M & E records and relevant reports Relevant BEP documentation provided to Evaluation Team Low response rate/ under￾reporting/ discrepancies Q.3 How successful was the project in creating conditions for increased finance for the private sector? Description – based on content analysis of expert opinions Key informant interviews with key personnel in BEP’s stakeholder’s organizations. Focus group sessions with selected beneficiaries. Interviewees identified and agreed by Evaluation Team, BEP personnel and USAID Description – based on content analysis of secondary data Review of relevant country reports, studies, white papers etc. Evaluation team will visit all BEP beneficiaries and stakeholders in this subject area. Effects of BEP initiatives not yet registered Description based Review of BEP M & E Relevant BEP Low Evaluation Question Type of Analysis Conducted Data Sources and Methods Used Interview Selection Process and Size of Sample Limitations / Concerns on content analysis of primary data records and relevant reports documentation provided to Evaluation Team response rate/ under￾reporting/ discrepancies Q.4 How successfully did the project gather the support of relevant groups for reform? Description – based on content analysis of expert opinions Key informant interviews with key personnel in BEP stakeholder’s organizations Interviewees identified and agreed by Evaluation Team, BEP personnel and USAID Description based on content analysis of primary data Review of BEP M & E records and relevant reports + focus groups Relevant BEP documentation provided to Evaluation Team As listed for Q1,2,3 + Difficulty in triangulating responses from respondents to reach meaningful conclusion Q.5 To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates? Description – based on content analysis of expert opinions Key informant interviews with key personnel in BEP’s stakeholder’s organizations Interviewees identified and agreed by Evaluation Team, BEP personnel and USAID Description based on content analysis of primary data Review of BEP M & E records and relevant reports + focus groups Relevant BEP documentation provided to Evaluation Team As listed for Q1,2,3 + Difficulty in triangulating responses from respondents to reach meaningful conclusion BEP Overall Monitoring & Evaluation Matrix PERFORMANCE MONITORING PLAN This document contains the Performance Monitoring Plan (PMP) for USAID’s Business Enabling Project (BEP) in Serbia. It is submitted by Cardno Emerging Markets USA, Ltd. (Cardno), prime contractor for BEP, pursuant to Section F.6 d) of Contract No. 169-C-00-11-00001-00 (“the Contract”). It was originally submitted with – and connected to – the Project Work Plan for Year 1. The PMP has been amended to add indicators for new Tasks and Activities and delete indicators relating to Component 4, which was not activated pursuant to USAID’s instructions. The PMP tracks results of BEP’s components, objectives, tasks, and activities as set forth in the Work Plan. Overview Performance monitoring is a critical part of project management. Performance monitoring on BEP will allow for 1) effective management decision- making, including selection of activities in which to invest project resources; 2) assigning responsibility and ensuring accountability for meeting progress milestones, deliverables and results; 3) learning and correcting; and 4) documenting and reporting results attributable to interventions. Performance Monitoring Plan (PMP) The PMP is the sets forth the procedure for monitoring, evaluating, and measuring the results of BEP’s interventions. The BEP work plan submitted in conjunction with the PMP is the vehicle for translating the general causal model into an implementation program. The PMP presents objectives, indicators, measurement units, data collection methodology, schedule of collection, baselines and targets over five years of the project. The BEP Tas k Order established five key Intermediate and Sub-Intermediate Results for the project: ■ Business Enabling Environment Improved ■ More Effective Business Regulation ■ Macroeconomic Environment Improved ■ Improved Fiscal Policy ■ Financial Markets Developed. The PMP is designed to capture progress toward each of the Intermediate Results. The PMP presents a top down causal model, s tarting with the project Results listed above. All of the indicators measuring the Intermediate and Sub-Intermediate Results are impact indicators that evaluate the substantive impact the project will have at the end of 5 years. The Intermediate Results are followed by indicators that assess the impact of activities under the major objectives in the 4 components listed in the Work Plan. The Work Plan presents a summary of the Project’s five year strategy in narrative form and specific activities to be implemented in the first year of BEP. Thus, even though the Work Plan presents activities over the first year of the Project, the objectives under each of the components are fixed and will span 5 years. Key elements of the PMP are as follows: ■ Objectives: Major areas of work under each of the 4 components listed in the Work Plan that span the life of the BEP project. ■ Indicators: Qualitative or quantitative measurements of competitiveness in a market economy. The PMP indicators are a mix of output, out come and impact measurements that assess both the intermediate and longer tem impact of proposed interventions. There are also custom qualita tive indicators under the 4 components that can be validated empirically through interviews or assessments whose results can be presented clear and consistently. ■ Measurement Unit: A qualitative or quantitative unit of measure of the indicators. ■ Data Collection Method: The source from which the indicators are derived. Indicators have been selected from a number of resources that measure effectiveness in areas relevant to USAID’s strategic goals, including: competitiveness; ease of doing business; financial market sophistication; and investment. These sources include World Economic Forum Global Competitiveness Indicators, OECD, International Budget Partnership, World Bank Doing Business, and Public Expenditure and Financial Accountability (PEFA) indicators. The project will also conduct custom project surveys, standard cost modeling and analyses where adequate measurements of project related objectives are not available. Finally, the project will utilize published reports and statistics produced by the GoS and other sources where available as baseline and target measurements. ■ Schedule of Collection: This refers to frequency in which the data will be available and collected. For example, the collection of data measuring the impact of an activity can be done on an annual basis. ■ Illustrative Baseline Value/ Source: The base or the start of measurement as compared to the target. ■ Targets: The qualitative or quantitative measurement of project related impacts over a 5 year horizon. Tailored Serbia BEP Surveys Standard Cost Modeling (SCM): Standard method for determining the administrative burdens for businesses imposed by regulation. The method can be used to measure a single law, selected areas of legislation or to perform a baseline measurement of all legislation in a country. Furthermore the SCM is also suitable for measuring simplification proposals as well as the administrative consequences of a new legislative proposal. It provides a quantitative indicator of the cost-savings of reforms, including both money and time (the latter also quantified in monetary terms). Custom Project Surveys: Tailored surveys performed each year of the project designed to capture and measure impacts of project related interventions where reliable information and data is not readily available. BEP project surveys will be conducted across all components establishing baselines and annual targets in a number of areas including public-private dialogue, outreach, education and access to finance. BEP Performance Management System The BEP project has a robust performance management and reporting system in place. The BEP project has a custom Project Activities Database (PAD) designed to record, analyze and report project data. It is a management tool for which performance indicators and qualitativ e and quantitative results are tracked for all project related activities. The database allows the project to capture and analyze a number of outputs related to activities and project resources not measured directly by the PMP, including all of the Intermediate Results that fall under the AO1 Results Framework. The database also provides the project team the ability to produce up-to-date and comprehensive reports on demand, informing USAID and stakeholders on project related interventions and results. Along these lines, the BEP project provides regular reports to USAID that will include tables that track progress against the PMP indicators. In addition, the BEP project has a full-time dedicated M&E Specialist for comprehensive and accurate tracking of project performance. The M&E Specialist’s tasks include: ■ Manage the development and implementation of the baseline, annual and final surveys ■ Design and implement procedures for data quality assurance and verification ■ Support BEP knowledge management ■ Prepare customized reports as requested by USAID ■ Manage the Project Activities Database software application ■ Establish standard procedures for entering and updating information in the PAD ■ Coordinate installation of and training on the PAD at implementing partners. Draft Outline of Evaluation Report TABLE OF CONTENTS LIST OF ACRONYMS AND ABBREVIATIONS EXECUTIVE SUMMARY 1. INTRODUCTION: BACKGROUND AND ABBREVIATIONS 1.1. Description of the Business Enabling Project (BEP) 1.2. Purpose of the Evaluation 1.3. Methodology of the Evaluation 2. REVIEW 1: ASSESS HOW SUCCESSFUL WAS THE PROJECT IN REDUCING REGULATORY BURDENS IN SERBIA 2.1. Summary of Evaluation Questions – Design and Methods 2.2. Findings 2.3. Conclusions 2.4. Recommendations 3. REVIEW 2: ASSESS HOW SUCCESSFUL WAS THE PROJECT IN INCREASING THE GOVERNMENT OF SERBIA’S CAPACITY AND TRANSPARENCY IN PUBLIC FINANCIAL MANAGEMENT 3.1. Summary of Evaluation Questions – Design and Methods 3.2. Findings 3.3. Conclusions 3.4. Recommendations 4. REVIEW 3: ASSESS HOW SUCCESSFUL WAS THE PROJECT IN CREATING CONDITIONS FOR INCREASED FINANCE FOR THE PRIVATE SECTOR 4.1. Summary of Evaluation Questions – Design and Methods 4.2. Findings 4.3. Conclusions 4.4. Recommendations 5. REVIEW 4: ASSESS HOW SUCCESSFULLY DID THE PROJECT GATHER THE SUPPORT OF RELEVANT GROUPS FOR REFORM 5.1. Summary of Evaluation Questions – Design and Methods 5.2. Findings 5.3. Conclusions 5.4. Recommendations 6. REVIEW 5: ASSESS TO WHAT EXTENT HAS THE PROJECT BEEN EFFECTIVE IN DEVELOPING PARTNERSHIPS THAT HAVE CATALYZED REFORM IN AREAS IN WHICH THE PROJECT OPERATES 6.1. Summary of Evaluation Questions – Design and Methods 6.2. Findings 6.3. Conclusions 6.4. Recommendations 7. LESSONS LEARNED: IDENTIFY OPPORTUNITIES TO IMPROVE IMPACT AND ENHANCE IMPLEMENTATION BY THE END OF THE PROJECT 7.1. Findings 7.2. Conclusions 7.3. Recommendations 8. CONCLUSION Annexes A. Statement of Work B. Evaluation Design and Methodology C. List of Persons Interviewed D. List of Documents Reviewed E. Overall Monitoring and Evaluation Matrix ANNEX III: SOURCES OF INFORMATION LIST OF PERSONS INTERVIED Government of Serbia: Person interviewed Position Organization 1 Zoran Milikic Assistant Minister Ministry of Finance and Economy 2 Katarina Obradovic Jovanovic Head of SME department Ministry of Finance and Economy 3 Jasna Matic Adviser on Competitiveness and the Knowledge Economy Ministry of Finance and Economy 4 Vladana Jovic Assistant Minister Ministry of Justice and Public Administration 5 Dusan Protic, Assistant Minister Ministry of Foreign and Internal Trade and Telecommunications 6 Zoran Martinovic State Secretary Ministry of Labor, Employment & Social Policy 7 Branko Budimir Advisor European Integration Office, Government of Serbia (SEIO) 8 Mira Prokopijevic Director Office for Regulatory Reform and Regulatory Impact Assessment, Government of Serbia 9 Dragana Aleksic Associate Office for Regulatory Reform and Regulatory Impact Assessment, Government of Serbia 10 Snezana Antonijevic Head of Training Centre Government of Serbia’s Human Resources Office (SUK) 11 Vida Uzelac Board Member and Acting Director Central Securities Depository and Clearing House 12 Zoran Cirovic President Securities Commission, Government of Serbia 13 Nikola Altiparmakov Member Fiscal Council, Serbia 14 Vesna Kovac Member Parliamentary Committee on Finance, State Budget and Control of Public Spending, The National Assembly Associations/NGOs/Research Institutions Person interviewed Position Organization 15 Amalija Pavic Deputy Executive Director AMCHAM – American Chamber of Commerce in Serbia 16 Milorad Bjeletic Executive Director Belgrade Open School – BOS 17- 18 Gordana Matkovic Marko Paunovic Director of Social Policy Studies Economist CLDS – Center for Liberal Democratic Studies 19 Ana Firtel Executive Director Foreign Investors Council 20 Sasa Randjelovic Assistant Professor/ Associate Faculty of Economics/ FREN 21 Olivera Popovic Vice-president Association of Business Women 22 Jelena Bojovic Policy Director NALED - National Alliance for Local Economic Development 23 Milos Damnjanovic Analyst NALED - National Alliance for Local Economic Development 24 Slobodan Krstovic Economic Analyst NALED - National Alliance for Local Economic Development 25 Mihailo Vesovic Vice-president Serbian Chamber of Commerce 26 Vidosava Dzagic Vice-president Serbian Chamber of Commerce 27 Svetlana Kisic Special Advisor Economics Institute 28 Jelena Bulatovic Executive Director Serbian Association of Managers - SAM 29- 32 Ivana Petrovic Sanja Veljkovic Jelena Karalejic Gordana Djurdjevic Manager General Manager Logo Markets Manager BUCK CEO Slovo Exporters Association of Serbia (EAS) 33 Toplica Spasojevic President Association of Corporate Directors of Serbia - UKDS 34 Branko Radulovic Advisor Balkan Center for Regulatory Reform International / Donor Organizations Person interviewed Position Organization 35- 36 Oliver Orton Igor Matijevic Regional Project Manager, Corporate Governance IFC Europe & Central Asia Operations Officer, Private Sector Development IFC – International Finance Corporation 37 Natasa Gajic European Integration and Trade Officer EU Delegation to Serbia 38 Lazar Sestovic Chief Economist World Bank in Serbia 39 Timo Menniken Head of Operations for the Western Balkans German International Development Agency - GIZ USAID Serbia BEP Project Team and USAID Serbia Person interviewed Position Organization 40- 42 Joe Lowther Dragana Stanojevic Dusan Vasiljevic Chief of Party Deputy Chief of Party Team Leader – Business Regulations and Economic Governance USAID Business Enabling Project 43 Aleksandar Djureinovic BEP Project Contracting Officer USAID Serbia Note: In-Brief meeting held with Timothy Donnay, Susan Kutor, Alison Case, Miodrag Bogdanovic, Aleksandra Krzic, Ivan Vukojevic, Jelena Mihajlovic BIBLIOGRAPHY OF DOCUMENTS REVIEWED Studies/Reports: 1. Action Plan for Fulfillment of Priorities under the EC 2010 Progress Report with the Aim of Accelerating Candidate Country Status, Republic of Serbia, Government, March 2010 2. Action Plan for Fulfillment of the European Commission Recommendations under the Serbia 2012 Progress Report in the European Integration Process, Republic of Serbia, Government, April 2013 3. National Plan for the Adoption of Acquis (2013-2016), European Integration Office, February 2013 4. Amended National Programme for Integration of the Republic of Serbia into the European Union, Republic of Serbia, Government, December 2009 5. Investment Compact for South Eastern Europe - Fostering investment, competitiveness and private sector development, OECD, 2011 6. SME Policy Index, Western Balkans and Turkey 2012, Progress in the Implementation of the Small Business Act for Europe, OECD, 2012 7. Serbia 2012 Progress Report, European Commission, 2012 8. Enlargement Strategy and Main Challenges 2012-2013, European Commission, 2012 9. White Book – Proposals for Improvement of the Business Environment in Serbia, Foreign Investors Council, 2012 Documents Provided by USAID: 1. A Roadmap for the Government of Serbia to Streamline Construction Permits, USAID Business Enabling Project, January 2013 2. Assessment of Constraints to Construction Permits in Serbia, USAID Business Enabling Project, May 2012 3. Business Survey: Serbia 2012, USAID Business Enabling Project, December 2012 4. Draft – Financing the Growth of Small and Medium-sized Businesses, Critical Issues and Recommendations for Serbia, USAID Business Enabling Project, September 2012 5. Performance Monitoring Plan, USAID Business Enabling Project, February 2012 6. USAID/Serbia and Montenegro, FY 2012 Semi-Annual Portfolio Review, USAID Business Enabling Project, November 2012 7. USAID’s Business Enabling Project Quarterly Report: First Quarter 2011, Cardno Emerging Markets, April 2011 8. USAID’s Business Enabling Project Quarterly Report: Q1 2012, Cardno Emerging Markets, April 2012 9. USAID’s Business Enabling Project Quarterly Report: Second Quarter 2011, Cardno Emerging Markets, July 2011 10. USAID’s Business Enabling Project Quarterly Report: Second Quarter 2012, Cardno Emerging Markets, July 2012 11. USAID’s Business Enabling Project Quarterly Report: Third Quarter 2011, Cardno Emerging Markets, October 2011 12. USAID’s Business Enabling Project Quarterly Report: Fourth Quarter 2011, Cardno Emerging Markets, January 2012 13. USAID Business Enabling Project First Year Work Plan (Draft), Cardno Emerging Markets, March 2011 14. USAID Business Enabling Project Second Year Work Plan (Revised and Approved), Cardno Emerging Markets, January 2012 15. USAID’s Business Enabling Project Quarterly Report: First Quarter 2013, Cardno Emerging Markets, April 2013 16. USAID’s Business Enabling Project Quarterly Report: Fourth Quarter 2012, Cardno Emerging Markets, January 2013 17. USAID Business Enabling Project Third Year Work Plan (Revised and Approved), Cardno Emerging Markets, January 2013 18. Contract No. 169-C-00-11-00001-00 USAID Business Enabling Project (BEP) 19. Policies to Increase Employment and Workforce Competitiveness, USAID Business Enabling Project and CLDS, November 2012 20. System of Non-tax and Para-fiscal Forms in the Republic od Serbia, USAID Business Enabling Project (Naled), May 2012 21. The Shadow Economy in Serbia, New Findings and Recommendations for Reform, USAID Business Enabling Project (FREN), March 2013 22. USAID BEP Gender Approach 23. USAID/Serbia and Montenegro, FY 2013 Semi-Annual Portfolio Review, USAID Business Enabling Project, April 2013 24. USAID BEP Trainings 2011-2012-2013 SUMMARY OF ONLINE SURVEY FINDINGS Question 1: Gender: Answer Options Response Percent Response Count M 35.9% 33 F 64.1% 59 answered question 92 skipped question 0 Question 2: Please note whether you are: Answer Options Response Percent Response Count Company 79.1% 72 NGO/Academic institution (including business associations) 9.9% 9 Other 11.0% 10 answered question 91 skipped question 1 Question 3: Please note 3 most important problems in Serbia’s business climate: Answer Options Response Percent Response Count Access to Finance 38.5% 35 Construction Permitting 11.0% 10 Consumer Protection 3.3% 3 Corruption 53.8% 49 Customs and trade regulations 13.2% 12 Enforcing Contracts 9.9% 9 Grey Economy 35.2% 32 Inspections Efficacy 4.4% 4 Labor Legislation 18.7% 17 Limited Investor Protection 7.7% 7 Limited Market Competition 4.4% 4 Non-transparent budget 7.7% 7 Public Enterprises 18.7% 17 Public Debt Management 4.4% 4 State Administration Efficacy (Ministries, Local Government) 35.2% 32 Taxes (complexity) 8.8% 8 Taxes and contributions (Level of financial burden on companies) 35.2% 32 Underdeveloped Financial Markets 16.5% 15 Other (please explain) 13.2% 12 answered question 91 skipped question 1 Question 4: Do you feel that these problems have improved or become worse in the last two years, on a scale of 1-5: Answer Options Response Percent Response Count 1 (much worse) 5.7% 5 2 (worse) 24.1% 21 3 (same – no change) 49.4% 43 4 (somewhat improved) 20.7% 18 5 (significantly improved) 0.0% 0 Please add any other comments here: 4 answered question 87 skipped question 5 Question 5: In your opinion, has Serbia’s business enabling environment improved in the last two years (from 2011)? Answer Options Response Percent Response Count Yes, significantly 1.1% 1 Yes, somewhat 21.8% 19 There is no change 41.4% 36 It has become somewhat worse 18.4% 16 It has become significantly worse 17.2% 15 Please add any other comments here: 4 answered question 87 skipped question 5 Question 6: More specifically, has construction permitting improved? Answer Options Response Percent Response Count Yes, significantly 0.0% 0 Yes, somewhat 25.4% 16 There is no change 68.3% 43 It has become somewhat worse 4.8% 3 It has become significantly worse 1.6% 1 Please add any other comments here: 8 answered question 63 skipped question 29 Question 7: Have inspections become more efficient? Answer Options Response Percent Response Count Yes, significantly 0.0% 0 Yes, somewhat 17.6% 13 There is no change 67.6% 50 It has become somewhat worse 14.9% 11 It has become significantly worse 0.0% 0 Please add any other comments here: 5 answered question 74 skipped question 18 Question 8: Has labor-related legislation become more effective? Answer Options Response Percent Response Count Yes, significantly 0.0% 0 Yes, somewhat 19.2% 15 There is no change 64.1% 50 It has become somewhat worse 10.3% 8 It has become significantly worse 6.4% 5 Please add any other comments here: 6 answered question 78 skipped question 14 Question 9: Has the removal of some non-tax and parafiscal charges improved doing business in Serbia? Answer Options Response Percent Response Count Yes, significantly 2.6% 2 Yes, somewhat 39.7% 31 There is no change 51.3% 40 It has become somewhat worse 7.7% 6 It has become significantly worse 0.0% 0 Please add any other comments here: 4 answered question 78 skipped question 14 Question 10: In your opinion, has public financial management improved in the last two years? Answer Options Response Percent Response Count Yes, significantly 1.3% 1 Yes, somewhat 19.0% 15 There is no change 44.3% 35 It has become somewhat worse 27.8% 22 It has become significantly worse 7.6% 6 Please add any other comments here: 4 answered question 79 skipped question 13 Question 11: In your opinion, have financial markets become stronger in the last two years? Answer Options Response Percent Response Count Yes, significantly 0.0% 0 Yes, somewhat 12.3% 10 There is no change 32.1% 26 It has become somewhat worse 37.0% 30 It has become significantly worse 18.5% 15 Please add any other comments here: 1 answered question 81 skipped question 11 Question 12: Have you heard of USAID before? Answer Options Response Percent Response Count YES 97.7% 84 NO 2.3% 2 answered question 86 skipped question 6 Question 13: Have you heard of USAID Business Enabling Project? Answer Options Response Percent Response Count YES 45.3% 39 NO 54.7% 47 answered question 86 skipped question 6 Question 14: If yes, how successful was the project in reducing the regulatory burdens to businesses in Serbia in your opinion, on a scale of 1-5, with 1 being not successful and 5 extremely successful? Answer Options Response Percent Response Count 1 3.3% 1 2 26.7% 8 3 53.3% 16 4 13.3% 4 5 3.3% 1 Please add any other comments here: 2 answered question 30 skipped question 62 Question 15: How successful was the project in increasing the Government of Serbia’s capacity and transparency in public financial management (please rate on a scale of 1 to 5)? Answer Options Response Percent Response Count 1 21.4% 6 2 42.9% 12 3 25.0% 7 4 10.7% 3 5 0.0% 0 Please add any other comments here: 2 answered question 28 skipped question 64 Question 16: How successful was the project in creating conditions for increased finance for the private sector (please rate on a scale of 1 to 5)? Answer Options Response Percent Response Count 1 14.8% 4 2 40.7% 11 3 22.2% 6 4 14.8% 4 5 7.4% 2 Please add any other comments here: 2 answered question 27 skipped question 65 Question 17: How successfully did the project gather the support of relevant groups for reform (please rate on a scale of 1 - 5)? Answer Options Response Percent Response Count 1 10.7% 3 2 35.7% 10 3 32.1% 9 4 14.3% 4 5 7.1% 2 Please add any other comments here: 2 answered question 28 skipped question 64 Question 18: To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates (please rate on a scale of 1 - 5)? Answer Options Response Percent Response Count 1 10.7% 3 2 21.4% 6 3 39.3% 11 4 21.4% 6 5 7.1% 2 Please add any other comments here: 2 answered question 28 skipped question 64 Question 19: How successful has the project been in creating business enabling environment for Public Private Partnerships (please rate on a scale of 1-5)? Answer Options Response Percent Response Count 1 20.7% 6 2 31.0% 9 3 41.4% 12 4 6.9% 2 5 0.0% 0 Please add any other comments here: 4 answered question 29 skipped question 63 Question 20: Have you worked with USAID Business Enabling Project? Answer Options Response Percent Response Count YES 22.6% 7 NO 77.4% 24 answered question 31 skipped question 61 Question 21: If yes, have you participated in trainings provided by USAID Business Enabling Project? If yes, on a scale of 1 to 5, how would you rate the effectiveness of training on business enabling process in Serbia? Answer Options Response Percent Response Count 1 0.0% 0 2 25.0% 2 3 25.0% 2 4 37.5% 3 5 12.5% 1 Please add any final comments here: 0 answered question 8 skipped question 84 Question 22: If yes, are you satisfied with your cooperation with USAID Business Enabling Project, on a scale of 1 to 5, with 1 being least and 5 most satisfied? Answer Options Response Percent Response Count 1 0.0% 0 2 12.5% 1 3 25.0% 2 4 12.5% 1 5 50.0% 4 Please add any final comments here: 0 answered question 8 skipped question 84 Question 23: This is the end of the questionnaire. Please add any final comments here. Thank you for your time and effort! Answer Options Response Count 7 answered question 7 skipped question 85 SUMMARY OF INTERVIEW NOTES (DIRECT RESPONSES TO QUESTIONNAIRE) Please note that not all interviewees were asked all questions. Gender: Answer Options Response Percent Response Count M 43.24% 16 F 56.76% 21 answered question 37 Question 1: Please note whether you are: Answer Options Response Percent Response Count NGO/Academic institution (including business associations) 9.9% 18 Government institution 11.0% 13 answered question 31 Question 2: Please note 3 most important problems in Serbia’s business climate: Answer Options Response Percent Response Count Access to Finance 4.4% 6 Construction Permitting 10.99% 17 Consumer Protection 1.1% 1 Corruption 14.29% 20 Customs and trade regulations 0% 0 Enforcing Contracts 6.59% 12 Grey Economy 12.09% 17 Inspections Efficacy 3.3% 4 Labor Legislation 6.59% 10 Limited Investor Protection 2.2% 4 Limited Market Competition 3.3% 5 Non-transparent budget 1.1% 2 Public Enterprises 3.3% 6 Public Debt Management 1.1% 2 State Administration Efficacy (Ministries, Local Government) 9.89% 13 Taxes (complexity) 4.4% 7 Taxes and contributions (Level of financial burden on companies) 5.49% 8 Underdeveloped Financial Markets 2.2% 3 Other (mainly note judiciary, policy unpredictability) 7.69 11 answered question 45 Question 3: Do you feel that these problems have improved or become worse in the last two years, on a scale of 1-5: When separating the responses in government and international representatives and non-governmental representatives (including business associations and research organizations), respectively, the results are as follows: Government/International Organizations Non-Government Organizations Question 4: In your opinion, has Serbia’s business enabling environment improved in the last two years (from 2011)? Aggregate Responses: Government/International Organizations Non-Government Organizations Question 5: More specifically, has construction permitting improved? Aggregate Responses: Government/International Organizations Non-Government Organizations Question 6: Have inspections become more efficient? Aggregate Responses Government/International Organizations Non-Government Organizations Question 7: Has labor-related legislation become more effective? Aggregate Responses Government/International Organizations Non-Government Organizations Question 8: Has the removal of some non-tax and parafiscal charges improved doing business in Serbia? Aggregate Responses Government/International Organizations Non-Government Organizations Question 9: In your opinion, has public financial management improved in the last two years? Aggregate Responses Government/International Organizations Non-Government Organizations Question 10: In your opinion, have financial markets become stronger in the last two years? Aggregate Responses Government/International Organizations Non-Government Organizations Question 11: Have you heard of USAID before? Aggregate Responses Question 12: Have you heard of USAID Business Enabling Project? Aggregate Responses Question 13: If yes, how successful was the project in reducing the regulatory burdens to businesses in Serbia in your opinion, on a scale of 1-5, with 1 being not successful and 5 extremely successful? Aggregate Responses Government/International Organizations Non-Government Organizations Question 14: How successful was the project in increasing the Government of Serbia’s capacity and transparency in public financial management (please rate on a scale of 1 to 5)? Aggregate Responses Government/International Organizations Non-Government Organizations Question 15: How successful was the project in creating conditions for increased finance for the private sector (please rate on a scale of 1 to 5)? Aggregate Responses Government/International Organizations Non-Government Organizations Question 16: How successfully did the project gather the support of relevant groups for reform (please rate on a scale of 1 - 5)? Aggregate Responses Government/International Organizations Non-Government Organizations Question 17: To what extent has the project been effective in developing partnerships that have catalyzed reform in areas in which the project operates (please rate on a scale of 1 - 5)? Aggregate Responses Government/International Organizations Non-Government Organizations Question 18: How successful has the project been in creating business enabling environment for Public Private Partnerships (please rate on a scale of 1-5)? Aggregate Responses Government/International Organizations Non-Government Organizations Question 19: Have you worked with USAID Business Enabling Project? Aggregate Responses Government/International Organizations Non-Government Organizations Question 20: If yes, have you participated in trainings provided by USAID Business Enabling Project? If yes, on a scale of 1 to 5, how would you rate the effectiveness of training on business enabling process in Serbia? Aggregate Responses Government/International Organizations Non-Government Organizations Question 21: If yes, are you satisfied with your cooperation with USAID Business Enabling Project, on a scale of 1 to 5, with 1 being least and 5 most satisfied? Aggregate Responses Government/International Organizations Non-Government Organizations Question 22: How familiar are you with the BEP objectives, approach & work-plan? Aggregate Responses Government/International Organizations Non-Government Organizations Question 23: How have you individually and your organization been involved in the USAID BEP Project? Aggregate Responses Government/International Organizations Non-Government Organizations Question 24: Is BEP is addressing the key issues/problems? Aggregate Responses Question 25: Do you think all relevant stakeholders were included in BEP’s consultations and activities? Aggregate Responses Government/International Organizations Non-Government Organizations Question 26: Has your organization been involved in directly seeking and hearing the views and arguments of the business community and other end￾beneficiaries of BEP’s activities? Aggregate Responses Government/International Organizations Non-Government Organizations Question 27: Do you feel your organization had sufficient opportunity to provide views and inputs affecting the design and implementation of project activities? Aggregate Responses Government/International Organizations Non-Government Organizations Question 28: Which of the BEP activities involving your organization have been the most useful/successful? Aggregate Responses Government/International Organizations Non-Government Organizations Question 29: What is your view of the greatest obstacles faced in implementing the activities and how were they overcome? How efficient were the methods applied? Responses only in form of comments (see below) Question 30: Are there any outstanding issues or problems connected with this activity and why do they remain? Responses only in form of comments (see below) Question 31a: How do you rate BEP from the point of view of a) effectiveness (on a scale of 1-5) Aggregate Responses Government/International Organizations Non-Government Organizations Question 31b: and b) efficiency in implementing the work-plan and achieving goals? (on a scale of 1-5) Aggregate Responses Government/International Organizations Non-Government Organizations Question 32: Are you satisfied with your cooperation with USAID Business Enabling Project, on a scale of 1 to 5, with 1 being least and 5 most satisfied? Aggregate Responses Government/International Organizations Non-Government Organizations ANNEX IV: PERFORMANCE ANALYSIS TABLE Q.1 Indicator (Source) Collection schedule Baseline (Year) 2011/Target 2012/Target 2013/Target (quarter) IR 1.1 Business Enabling Environment improved Business Environment Improved (WEF GCI) Annual 2.34 (2011) 2.34/2.34 2.4/2.34 ?/2.44 IR 1.1.1 More Effective Business Regulation Decrease in costs for businesses to comply with regulatory framework (SCM) Annual 4.2% GDP (2011) ?/3.2% reduction incl. inflation & 8% by Year 5 IR 1.4 Macroeconomic environment improved Increase in FDI (NBS) Annual 3.3% GDP (2010) 5.8%/3.3% (reported in March 2012) /3.3% /4% IR 1.4.1Improved fiscal policy Improvements in the Open Budget Index (International Budget Partnership OBI Survey) Bi-Annual 54 (2010) N/A 39 ? IR 1.4.2 Financial markets developed Improved Government Balance as % GDP (MoF). Annual -4.4% (2010) /-4.5% -6.6%/-4.5% ?/N/A Improved score in WEF GCI Financial Market Development (formerly Sophistication) 8th pillar. Annual 3.74 (2010) 3.74/3.74 3.7/3.8 (September 2012) ?/3.84 Credit to private sector as % GDP (NBS) Annual 40% of GDP (20100 43.7%/60% by Year 5 54%/60% by Year 5 ?/60% by Year 5 Number of person hours of training of journalists from publications covering business (Project report) N/A /400 1035/800 90/800 `Component 1 1. Improve & Streamline business regulation No. of commercial laws & regulations simplified and implemented with international standards as a result of USG assistance (Project reports) Annual 3 (2011) 3/3 42/10 ?/10 IN YEAR 1 WP. NOT IN FOLLOWING. Reductions in inspections related costs to businesses (SCM) Annual IN REVIZED PMP: Average time spent by management working with inspections (USAID Business Survey) Annual 59 hours (2011) 59/59 34/59 ?/59 Number of professional enforcement officers trained, passed the exam, and ready to get appointed (MoJ Data) Annual N/A 121/300 ? Number of days to enforce a judgment (WDBD) Annual 635 (2011) 635/635 635/530 Number of days dealing with construction permits (WBDB) Annual 279(2011) 279/279 269/220 (by Yr 5) Business perception of the burden of laws and regulations improves (USAID Business Survey) Annual 17% (2011) 17%/17% 26%/18% IN YR 1 WP: Increase in business sector participation on laws improving regulatory environment (USAID Business Survey) Annual 50% involved by Yr 5 50% involved by Yr 5 50% involved by Yr 5 IN YR 1 WP: Quality of regulatory reforms introduced & sustained (USAID Business Survey) Annual Perception improves 25% Perception improves 25% Perception improves 25% Significant progress of realization of new labor legislation recommended by the 2011 White Book by the FIC (FIC White Book) Annual N/A /1 /2 Decrease in % of businesses responding that labor law has negative impact on business (USAID Business Survey) Annual 75%(2011) 75%/75% 66%/65% by year 3 and 55% by year 5 ?/65% 2.Business effectively partners with GoS Increase in business sector participation on laws improving regulatory environment (USAID Business Survey) Annual 19%(2011) 19%/19% 25%/21% ?/25% Number of PP dialogue mechanisms/events on business regulation issues created by the project (Project reports) Annual N/A 3 18/3 15/5 3. Improve & support RIA IN YR 1 WP: Number of RIAs conducted /90 /100 /140 Number of substantial RIAs conducted by GoS (ORR reports, Project Reports) Quarterly 8(2010) 17/10 27/15 2/20 Business perception of transparency & predictability increase (USAID Business Survey) Annual 23%(2010) 23%/23% 23.5%/25% ?/28% 4. Improve the Gov’s competition policy IN YR 1 WP: Number of members of the Commission for the Protection of Competition, Admin & Supreme Court judges trained in Competition Law (Project reports) Quarterly IN YR 1 WP. Improvement in WEF GCI Anti-Monopoly Policy IN YR 1 WP: Number of Corporate Governance OECD Principles complied with in Serbian legislation 5. Improve Corporate Governance standards, policies & regulations Improvement in Doing Business Strength of Investor Protection Index (WBDB) Annual 5.3(2011) 5.3/5.3 5.3/5.3 ?/6 Number of Commercial Court judges trained in the Implementation of Company Law (Project reports) Quarterly N/A 20???? 44 ? 6. Create enabling environment for PPPs Number of training hours to representation of public sector entities and private companies trained in PPP (Project reports) Annual N/A 1438.5/200 322.5/200 Completion of regulatory framework (bylaws and methodologies) for PPP (Project reports) Annual Non-existence 0 0/0 0/Complete in 2013 Component 2 1. Improve GoS budgeting, fiscal policy analysis, and financial management efficiency Improvement in the quality of Medium Term Budgeting & Prioritization (Project Reports) Annual No effective budgeting framework in Place Year 1 target FSR report includes GDP estimates. Reported to be met. Year 2 target of FSR including estimates of deficit allegedly achieved. Reported to be well on track to achieve Yr 4 target (Ministry budgeting reflects strategic priorities)?? PEFA Multi-Year Score perspective in fiscal planning (PEFA Assessment) Annual/published every 3 years C score (2010) Score due Yr 3 Scheduled for November 2013 PEFA Assessment Score on predictability of availability of funds (PEFA Assessment) Annual/Published every 3 years C+ score (2010) Score due Yr 3 Scheduled for November 2013 Number of articles published in area of fiscal policy …with BEPA assistance supporting changes in fiscal policy (Project report) Monthly N/A 4/1 7/3 1/3 IN YR 1 WP: Quality of public dialogue and debate on macroeconomic and fiscal issues (Project report, phone survey) 1 Number of activities related to macroeconomic issues aimed at improving public-private dialogue (Project report) Annual N/A /2 7/3 5/4 2. Improve public debt management and conditions for Government financing Value of dinar denominated debt (PDA) Annual 14% of total borrowing (2010) 14%/20% by year 5 18.27%/20% by year 5 (November 2012) 18.1%/20% by year 5 (Feb 2013) Government initiates a public issue of debt to more diversified investor base (PDA) Annual 87.1% debt owned by banks (PDA, 2011) 87.1%/75% by year 5 80.38%/75% by year 5 53.6%/75% by year 5 (February 2013) Operational status of debt management system and reporting (PDA report) Annual N/A /Not functioning /2 reports (operational) /6 reports Proportion of key PDA staff trained in debt management, economic and risk analysis (Project records) Annual N/A /10% 35%/? ?/50% 80% of staff from front & middle offices trained 3. Improve macroeconomic policy & strategy Improvements in WEF GCI on Macroeconomic Environment addressing National Savings Rate, as % GDP (WEF GCI reports) Annual 14.8% (2011) 14.8%/18% by Year 5 16.1%/18% by Year 5 ?/18% Number of published studies and articles….with BEP assistance…supporting changes in macroeconomic policy (Project report) Annual N/A 3/2 3/5 2/8 Awareness of Government’s Economic Growth Strategy (USAID Business Survey) Annual 20% (2011) 20%/20% 38%/20% ?/25% Number of person hours of training of GoS staff on macroeconomic policy and public financial management topics (Project reports) Annual N/A 633/500 4100/1000 379/1000 Component 3 1. Increase capital available to business through NBFIs IN YR 1 WP: Number of hours of training of journalists from publications that cover business (Project reports) Annnual 163/400 Value of credit financed from NBFI’s to business N/A /0 /0 /5% Status of the legal and market framework for Regulatory Commodities /Establishment of a commodities exchange framework not in place exchange & regulatory framework established commodities exchange 2. Strengthen integrity of the capital markets by helping GoS to build capacity in ensuring compliance with securities laws Number of intermediaries trained and advised about upcoming securities laws (Securities Commission report) 205 advises/75% trained by Year 5 417 trained including year 1/75% trained by Year 5 /75% trained by Year 5 IN YR 1 WP: Number of financial regulators trained in non￾bank regulatory matters 3. Improve GoS access to finance through development of LT, liquid public market Value of dinar debt (Statistics Office) Annual 14% (2010) 15.3%/20% by Year 5 (November 2011) 18.9/20% by Year 5 (December 2012) 18.1%/20% by Year 5 (February 2013) In YR 1 WP: Issuance of public debt through a primary dealer network or other distribution mechanism. Government initiates a public issue of debt to more diversified investor base. (PDA) Annual 87.1% debt owned by banks (PDA, 2011, March) 87.1%/75% by year 5 80.38%/75% by year 5 (November 2012) ?/75% by year 5 4. Strengthen public￾private dialogues on the financial sector Increase the awareness of the role of the financial sector in economic development (USAID Business Survey) Annual 25% awareness 25%/50% by year 5 83.9%/50% by year 5 ?/50% by year 5 5. Strengthen the disclosure and transparency supporting new investment by assisting GoS to implement accounting, audit & disclosure standards Compliance with IOSCO standards in areas of securities intermediaries and equity issuance improves (Project & IOSCO reports) Annual 50% /Compliance 60% by Year 5 /Compliance 60% by Year 5 ?/Compliance 60% by Year 5 (IOSCO data not yet available) 6. Improve access to SME finance Improve access to finance (USAID Business Survey) 25% access to finance improved (2011) 23.2%/30% in Year 3 ?/30% Increase in average value of loan size and 89,475 EUR 57,927 EUR/0% ?/10% increase Color code Indication Insufficient information/To be measured Appears to have met or exceeded targets. Has not met targets (even if shortfall is small) borrowing by SMEs (2011) ?????? increase