ANTI-CORRUPTION CIVIC ORGANIZATIONS’ UNIFIED NETWORK FOLLOW-ON ACTIVITY (ACCOUNT) PERFORMANCE EVALUATION OF USAID/BIH’S ANTI￾CORRUPTION CIVIC ORGANIZATIONS’ UNIFIED NETWORK FOLLOW-ON ACTIVITY (ACCOUNT) FEBRUARY 2019, FINAL REPORT This document is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of authors and do not necessarily reflect the views of USAID or the United States Government. PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE-BIH) USAID/BIH’S ANTI-CORRUPTION CIVIC ORGANIZATIONS’ UNIFIED NETWORK (ACCOUNT) FOLLOW-ON ACTIVITY PERFORMANCE EVALUATION OF USAID/BIH’S ANTI-CORRUPTION CIVIC ORGANIZATIONS’ UNIFIED NETWORK FOLLOW-ON ACTIVITY (ACCOUNT) February 2019, Final Report Prepared under the USAID Monitoring and Evaluation Support Activity (MEASURE-BiH), Contract Number AID-168-C-14-00003 Submitted to: USAID/Bosnia and Herzegovina Submitted by: IMPAQ International, LLC PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ABSTRACT This evaluation was performed for the U.S. Agency for International Development Bosnia and Herzegovina (USAID/BiH) to examine USAID/BiH Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) follow￾on activity. The main objective of ACCOUNT is to create an environment that increases civil society participation to promote reform through collaboration and cooperation in anti-corruption initiatives. This Activity involves evidence-based research, civic monitoring, and investigative reporting on corruption cases in five sectors: i) public procurement, ii) whistleblower protection, iii) education, iv) health, and v) public employment. The evaluation focuses on the direct beneficiaries of the program and answers five research questions. The first question examines to what extent ACCOUNT has increased Civil Society Organizations’ (CSOs) involvement and input in determining anti-corruption legislation/regulation. The second question examines ACCOUNT’s major achievements in public procurement, whistleblower protection, education, health, public employment, and local-level anti-corruption strategies/action plans. The third question examines to what extent ACCOUNT has increased the quality and quantity of investigative journalism targeting public corruption in BiH. The fourth question examines to what extent interventions under ACCOUNT have influenced public awareness of corruption. Finally, the fifth question examines if ACCOUNT’s legal-aid model is perceived as effective by program beneficiaries. The evaluation team employed a mixed-methods approach to answer each of the evaluation questions through triangulation. In addition to the review of ACCOUNT’s design and implementation documentation, we held 39 semi-structured interviews and a media panel discussion, with a total of 65 individuals participating in the evaluation. We also conducted surveys of media and CSO representatives and education workers, as well as reviewing the quality of investigative reports produced by ACCOUNT media outlets. Our evaluation shows that overall ACCOUNT performance is perceived as a positive CSO endeavor in the field of anti-corruption in Bosnia and Herzegovina. ACCOUNT gathered a significant number of CSOs in the Network on anti-corruption issues. Through advocacy activities, ACCOUNT facilitated the putting forward of quality CSO inputs and their partial integration into legislation in the field of anti￾corruption. Achievements are evident in all five sectors, especially in relation to whistleblower-protection legislation and draft public-procurement legislation. The demand-driven approach was effective and contributed to the visible results. Since Annual Reports did not document progress against goals and objectives but rather according to activities, some achievements were difficult to identify and measure. Media reporting on corruption has been intensive among ACCOUNT media outlets, and an increased level of readership is noted. However, the team was not able to confirm a change in the quantity of media reporting due to the lack of any baseline data on corruption investigative reports, either in the aggregate or per individual media outlet. The quality of ACCOUNT investigative reporting within the content evaluation is generally rated above good, with one in four articles being very good or excellent. It is important to note that media reporting focuses primarily on quantity with regard to advocacy and awareness raising. However, the majority of citizens (80%) remain tied to traditional media outlets over online portals/social media, and the outreach of ACCOUNT investigative reporting remained limited. Citizens’ perceptions on corruption remain unchanged since 2015, which indicates that the ACCOUNT assumption that citizens would consume their online media products remains largely unfulfilled, as only 20 percent of citizens use the online sources as a means of consuming civic and political news. Regarding ACCOUNT’s work on the establishment of an effective free legal-aid model to protect individuals reporting corruption and whistleblowers, the model has been useful but insufficient to protect citizens in the absence of a strong institutional and judicial system. In the end, we identified recommendations for future programming to support civil society engagement in anti-corruption, involvement of the media, and provision of free legal aid. PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV TABLE OF CONTENTS ABSTRACT............................................................................................................................................................................................I ACKNOWLEDGEMENTS ................................................................................................................................................................i ACRONYMS........................................................................................................................................................................................ii EXECUTIVE SUMMARY..................................................................................................................................................................iii I. EVALUATION PURPOSE AND EVALUATION QUESTIONS.......................................................................................1 1.1. EVALUATION PURPOSE AND EVALUATION QUESTIONS............................................................................1 1.2. EVALUATION QUESTIONS.........................................................................................................................................2 2. ACCOUNT BACKGROUND...........................................................................................................................................4 3. EVALUATION METHODS AND LIMITATIONS.........................................................................................................8 3.1. METHODOLOGY AND DATA COLLECTION.....................................................................................................8 3.2. EVALUATION LIMITATIONS................................................................................................................................... 11 4. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS............................................................................... 13 4.1. EVALUATION QUESTION 1: TO WHAT EXTENT HAS ACCOUNT INCREASED CSO INVOLVEMENT AND INPUT IN DETERMINING ANTI-CORRUPTION LEGISLATION/ REGULATION? ... 13 4.2. EVALUATION QUESTION 2: WHAT ARE ACCOUNT’S MAJOR ACHIEVEMENTS IN PUBLIC PROCUREMENT, WHISTLE-BLOWER PROTECTION, EDUCATION, HEALTH, AND PUBLIC EMPLOYMENT?........................................................................................................................................................................... 18 4.3. EVALUATION QUESTION 3: TO WHAT EXTENT HAS ACCOUNT INCREASED THE QUALITY AND QUANTITY OF INVESTIGATIVE JOURNALISM TARGETING PUBLIC CORRUPTION IN BIH?.......... 27 4.4. EVALUATION QUESTION 4: TO WHAT EXTENT HAVE INTERVENTIONS UNDER ACCOUNT INFLUENCED PUBLIC AWARENESS OF CORRUPTION? ........................................................................................... 34 4.5. EVALUATION QUESTION 5: IS THE ACCOUNT LEGAL-AID MODEL TO PROTECT INDIVIDUALS REPORTING CORRUPTION AND WHISTLEBLOWERS PERCEIVED AS EFFECTIVE BY ACCOUNT BENEFICIARIES?.......................................................................................................................................................................... 37 4.6. RECOMMENDATIONS .................................................................................................................................................... 39 ANNEXES - LIST ............................................................................................................................................................................. 41 ANNEX I: EVALUATION STATEMENT OF WORK............................................................................................................ 42 ANNEX II: ACCOUNT SUB-AWARDS AND ROLES OF CSOS...................................................................................... 55 ANNEX III: ACCOUNT M&E PLAN INDICATORS ............................................................................................................. 59 ANNEX IV: LIST OF KEY INFORMANTS................................................................................................................................ 75 ANNEX V: QUALITY OF INVESTIGATIVE REPORTS PRODUCED WITHIN ACCOUNT - EVALUATION..... 76 ANNEX VI: ACCOUNT GOOGLE ANALYTICS FOR INVESTIGATIVE REPORTING.............................................. 84 ANNEX VII: MAIN DATA COLLECTION INSTRUMENTS................................................................................................ 87 PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX VIII: COMMENTS FROM THE IMPLEMENTING PARTNER AND USAID/BiH MISSION ON THE ACCOUNT EVALUATION........................................................................................................................................................128 ANNEX IX: THE EVALUATION TEAM RESPONSES TO COMMENTS FROM THE IMPLEMENTING PARTNER AND USAID/BiH MISSION.........................................................................................................................................................132 FIGURES: Figure 1. ACCOUNT Results Framework and Components..................................................................................................6 Figure 2. Distribution of Key Informants Per Stakeholder Group..........................................................................................8 Figure 3. Statistics for Online Surveys Administered to Stakeholder Groups.....................................................................9 Figure 4. National Survey of Citizens’ Perceptions 2015–2018 Sample Size........................................................................9 Figure 5. Sampling of ACCOUNT Investigative Reports....................................................................................................... 10 Figure 6. Evaluation Matrix ............................................................................................................................................................ 10 Figure 7. ACCOUNT Network Membership in 2015 and 2017.......................................................................................... 13 Figure 8. Geographic Distribution of Sub-grantees in the ACCOUNT Follow-on Phase ............................................. 14 Figure 9. ACCOUNT Progress in Achieving Targets for Indicators in the M&E Plan .................................................... 15 Figure 10. ACCOUNT Network CSOs:Total membership, members, and leads of SGs and MTs............................ 16 Figure 11. ACCOUNT Results Framework.............................................................................................................................. 18 Figure 12. ACCOUNT Institutional Map ................................................................................................................................... 20 Figure 13. ACCOUNT Achievements in Component 1, per sector................................................................................... 21 Figure 14. ACCOUNT Progress in Achieving Targets for Indicators in the M&E Plan, per sector............................ 23 Figure 15. ACCOUNT Geographic Distribution of Institutions Involved in Activities.................................................. 24 Figure 16. Investigative Reports Produced within ACCOUNT............................................................................................ 27 Figure 17. Most Frequently Mentioned Media Sources among Surveyed CSO and Media Representatives............. 28 Figure 18. Citizens’ Awareness of Anti-corruption Organizations in General and Based on Media Source............. 29 Figure 19. ACCOUNT Investigative Reporting, average scores.......................................................................................... 30 Figure 20. Surveyed Media Pool Members’ Frequency of Reporting on ACCOUNT’s Five Sector Areas.............. 33 Figure 21. Citizens’ Perceptions on Corruption in Public Employment and Public Procurement.............................. 35 Figure 22. Citizens Informed about ACCOUNT Local Anti-corruption Initiatives......................................................... 35 Figure 23. Corruption Cases Reported to Prosecutors’ Offices; and to ACCOUNT................................................... 36 Figure 25. Perceptions of Surveyed CSOs and Media Representatives on ACCOUNT Legal-Aid Provision ......... 37 i |PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ACKNOWLEDGEMENTS Guided by the U.S. Agency for International Development (USAID) Evaluation Policy, USAID Bosnia and Herzegovina (USAID/BiH) commissioned IMPAQ International through USAID/BiH Monitoring and Evaluation (M&E) Support Activity (MEASURE-BiH) to design and conduct a performance evaluation of the USAID/BiH’s Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) follow-on activity 2015–2019. MEASURE-BiH’s Evaluation Team, which worked on this evaluation and prepared this report, includes: Emina Ćosić-Puljić, Evaluation Methods Lead and MEASURE-BiH Senior Research Analyst; and Snežana Mišić Mihajlovć, Evaluation Expert Civil Society Lead and MEASURE-BiH Research Fellow. The extended team also included Boris Badža, MEASURE-BiH GIS IT/Database Management Specialist; and Mahir Hadžić, external quantitative data collection and analysis expert, both of whom have provided instrumental assistance in preparing the logistics for all data collection, organization of databases for analysis, preparing individual analyses, and providing assistance during the field-work data collection. Furthermore, this evaluation benefitted from the expert assistance of Aleksandra Krstic, PhD, teaching in the Department for Journalism and Communication at the Faculty of Political Sciences, University of Belgrade; and renowned investigative reporter Dragutin Hedel, who is currently retired but remains an active partner to Telegram.hr. The authors would like to thank those who have contributed to the design and implementation of this evaluation. At USAID/BiH, Selma Sijerčić Dilber from Democracy Office and Elma Jusić Bukvić from Program Office (also MEASURE-BiH Contracting Officer’s Representative), have been instrumental in the design of the evaluation and have assisted in its implementation. At ACCOUNT, we would like to thank both of the Implementing Partners for sharing their documentation and monitoring data with the team, as well as for clarifying any questions we had regarding those data and for proactively encouraging all the Network CSOs and Media to take part in this evaluation. At IMPAQ International, Amy Kracker Selzer, MEASURE-BiH Technical Director, provided technical guidance and advice throughout the process of preparing the report. These organizations and individuals have contributed tremendously to this report and to the MEASURE-BiH activity more generally, but any remaining errors ii |PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ACRONYMS AC Anti-Corruption ACCOUNT BiH’s Anti-Corruption Civic Organizations’ Unified Network follow-on activity APIK Agency for Prevention of Corruption and Coordination of the Fight Against Corruption BiH Bosnia and Herzegovina CGS Centar za građansku saradnju (Eng. Center for Civic Cooperation) CIN Centre for Investigative Reporting CRMA Centre for Media Development and Analysis CSO Civil Society Organization EDU Sector Area on Education EMPL Sector Area Public Employment EQ Evaluation Question EU European Union FBiH Federation of Bosnia and Herzegovina (Entity in BiH) HEALTH Sector Area Public Healthcare HJPC High Judicial and Prosecutorial Council HoR House of Representatives IH INFOHOUSE IP Implementing Partner KI Key Informant KII Key Informant Interviews KS Canton Sarajevo MEL Monitoring, Evaluation and Learning MoU Memorandum of Understanding MT Monitoring Team NGO Non-Governmental Organization NSCP National Survey of Citizens’ Perceptions PD Program Description PP Sector Area on Public Procurement PPL Public-Procurement Law RS Republika Srpska (Entity in BiH) SG Sector Group TA Technical Assistance TI Transparency International USAID U.S. Agency for International Development USAID/BiH U.S. Agency for International Development in Bosnia and Herzegovina WBP Sector Area Whistleblower Protection WBPL Whistleblower-Protection Law WG Working Group ZeDo Zenica-Doboj iii | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV EXECUTIVE SUMMARY IMPAQ International (IMPAQ) has been commissioned by the U.S. Agency for International Development Bosnia and Herzegovina (USAID/BiH) within the USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH) to conduct the evaluation of USAID/BiH’s Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) follow-on activity. The implementation of ACCOUNT began in July 2015 and will be completed by July 2019. The overall budget is $ 3,449,474, and the two implementing partners (IPs) for the Activity are the Centre for Media Development and Analysis (CRMA) and INFOHOUSE (IH). EVALUATION PURPOSE AND EVALUATION QUESTIONS Evaluation of the USAID/BiH ACCOUNT follow-on activity has three main objectives: 1. Assess implementation performance against deliverables outlined in the cooperative agreement. 2. Learn best practices from local partners for Civil Society Organizations (CSOs) fighting corruption and gain insights into the effectiveness of program mechanisms for strengthening CSOs’ ability to advocate for reform. 3. Provide recommendations based on the lessons learned to inform the design of future USAID/BiH anti￾corruption interventions. This evaluation answers the following five research questions: 1. To what extent has ACCOUNT increased CSOs’ involvement and input regarding anti-corruption legislation/regulation? 2. What are ACCOUNT’s major achievements in the selected five sector areas (public procurement, whistleblower protection, education, health, and public employment) and on the local level? 3. To what extent has ACCOUNT increased the quality and quantity of investigative journalism targeting public corruption in BiH? 4. To what extent have interventions under ACCOUNT influenced public awareness of corruption? 5. Is the ACCOUNT legal-aid model to protect individuals reporting corruption and whistleblowers perceived as effective by ACCOUNT beneficiaries (e.g., legal-aid recipients, beneficiaries of joint registry, and beneficiaries of the ACCOUNT legal team)? EVALUATION METHODS AND LIMITATIONS To address challenges faced in the data collection process, the research team employed a mixed-methods approach to answer each of the evaluation questions through triangulation. The following data sources were used: • ACCOUNT implementation documentation • 39 semi-structured key informant interviews (KIIs) with 58 key informants (KIs) • Panel discussion with media pool representatives with 7 KIs • Online survey with ACCOUNT Network CSOs and affiliated media outlets/journalists, with 30% response rate • Analysis of relevant National Survey of Citizens’ Perceptions (NSCP) 2015–2018 data • Online survey with schools/teachers that participated in ACCOUNT activities, with 40% response rate iv | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV • Evaluation of quality of investigative reports produced by ACCOUNT media outlets The main limitations encountered during the evaluation included recall bias; selection bias exacerbated by the survival bias; social desirability bias; limitations related to implementation documentation and the unstandardized reporting style at the time of the interventions; and a limited media archive. MAIN FINDINGS, CONCLUSIONS ACCOUNT’s Network has been reduced in size by focusing on a narrower set of CSOs (from 170 to 100). However, the number of CSOs actively involved in anti-corruption is higher than was the case before ACCOUNT as a result of the Network consolidation efforts. Yet CSOs with adequate expertise to deal with corruption issues still remain scarce. The Network has tried to mitigate this by focusing on the primary field of expertise of CSOs in specific sectors and introducing anti-corruption activities into their operations. Although not all CSOs’ input was accepted by policy-makers (50% of policy proposals were adopted), their input was deemed meaningful and shaped the design of the anti-corruption legislation and regulation where adopted. This input is most visible in the processes of drafting legislation for the public-procurement law and the whistleblower-protection laws on state and entity levels, with the Republika Srpska (RS) law on whistleblower protections being adopted during ACCOUNT implementation. CSOs’ expertise has improved over the years, although the dwindling of the Network’s size coupled with targeting the quality of their input resulted in fewer (29) Network CSOs being substantially involved, with most of them taking part in various types of activities and across the five sectors and the local level. Achievements are registered in all sectors. However, the ACCOUNT Program Description (PD) defined anticipated results for two sectors against which achievement can be measured (public procurement and whistleblower protection), while for the remaining sectors (health, education and employment) the results were not predefined. Furthermore, there is significant overlap between the activity components as well as the sectors (e.g., monitoring of the public-procurement process in public healthcare institutions), such that achievements cannot be documented in the same format and compared per sector in the same manner. It is important to note that the demand-driven approach was effective where partnership relations with government bodies that expressed the need for assistance led to visible results with respect to adoption of ACCOUNT recommendations or suggested procedures. However, the political will of the public institutions was also a critical element in these achievements, and ACCOUNT exploited this opportunity wherever possible. Standing out in particular is their cooperation with the Agency for Prevention of Corruption and Coordination of the Fight Against Corruption (APIK), Republika Srpska Ministry of Justice, and the Anti-Corruption Team of the Government of Canton Sarajevo (KS). Efforts across the five sectors facilitated broad outreach in terms of themes, geographic reach, and stakeholders on the development of strategies/action plans/procedures. However, in-depth work was not equally possible across all sectors and institutions, with efforts thus coming up short on implementation of adopted policies/regulations. With regard to investigative reporting, ACCOUNT over-performed with respect to the production of investigative reports in the online portals, producing 1,257 reports, with over 1,000 of these produced by Zurnal.info. Furthermore, Zurnal.info gained readership. With respect to quantity of investigative reports in general, there are two differing viewpoints between ACCOUNT’s CSOs and media outlets as opposed to governmental and public institutions. While the former indicate increased activity in reporting on corruption among ACCOUNT’s media outlets over the past three years, the governmental and public institutions note an v | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV oversaturation of media reporting on corruption in general, while showing reluctance to isolate ACCOUNT’s media outlets from the general picture. Based on the randomly selected sample of ACCOUNT’s reports, the quality of ACCOUNT’s investigative reporting, in particular that of Zurnal.info, has been assessed as good on average, with one in every four reports being assessed as very good or excellent. The reporting was focused on quantity, advocacy, and drawing attention to corruption. Timeliness or reporting on issues of interest to citizens was noted as important, and ACCOUNT through its media component had the image of being a continual watchdog over government and the public sector. The difficult and risky environment in which journalists operate, particularly those who report on corruption, has been frequently mentioned as a limiting factor for independent journalism. The media in general is subject to the strong influence of political pressure and private-sector funding, which has forced investigative and independent journalism to remain largely dependent on donor support. The evaluation team is not able to identify changes in public awareness about corruption, although CSOs, particularly those on the local level working on healthcare and education, note limited improvements in awareness among beneficiaries. Public perception has largely remained unchanged since 2015, with over 90 percent of citizens expressing the belief that petty, grand, and systemic corruption exists, including corruption in public procurement and public employment (95%). Furthermore, the public’s reaction to reporting on corruption as an indicator for change in the awareness suggests a decreasing trend from the previous year. Administrative data from both the Prosecutors’ Offices and the ACCOUNT Legal Aid Team record a decrease in the number of reports filed by citizens on corruption. The public remains faintly aware of local anti-corruption initiatives by CSOs (3%) and ACCOUNT (9%). Fear is identified as a dominating factor in not reporting corruption, to some extent due to the absence of an effective judicial system and law enforcement. Regarding the establishment of an appropriate and effective free legal-aid model to protect whistleblowers and other individuals reporting corruption, apart from the interventions within the whistleblower-protection legislation discussed above, the public does have some familiarity with ACCOUNT’s legal-aid provision, acknowledging that it is easily identifiable to citizens, accessible, and professional. However, few respondents had a detailed understanding of it, and those who did expressed the need for its further expansion. Citizens have trust in non-governmental legal-aid providers over public-sector legal-aid centers. Key informants indicated that non￾governmental legal-aid providers have developed expertise in more complex corruption cases but also expressed concern that these efforts may shrink with donor withdrawal amidst the establishment of public free legal-aid centers. vi | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV RECOMMENDATIONS Recommendation 1. Consider further consolidation of the ACCOUNT Network organized around CSOs with expertise in specific sectors and continue expanding the CSOs’ anti-corruption expertise to build on current results and credibility gained. Recommendation 2. Consider streamlining support for activities focusing on in-depth implementation of policies and procedures initially developed within all five ACCOUNT sector areas, in cooperation with government(s) and public institutions. Recommendation 3. Consider increasing cohesion between the different intervention mechanisms and coordinating between the different roles of CSOs in the Network. Recommendation 4. Maintain flexibility in adjusting the activity design to build on the opportunities of a changing political landscape to achieve better results. Recommendation 5. Continue support to independent media for providing evidence-based reporting on corruption, as the current political environment and funding sources for independent media are highly volatile and subject to political influence. Recommendation 6. Improve communication between media and CSOs to streamline high-quality reporting. Recommendation 7. Consider placing greater focus on increasing the level of quality of media reporting in general. Recommendation 8. Emphasize delivering positive, informative anti-corruption content through media reporting and improve communication of the results achieved by directing citizens’ attention to issues uncovered through investigative reporting. Recommendation 9. Continue providing support to free legal-aid providers, especially non-governmental providers, until public institutions can improve their public image. 1 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV I. EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1. EVALUATION PURPOSE AND EVALUATION QUESTIONS IMPAQ International (IMPAQ) has been commissioned by the U.S. Agency for International Development Bosnia and Herzegovina (USAID/BiH) within the USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH) to conduct this evaluation of USAID/BiH’s Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) follow-on activity. The implementation of ACCOUNT began in July 2015 and will be completed in July 2019. The overall budget is $ 3,449,474, and the two implementing partners (IPs) for ACCOUNT are the Centre for Media Development and Analysis (CRMA) and INFOHOUSE (IH). The purpose of this evaluation of the USAID/BiH ACCOUNT follow-on activity is threefold: i. Assess the Activity’s implementation performance against the stated deliverables in the cooperative agreement. ii. Learn from local partners about the best approaches for CSOs in fighting corruption, gain insights about ACCOUNT implementation mechanisms for strengthening BiH CSOs’ ability to advocate based on evidence-based arguments, and influence policy to improve the anti-corruption regulation through a coordinated approach. iii. Provide recommendations based on the lessons learned from ACCOUNT to inform the design of a potential new USAID/BiH anti-corruption intervention, as well as providing information to implementers and other international development organizations working in the same area. A variety of stakeholder groups have played a significant role in this evaluation process. The ACCOUNT funders and implementing partners provided their insights into the program design and implementation. ACCOUNT beneficiaries and representatives provided information on different aspects of the program implementation and their roles in the overall implementation of activities. This information has played an important role in assessing the intervention’s successes, measuring progress toward the overall Activity objectives, and identifying potential for future improvements. The beneficiary CSOs, who participated in implementation of various activities, including collaborations with different BiH government(s) and public institutions, provided insights into the major challenges and successes relevant to CSOs’ input in government decision making and policy design related to anti-corruption in the five main ACCOUNT sector areas (public procurement, whistleblower protection, education, health, and public employment), as well as processes relevant at the local level (municipal and cantonal levels) for anti￾corruption initiatives and procedures. Relevant government and public institutions, stakeholders, and representatives provided the same type of input from the other side, namely as recipients of the assistance, thus ensuring a more balanced picture of the usefulness of CSOs’ involvement in the decision-making processes. In addition to the already mentioned stakeholders, representatives of the media outlets assisted by the intervention provided incremental input with regard to the activities undertaken in researching and publishing investigative reports aiming to affect public awareness about corruption. They offered their insights into the volume and quality of investigative reporting and their perceptions as to the intervention effectiveness. Findings, conclusions, and recommendations from the ACCOUNT evaluation are intended to contribute to knowledge and learning of a diverse group of stakeholders: 1. USAID/BiH can reassess the implementation of the current anti-corruption intervention in BiH on involving civil society organizations, the dynamics of their participation in the decision-making processes, 2 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV achievements of the media in independent investigative reporting in terms of quality and quantity as a means of providing meaningful information to the public on current issues relating to corruption in BiH. 2. Implementing partners can learn about their strengths and areas for improvement. 3. ACCOUNT beneficiaries can reflect and understand to what extent ACCOUNT’s specific implementation mechanisms and assistance furthered their initiatives with the BiH government(s)/public institutions, to inform their future strategic approaches. The evaluation employed a rigorous design and methodological approach in order to address the evaluation questions and to provide insights into the ACCOUNT implementation and achievements. This evaluation report relates the findings to conclusions and recommendations on the civil society organizations’ anti-corruption initiatives after the first three years of ACCOUNT implementation. The ACCOUNT follow-on phase began implementation in June 2015, as a continuation of the previous ACCOUNT intervention. The follow-on phase (henceforth referred to merely as ACCOUNT) is still ongoing and will be completed in 2019. During the first three years of implementation, the intervention has seen six (6) modifications, including two program extensions (one to expand the media component, and a second related to the legal-aid provision; further details are provided in the Background section). The MEASURE-BiH team conducted this evaluation in November 2018, with the data-collection activities beginning on October 29, 2018, and completed on November 11, 2018. During the data collection process, the team extensively reviewed background documentation received from ACCOUNT implementing partners and analyzed data collected from 65 individuals through Key Informant Interviews (KIIs) and a panel discussion with media representatives. Additionally, individuals participating in the online surveys conducted with ACCOUNT Network CSOs, Media representatives, and Teachers contributed by responding on several questions on selected segments of the intervention. An evaluation of the quality of the investigative reporting was conducted by media experts who scored a randomly selected sample of investigative reports to inform the evaluation. Finally, the team analyzed the National Survey of Citizens’ Perceptions data, available 2015–2018, to inform the findings (which also included data on several ACCOUNT indicators). 1.2. EVALUATION QUESTIONS This performance evaluation addresses ACCOUNT implementation over the three-year implementation period against the stated objectives in the Program Description and provides insights into ACCOUNT’s achievements. Specifically, the evaluation addressed the following five evaluation questions (EQs): 1. To what extent has ACCOUNT increased CSOs’ involvement and input regarding anti￾corruption legislation/regulation? 2. What are ACCOUNT’s major achievements in the selected five sector areas (public procurement, whistleblower protection, education, health, and public employment) and on the local level? 3. To what extent has ACCOUNT increased the quality and quantity of investigative journalism targeting public corruption in BiH? 3 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 4. To what extent have interventions under ACCOUNT influenced public awareness of corruption? 5. Is the ACCOUNT legal-aid model to protect individuals reporting corruption and whistleblowers perceived as effective by ACCOUNT beneficiaries (e.g., legal-aid recipients, beneficiaries of joint registry, and beneficiaries of the ACCOUNT legal team)? The first two evaluation questions relate to ACCOUNT implementation mechanisms, including assistance provided to CSOs on organizing anti-corruption initiatives collaboratively in the five sector areas and at the local level as defined by the program (whistleblower protection, public procurement, employment/recruitment in the public sector, health and education, and local-level development of anti-corruption action plans and other procedures in public institutions). Furthermore, their involvement extends to approaching decision makers strategically and participating in the decision-making process. The third and fourth evaluation questions relate to ACCOUNT interventions relevant to affecting public awareness on the key corruption issues in ACCOUNT’s five sector areas and at the local level and beyond by streamlining investigative reporting and increasing primarily the quantity, but also the quality of investigative reports. The fifth question relates to ACCOUNT’s legal-aid service provision to whistleblowers and other individuals reporting corruption, a mechanism that purports to encourage individuals to act against corruption by reporting on it. In addition, this component collated all relevant information gleaned from legal-aid beneficiaries reporting corruption and streamlined it into data to support CSO input into the whistleblower-protection legislation discussed above. For additional details on the Evaluation Purpose and Evaluation Questions please see Annex 1 – the Evaluation Scope of Work. 4 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 2. ACCOUNT BACKGROUND The four-year USAID activity ACCOUNT began implementation in July 2015 as a follow-on activity, building on the successes of the previous USAID anti-corruption intervention. The overall objective of the follow-on intervention was to create an environment that increases civil society participation and reforms through strong collaboration and cooperation in anti-corruption initiatives. Furthermore, the intervention strove to align its priorities and activities in accordance with the anti-corruption obligations required for Euro-Atlantic integration. ACCOUNT envisioned providing sub-awards for up to 50 ACCOUNT CSO members to undertake public advocacy and awareness-raising activities. Through this sub-award mechanism, ACCOUNT was to support the investigative research and production of more than 100 media and CSO reports intended to stimulate initiatives on public anti-corruption policies that respond to concerns raised by citizens. ACCOUNT envisioned ensuring evidence-based research, civic monitoring, and investigative reporting on corruption cases in five sectors: 1) public procurement, 2) whistleblower protection, 3) education, 4) health, and 5) public employment. Furthermore, ACCOUNT also planned on Network members engaging at the local level to develop and/or implement sound anti-corruption strategies (at the community and cantonal levels). Additional effort was invested in increasing the volume, quality, and outreach of sector-based monitoring reports and ensuring they were accompanied by appropriate and timely investigative media reports to raise public awareness and stimulate civic participation. The Activity’s sub-awards/grants model was intended to stimulate and support the ACCOUNT Network members, including 100 CSOs through the mechanisms described below. • Sub-awards are provided to ACCOUNT Network members engaged under the relevant Sector Group to lead efforts in the improvement of anti-corruption regulation and strategies, utilizing evidence-based intervention approaches. • Sub-grants are provided for public advocacy to lead organizations in the Sector Groups and their partners (five grants of approximately $15,000 each per year) as well as to the lead organizations for the anti￾corruption strategies in the project municipalities and cantons (7 grants of approximately $5,000 each per year). • Sub-awards are also implemented in the form of contracts with non-governmental organizations (NGOs) and media outlets or journalists involved in the following project components: “Monitoring and reporting on corruption in the selected sectors” and “Public awareness through strategically targeted media campaigns.” The monitoring component involves five sectoral Monitoring Teams, each comprised of up to five CSOs. Up to $8,000 are awarded to these teams annually as a contribution toward the costs of monitoring and reporting. • To increase the volume, quality, number, and outreach of investigative reports, ACCOUNT is supporting circa 120 investigative stories over the four project years (all media formats), equivalent to $300,000.1 ACCOUNT intended to implement an operational model based on the elements presented below. 1 The sums expressed here are from the original Program Description and prior to the 6 Award Modifications. 5 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV • Strengthening civil society grassroots organizations. Capacity building was used as an essential component of the bottom-up approach and involved awareness raising, training, analysis, and setting criteria for activities. ACCOUNT expected that this method would contribute to the goal of fostering civil sector sustainability. • Research/surveys, monitoring, and reporting. This was envisaged as continuous and cumulative work based on empirical data, fact gathering, public reports/surveys on sector-based corruption, and “human features stories.” This was expected to ensure public awareness of corruption and its consequences, including its impact on social and economic development. • Anti-corruption initiatives and work on corruption cases. It was expected that ACCOUNT would engage in advocacy initiatives for anti-corruption policies––backed up by actual cases and examples and the experiences of whistleblowers and other individuals reporting corruption––to demonstrate how to prevent or suppress corruption in practice. ACCOUNT aimed to achieve four main results as expressed in the ACCOUNT monitoring, evaluation, and learning (MEL) Plan: 1) Increased CSOs’ involvement and input regarding anticorruption legislation. 2) Strengthened CSOs’ capacity for in-depth monitoring, data processing, synthesis and analysis of corruption in selected sectors. 3) Strengthened public awareness of the problem of corruption and public engagement in anti-corruption initiatives. 4) An appropriate and effective legal aid model established to protect individuals reporting corruption and whistleblowers established. The ACCOUNT assistance is organized into four main components (public advocacy, monitoring and reporting, public awareness, and free legal assistance). Each of the components focused on one of the main results; however, as a whole they complement and reinforce each other to achieve the outlined results. An illustrative presentation of the Activity results and sub-results, as well as the corresponding components, is presented in Figure 1. 6 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV I Activity Purpose: An environment that increases civil society participation in anti-corruption initiatives and reforms through strong collaboration and cooperation Activity Sub￾purpose 1: Increased CSOs involvement and input regarding anticorruption legislation Activity Sub￾purpose 2: Strengthened CSOs’ capacity for in-depth monitoring, data processing, synthesis and analysis of corruption in selected sectors Activity Sub￾purpose 3: Strengthened public awareness of the problem of corruption and public engagement in anti-corruption initiatives Activity Sub￾purpose 4: An appropriate and effective legal-aid model to protect individuals reporting corruption and whistleblowers Component 1: Public Advocacy on selected anti￾corruption legislation Component 2: Monitoring and Reporting on corruption in the selected sectors Component 4: Free Legal Assistance to victims of corruption and whistleblowers Component 3: Public Awareness through strategically targeted media campaigns Anticipated Result 1.1: CSO recommendations to improve implementation of the state￾level Whistleblower Protection Law taken into consideration by government Anticipated Result 1.2: CSO inputs for passage of the entity-level whistleblower protection laws taken into consideration by government Anticipated Result 1.3: CSO recommendations to improve implementation of the Public Procurement Law taken into consideration by government Anticipated Result 2.1: CSOs effectively analyze, monitor, and publish reports on corruption in specified sectors Anticipated Result 2.2: Increased recognition of local CSOs within their communities Anticipated Result 3.1: Increased media reporting on corruption Anticipated Result 3.2: Increased public awareness of corruption cases in the specified sectors Anticipated Result 4.1: Victims of corruption and whistleblowers to receive free legal aid to make them more secure about reporting corruption Figure 1. ACCOUNT Results Framework and Components 7 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Within Component 1 and Component 2, jointly lead by CRMA and INFOHOUSE, each of which is a lead of one component but has also assumed leading roles within specific sectors of each component. In these two components, ACCOUNT worked with the Network organizations to improve anti-corruption strategies and practices in the five selected sectors. Within these two components, ACCOUNT has supported 292 CSOs through the sub-award mechanisms related to: 1) sector-area advocacy; and 2) establishment of watchdog teams and their monitoring and analysis. For both mechanisms, ACCOUNT has selected sector/monitoring team leads or members for each of these groups. CSOs that were supported through the various rounds of small-grant awards assumed multiple roles throughout the implementation process, including participation in multiple sectors. In addition to the five sectors, ACCOUNT also supported CSOs’ work on the local level (municipal and cantonal levels) to engage in cooperation with the government in developing or implementing Anti-corruption Action Plans, assisting 29 municipalities and 8 cantons over the three-year period. Annex II illustrates the sub-award mechanisms and grantees per sector and role. Jointly these activities were to contribute to CSOs’ involvement in the governmental decision-making process by providing input into anti-corruption legislation/regulation. Based on the ACCOUNT reports, up to 55% of their proposed recommendations for policy proposals have been adopted by the various levels of government(s), from municipal to state level, with whom they have established cooperation by signing Memoranda of Understanding (MOUs). ACCOUNT MEL Plan indicator data are presented in Annex III. Within Component 3, primarily led by CRMA, ACCOUNT supported the production of investigative reporting on corruption issues primarily through Zurnal.info, which over the past three years has produced 1,034 investigative reports, including 20 stories in video format published on YouTube, such as AFERA TV Zurnal. During this period, ACCOUNT also provided support though small-grant awards and fixed-amount awards to 14 independent media outlets, i.e., the ACCOUNT Media Pool. In the third year of implementation, this component was reinforced through the fourth modification of the Award, which added additional funding ($850,000) to support independent media outlets with a focus on those from the Republika Srpska. CRMA led all the work related to the investigative reporting of Zurnal.info and the Media Pool. In addition to the investigative reporting, ACCOUNT also developed the website Interview.ba, led by INFOHOUSE, which published 393 interviews with a wide array of individuals who were willing to speak about various issues concerning the prevention of and the fight against corruption and about themes based on the Sector Groups’ and Monitoring Team findings. Finally, within Component 4, led by INFOHOUSE, ACCOUNT provided legal-aid assistance to 49 individuals who reported corruption to ACCOUNT. The majority of these cases were reported by men. In addition, an online registry, Besplatna Pravna Pomoc – bpp.ba (Free Legal Aid), has been established in an effort of coordinate between the various legal-aid providers, as well as providing individuals with access to information about their rights and protection mechanisms available to those reporting corruption. This component was reinforced by and reinforced the activities under Components 1 and 2, with the Legal Aid Team participating in and providing trainings on legal-aid provision as well as inputs to the relevant whistleblower-protection legislation. Furthermore, Award Modification 5, introduced a reinforcement to this component ($100,000), by extending it to providing Free Legal Aid assistance to Women Candidates in the 2018 election, effective as of June 2018, an activity that has just begun. The total amount of the award after modifications and the two program extensions is $ 3,449,474. 2 ACCOUNT has granted sub-awards to a total of 31 CSOs over the three-year period; however, these 29 are within the Components 1 and 2 sub-award mechanisms. In the third year of implementation, an additional sub-award mechanism was introduced for legal-aid provision to women candidates in the 2018 general elections, through which an additional two CSOs were supported, although those are not considered here. 8 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 3. EVALUATION METHODS AND LIMITATIONS 3.1. METHODOLOGY AND DATA COLLECTION MEASURE-BiH designed a mixed-methods approach for the ACCOUNT evaluation. The evaluation team relied on a mix of qualitative and quantitative data to inform and answer each evaluation question and develop findings, conclusions, and recommendations. Our main data sources included the following: 1. ACCOUNT Program documents and ACCOUNT products (e.g., investigative reports) 2. Semi-structured key informant interviews 3. Online mini surveys with ACCOUNT Network CSOs and Media Pool 4. Online mini-survey with educational institutions/teaching personnel 5. Panel discussion with media representatives (Media Pool and Interview.ba journalist) 6. National Survey of Citizens’ Perceptions data 2015–2018 7. Evaluation of the quality of investigative reports based on a sample of reports produced within ACCOUNT 8. Secondary/administrative data (e.g., Higher Judicial and Prosecutorial Council administrative data) The data from the above-listed sources were triangulated wherever possible to address the same evaluation questions from multiple perspectives. Comparing and contrasting data enabled us to gain a more complete understanding of each issue and provide greater confidence in the findings. The evaluation team conducted a total of 39 interviews and a panel discussion with media representatives, with a total of 65 individuals participating in the data collection process, from the nine (9) stakeholder groups illustrated in Figure 2 (a detailed list of KIs is available in Annex II). Furthermore, in addition to the stakeholder groups, the team tried to ensure sufficient representation among the stakeholders linked to specific ACCOUNT sector areas. Of the KIIs, the following numbers are affiliated with: public procurement – 10 KIIs; whistleblower protection – 15 KIIs; education – 11 KIIs; health – 10 KIIs; public employment – 9 KIIs; and local-level anti-corruption action plans – 13 KIIs. Figure 2. Distribution of Key Informants Per Stakeholder Group Stakeholder Group Number of Interviews Number of KIs USAID/BiH 1 1 BiH Government/Public Institutions 10 18 USAID/BiH Implementing Partners 2 5 International and Bilateral Donors/Organizations 3 3 Network Member CSOs 12 17 Whistleblowers 4 4 Municipalities 5 8 Media Representatives Interview and Panel Discussion 1 8 Other/Expert 1 1 TOTAL 39 65 9 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV For the online survey, we used e-mail addresses provided by the ACCOUNT implementing partners for the ACCOUNT CSOs, Media Pool, and Interview.ba members/journalist, with an overall response rate across these three categories of 34 percent, the highest response rate occurring among the teachers as illustrated in Figure 3. Figure 3. Statistics for Online Surveys Administered to Stakeholder Groups Online surveys with ACCOUNT beneficiaries # of Addresses Contacted # of Respondents Response Rate ACCOUNT Network CSOs 100 29 29% Partner Media Outlets and Journalists 33 12 36% Schools/Teachers 75 30 40% TOTAL 208 71 34% In addition, we relied on the National Survey of Citizens’ Perceptions findings (2015–2018) to extract citizens’ responses relevant to all five Evaluation Questions. The complete NSCP reports, methodology, and data are available on the MEASURE-BiH website. Figure 4 presents details on the overall sample size of the survey by year. Figure 4. National Survey of Citizens’ Perceptions 2015–2018 Sample Size National Survey of Citizens' Perceptions Nationally representative sample of civilian, non-institutionalized 18+ BiH residents 2015 2016 2017 2018 2,906 3,004 3,084 3,024 To evaluate the investigative reporting relevant for Evaluation Question 3, a random sample of the reports produced by Zurnal.info and the Media Pool members was created, extracting a total of 30 investigative reports produced within ACCOUNT, as illustrated in Figure 5. Each of these reports was evaluated against nine (9) categories of quality, including: 1) Fairness; 2) Accuracy; 3) Clarity; 4) Public Interest; 5) Respect; 6) Ethical Conduct; 7) Rights; 8) Advocacy; and 9) Investigative Approach. For a detailed methodology and a description of each of the categories, including the results, please see Annex IV on the Quality of Investigative Reports Evaluation. 10 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 5. Sampling of ACCOUNT Investigative Reports ACCOUNT Investigative Reports Total population Sample size Written reports Video Total Zurnal.info 1,034 15 1 16 Media Pool 217 13 1 14 TOTAL 1,251 28 2 30 The evaluation team carefully examined all the data sources and completed a coding process for key informant interviews and the panel discussion for common themes and responses. Data sources were triangulated wherever feasible to provide as comprehensive a picture as possible on each Evaluation Question. The methodology and sources used for each Evaluation Question are presented in Figure 6, the Evaluation Matrix, while the main data collection instruments are presented in Annex VII. Figure 6. Evaluation Matrix EVALUATION QUESTIONS DATA SOURCES METHODOLOGY 1. To what extent has ACCOUNT increased CSO involvement and input regarding anti-corruption legislation/regulation? 1) ACCOUNT reports 2) 39 semi-structured KIIs of ACCOUNT implementers and beneficiaries (CSO and Government/Public institutions, including municipalities, and donors) 3) Online mini-survey of ACCOUNT CSO Network members and Media Pool and Teachers. 4) National Survey of Citizens’ Perceptions 2015–2018 Mixed-method triangulation based on: ▪ Desk research of ACCOUNT reports ▪ Qualitative analysis of semi￾structured KIIs ▪ Descriptive analysis of online survey findings ▪ Descriptive analysis of NSCP data 2. What are ACCOUNT’s major achievements in selected five sector areas (public procurement, whistle-blower protection, education, health and public employment) and on the local level? 3. To what extent has ACCOUNT increased the quality and quantity of investigative journalism targeting public corruption in BiH? 1) ACCOUNT Reports 2) 39 semi-structured KIIs, with all stakeholder groups 3) Online mini-survey of ACCOUNT Network members and Media Pool 4) Panel discussion with 7 media representatives 5) NSCP data 2015–2018 6) Evaluation of the quality of investigative reporting. Mixed-method triangulation based on: ▪ Desk research of ACCOUNT reports ▪ Qualitative analysis of semi￾structured KIIs and Panel Discussion KIs ▪ Descriptive analysis of online mini￾survey data ▪ Descriptive analysis of NSCP 2015–2018 findings ▪ Descriptive analysis of findings from the evaluation of the quality of investigative reporting. 4. To what extent have interventions under ACCOUNT influenced public awareness on corruption? 11 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 5. Is the ACCOUNT legal-aid model to protect individuals reporting corruption and whistleblowers perceived as effective by ACCOUNT beneficiaries (e.g., legal-aid recipients, beneficiaries of joint registry, and beneficiaries of the ACCOUNT legal team)? 1) ACCOUNT Reports 2) 39 semi-structured KIIs, with all stakeholder groups (including whistleblowers assisted through legal aid) 3) Online mini-survey of ACCOUNT Network members and Media Pool 4) NSCP data 2015–2018 5) Secondary data (administrative data from the High Judicial and Prosecutorial Council) Mixed-method triangulation based on: ▪ Desk research of ACCOUNT reports ▪ Qualitative analysis of semi￾structured KIIs ▪ Descriptive analysis of all online survey findings ▪ Descriptive analysis of secondary data. 3.2. EVALUATION LIMITATIONS The evaluation was completed in a little over one-month period, including all data collection activities and data analysis. Below we discuss limitations of this evaluation. Limitation 1: Response bias as a majority of evaluation participants had direct interaction with or benefitted from ACCOUNT intervention and activities. Most of the key informants and online survey participants are either Activity beneficiaries or had direct interaction with the Activity and may overstate the positive effects of the ACCOUNT intervention and understate the negative effects. The evaluation team tried to mitigate these issues by probing in detail about information provided by respondents to verify their claims and by reaching a wider audience through two online surveys to estimate the veracity of those claims. Limitation 2: Social desirability bias. Issues relevant to corruption in the public sector are difficult to measure and, in most instances, rely on perception data, which is widely distributed and may not represent actual conditions. Respondents’ views may taint the general perceptions and cause them to provide socially desirable answers. The team mitigated response bias to the extent possible by drawing on multiple sources of information for each Evaluation Question, by requesting examples from KIs during the interview process, by broad coverage of the ACCOUNT stakeholders in KIIs/survey, and by including information relevant to the Evaluation Questions from the nationally representative National Survey of Citizens’ Perceptions collected annually since 2015. Limitation 3: Recall bias, due to selected exposure to certain segments of the ACCOUNT intervention and the intensity thereof. Respondents may have participated in a select and narrow set of activities, and their responses may be limited to only certain segments of interest to the evaluation. In addition, they may not be able to differentiate between other donor interventions or contributions to the same objectives that the ACCOUNT intervention addressed. The team did not map other donor interventions in the area of anti-corruption and cannot isolate ACCOUNT’s contribution to all five of the Evaluation Questions. Limitation 4: No baseline data. The ACCOUNT MEL Plan does not have baseline data relevant to Evaluation Questions 1, 3, and 4 (CSOs’ level and amount of input related to anti-corruption legislation/regulation; data on quality or quantity of media reporting, 12 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV neither in general nor among beneficiaries; and citizens’ awareness of corruption). In order to alleviate this challenge, the team consulted available documentation and KIIs to extract perceptions and provide descriptions of the baseline data as well as relying on other sources of information (surveys and available reports produced by third parties). Limitations 5: Limited sample size for the evaluation of the quality of investigative reporting. Random sampling and the timeframe for the evaluation allowed the team to examine only a small portion of the investigative reports produced within ACCOUNT. This may not entirely capture the depth of quality of all investigative reporting but rather the average of produced reports related to quantity. The evaluation team tried to mitigate these limitations by including two experts who separately evaluated the content of the same sampled reports and then by cross-referencing their analysis with inputs received from the KIs, the panel discussion, and the online surveys to provide broader explanations. Limitations 6: Limited response rate for online survey participants. Online survey participants are to a large extent ACCOUNT’s most active CSOs and Media beneficiaries. In addition to the small sample size in reaching the overall ACCOUNT Network, the survey results also relate to the first Limitation 1, discussed above. 13 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 4. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4.1. EVALUATION QUESTION 1:TO WHAT EXTENT HAS ACCOUNT INCREASED CSO INVOLVEMENT AND INPUT IN DETERMINING ANTI-CORRUPTION LEGISLATION/ REGULATION? 4.1.1 FINDINGS Finding 1: According to KIIs and Activity reports, CSO involvement in anti-corruption efforts is higher today than three years ago. The ACCOUNT Network is familiar to all KIs, but a relatively small number of CSOs are recognized as experts in the field and accepted by governments in cooperation. Most KIs (19) confirm that more CSOs are involved in the field of anti-corruption today than 3 years ago. They also recognize the ACCOUNT Network as an important player in the field, as it has facilitated involvement of numerous CSOs. According to the Activity documentation for the follow-on phase, the Network membership was strategically downsized from 169 in 2015 to 114 members in 2017, where a great majority are CSO members (Figure 7). The downsizing was conducted in order to filter out those that can provide quality input and, thus, increase the recognition of CSOs and their expert involvement in the field of anti-corruption. The majority of KIs (21 KIs) think that only a small number of CSOs have sufficient expertise to provide quality technical assistance in the anti-corruption sector; the KIs from public institutions recognize ACCOUNT but often not individual CSOs as they state that their primary contacts were implementing partners (either CRMA or INFOHOUSE). Figure 7. ACCOUNT Network Membership in 2015 and 2017 ACCOUNT Network Membership Type 2015 2017 CSOs 137 100 Individuals 22 9 Public Institutions 8 4 Other 2 1 All Network Members 169 114 Note: 2015 figures related to the Account network membership are provided by the IP. The documents made available to the evaluation team did not contain data on the breakdown of the ACCOUNT network membership, Membership in the Network opened a new field of action for some CSOs; 12 out of 29 surveyed CSOs became involved in anti-corruption projects for the first time by joining ACCOUNT. The CSOs’ involvement intensified mainly thanks to the activism of ACCOUNT sub-grantees as 13 KIs (CSOs) confirm that sub-award was critical to their engagement in anti-corruption efforts; they particularly appreciate sector-thematic capacity building provided in the Activity. In total, in the ACCOUNT follow-on phase, 31 CSOs (29 sub-grantees) received U.S. Government assistance for engaging in advocacy and monitoring interventions. 14 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 8. Geographic Distribution of Sub-grantees in the ACCOUNT Follow-on Phase Geographic distribution of sub-grantees contributed to the visibility of the Network’s actions all over the country (Figure 8). In addition, sub-grantees were in touch with many institutions in the five target sectors: public procurement (PP), whistleblower protection (WBP), education (EDU), health (HEALTH), and public employment (EMPL). The number of CSOs involved in each sector was in the range of 30 (PP sector) to 62 (EDU sector) while each sub-grantee participated in 2 sector groups in average. Finding 2: Based on KII’s, the CSO survey, and Activity documentation, quality of CSO participation in policy-making processes was meaningful and introduced into the design of anti-corruption legislation and regulation, in particular in the field of public procurement and whistleblower protection. Public perception about CSOs’ influence on decision-making in BiH in general is not optimistic. According to the NSCP data for the past four years (2015–2018), a major portion of the BiH public thinks that CSOs mainly advocate for issues of interest only to themselves (31% in 2015; 27% in 2018). A segment of the public thinks that CSOs have no influence over governments at any level (33% in 2015; 27% in 2018), while another segment thinks that CSOs have some influence at some government levels (33% in 2015; 43% in 2018). In the field of anti-corruption, the ACCOUNT Network succeeded in providing meaningful inputs for the new legislation in the field of anti-corruption (primarily on the BiH Law on PP; BiH and entity Laws on WBP) and relevant technical assistance to governmental institutions on design of rule books and anti-corruption (AC) action plans. Governments decided to accept only some of the inputs suggested by CSOs. Referring to activities of the ACCOUNT Network, more than one-half of KIs (20) claim that CSOs today play a more prominent role in anti￾corruption and that governments take more of the CSOs’ inputs into account in decision-making than was the case three years ago. 15 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV All KIs from public institutions expressed very positive impressions about ACCOUNT, and 7 (of 14) of them explicitly articulated the need for further similar assistance, recognizing that governmental capacities were limited and that achievements had been more effective when multi-stakeholder actions accompanied the process. This was stressed by KIs from APIK, BiH Agency for PP, and both entity Ministries of Justice. The CSO online survey confirms that CSOs are more frequently invited to cooperate with governments; 19 of 29 CSOs strongly agree and somewhat agree that their organizations frequently participate in public consultations organized by government on the procedure of adopting laws/regulations, and 17 of them are more frequently invited for collaboration with governmental institutions to participate in the development of new laws/regulations/procedures to reduce corruption (e.g., in joint projects/initiatives, working groups formed by Ministries or Parliaments). In relation to the CSOs’ influence on new legislative solutions within the ACCOUNT follow-on phase, Network members register achievements in several Activity indicators. Based on the M&E Plan and annual reports, ACCOUNT achieved the following (Figure 9): - CSOs submitted 59 public policies, of which 33 were accepted and 26 are pending (ref. indicator: Number of public policy proposals submitted to executive governments and parliaments with citizens’ input). This achievement is below the target of 65 submitted policies. However, ACCOUNT is over-performing with respect to adopted policies. Over the last three years 33 policies have been accepted, exceeding the target of 27. To reach the cumulative four-year target, CSOs should submit additional 46 policy proposals, with additional 14 to be adopted. - The Number of policy recommendations provided by Sector Groups (SGs) and Monitoring Teams (MTs) also exceeded targets (76 recommendations have been provided compared to a target of 60 to date). This was driven in part by the large number of recommendations in WBP and PP in the Activity’s first year. The number of policy recommendations in other sectors has either been at or below target over the past 3 years. - In the first three years of the Activity, 13 monitoring reports on sector-based corruption were prepared. This falls below the target of 15 reports (annual target is 5 reports with 1 report per sector). More than targeted number of reports were issued in Year 2 and Year 3, as a result of over performance in the PPL and HEALTH sectors. Figure 9. ACCOUNT Progress in Achieving Targets for Indicators in the M&E Plan Note: Due to revision of the Monitoring, Evaluation and Learning plan, and changes in the definitions of indicators, the Activity exceeded targets in Year 1. Indicator (M&E Plan) 4-year target Actuals Progress 105 59 47 acc. + 58 pend. 33 acc. + 26 pend. No. of policy recommendations provided by SGs and MTs 80 76 No. of monitoring reports on sector-based corruption 20 13 No. of public policy proposals submitted to executive governments and parliaments with citizens' input 16 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Finding 3: Based on KIIs, the CSO survey, and Activity documentation, ACCOUNT CSOs’ expertise has improved over the past few years. Due to a conscious shrinking in the size of the Network and increased focus on the quality of CSO input, only a fraction of Network CSOs have been substantially involved in anti-corruption activities. Of the total Network CSO members (100), 29 CSOs were involved in SGs/ MTs, either as members or as leads, and 16 CSOs took up the lead position in SGs or MTs, at least in one cycle (Figure 10). The Network CSO members improved their capacities for technical-assistance provision, advocacy initiatives, and monitoring in the field of anti-corruption as stated by 10 KIs (CSO representatives). Twelve KIs (of which 9 are CSO representatives) think that cooperation and coordination among ACCOUNT member CSOs has been constructive––SG and MT lead organizations accepted their roles and performed them with enthusiasm, while most of them also took up membership in other sectors’ SGs and MTs. Figure 10. ACCOUNT Network CSOs: Figure 10. ACCOUNT Network CSOs: Total membership, members, and leads of SGs and MTs Looking at CSO participation in various ACCOUNT activities, the online CSO survey indicates that of 29 CSOs, an average of 15 CSOs participated in general sector-group activity per sector, 12 CSOs participated in sector￾group meetings, 8 provided input to governments through meetings, 8 participated in monitoring activities, 12 participated in advocacy activities, while 8 provided technical assistance to governments. According to Activity documentation, each sub-grantee participated in two sectors on average and in different roles. SG leads for HEALTH (ICVA), EMPL (Helsinki Committee for Human Rights Bijeljina), and EDU (CROA) were stable in at least two years. Similarly, the role of the MT lead was consistently taken up by the same organizations in the WBP (CRCD) and PP (Justicia) in all three years. The other lead positions changed over time, depending on the interests of sub-grantees and their respective capacities to carry out activities. 17 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Due to multiple roles and parallel activities, in 7 of 12 KII’s with Network CSO members, CSOs were not able to make a clear distinction between ACCOUNT Activity structures (SGs/ MTs), implementation mechanisms (types of sub-awards), and the four components (public advocacy, monitoring and reporting, public awareness, and free legal assistance). While all 12 sub-grantees interviewed expressed a clear focus on deliverables and commitment to fulfilment of their contractual obligations, they recognized that it was not possible to address all sectors with the same level of effort. One KI shared his/her own position that ACCOUNT was striving for mobilization of as many CSOs as possible to ensure collaborative work on anti-corruption, which has not turned out to be the most effective approach in practice. The KI thinks that an alternative would be to work with a smaller number of CSOs that provide better quality work. 4.1.2. CONCLUSIONS Based on Findings 1 and 3 above, ACCOUNT integrated a significant number of CSOs and stakeholders that previously were not focused on corruption into the Network to address corruption issues. Downsizing the Network to filter out CSOs with relevant sector-area expertise and capacity, which in general is scarce, was a positive strategy for profiling the Network as a credible partner on corruption-related issues and policies. The Network is now recognized by public institutions as such. Based on the positive reputation of ACCOUNT as a brand, the CSOs’ implementing activities benefit from membership in the Network by gaining higher credibility among governmental counterparts on the individual level. Furthermore, based on Findings 2 and 3, ACCOUNT CSOs’ involvement in providing policy recommendations and technical assistance to governments for implementation of anti-corruption policies was noted to be higher than previously. Advocacy processes have also been more intensive than before the ACCOUNT follow-on phase was initiated. CSOs’ advocacy efforts in the five target sectors may be viewed on two levels, as described below. 1. CSOs provided a significant contribution to the state and entity governments in the sectors of WBP and PP in support of improving the design of new or existing legislation. Successful adoption of their recommendations and policies indicate the good-quality input provided by ACCOUNT and should be felt by a range of institutions/stakeholders country-wide. 2. In the sectors of HEALTH, EMPL, and EDU, as advocacy processes were conducted with the objective of rolling out new by-laws and policies among institutions at all levels, but mainly at the local level, this led to very time- and resource-intensive efforts. ACCOUNT was striving for the broadest outreach possible, with a higher number of targeted stakeholders. However, at the same time this expansion limited efforts to stay more focused on monitoring or technical assistance for implementation of adopted policies and tailoring the practice templates to the specific needs of each institution. “I think that the quantity of CSOs is not the right approach. I think that ACCOUNT aimed to engage as many CSOs as possible for joint work. However, my view is that in practice this turned out not to be very important. It may be more important to have fewer CSOs who are capable of delivering better quality.” -Interview statement by an ACCOUNT sub-grantee KI 18 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 4.2. EVALUATION QUESTION 2: WHAT ARE ACCOUNT’S MAJOR ACHIEVEMENTS IN PUBLIC PROCUREMENT, WHISTLE-BLOWER PROTECTION, EDUCATION, HEALTH,AND PUBLIC EMPLOYMENT? 4.2.1 FINDINGS Finding 4: Achievements per sector are difficult to assess as the ACCOUNT Program Description did not define clear and measurable results for each sector and Annual Reports document progress along detailed activities instead of goals and objectives. Additionally, the ACCOUNT Results Framework structure and terminology are not consistent across Activity documentation and reporting periods. In efforts to evaluate achievements by sector, the evaluation team encountered difficulties in determining achievements, as the ACCOUNT Program Description (PD) did not include anticipated results for all sectors. In the PD and the M&E Plan, the sectors of WBP and PP were accentuated in Strategic Objective 1 and the anticipated results under Component 1 (C1) see Figure 11. The remaining three sectors (HEALTH, EMPL, EDU) were addressed by advocacy activities under C1, but no results were anticipated in the PD. According to the implementing partners, this was intentional to allow for flexibility in the implementation, thus deliverables were defined on an annual basis in the annual work plans. Furthermore, in Component 2, the five target sectors were addressed through the monitoring activity (Activity 2.2.), while the local-level activities were addressed by Activity 2.3. Anticipated results did not distinguish any of the target sectors or local-level activities as results related to CSOs’ involvement in the sectors themselves. Here, deliverables were also defined on an annual basis (Figure 11). From this perspective, it is difficult to assess achievements per sector, given that they were subject to annual modifications, with the exception of achievements relating to involvement of CSOs in these sectors (addressed in EQ1 findings). Figure 11. ACCOUNT Results Framework 19 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV While the more open and flexible technical approach of having no pre-determined expected results contributed to the effectiveness of ACCOUNT (ref. the positive assessment of the demand-driven approach described in Finding 7), it made the reporting system very detailed and complicated for analysis of individual sectors. Since deliverables were defined on an annual basis, the annual reports document progress primarily along activities and per component, without any reflection on goals and objectives. Moreover, the Results Framework structure and terminology are not consistent across Activity documentation and reporting periods. For example, the Y3 report and the Y4 annual work plan introduce new language for 3 activities in C1 “intermediate results” and “key results” (Activities 1.1, 1.3, and 1.5) that did not previously exist. Those same result levels are not introduced for other activities in the same Component or for other Components. Based on the implementing partners’ explanation, as the activities are developing and results are happening, this modification was introduced in order to show the incoming results. Finding 5: ACCOUNT facilitated the implementation of numerous activities and engaged a large number of stakeholders. The program design established a very complex structure reflecting itself across program components, implementation responsibilities, and actors involved, making the interconnections difficult to follow. Furthermore, in-depth work was not possible in all sectors, due to diverse foci and the expansion of coverage across governance levels and institutions. The Program Description, the M&E Plan, yearly work plans, and reports register numerous activities and stakeholders involved in ACCOUNT. The implementing partners shared responsibilities for implementation of components and key activities in accordance with the organizational structure defined in the M&E Plan, Chapter 1.3. However, the implementing partners did not implement all activities directly, but through other organizations and Activity structures, primarily through sub-awards to sector groups (SGs), monitoring teams (MTs), and media outlets. Therefore, the lines of coordination and communication were very complex (Figure 12). These complexities were successfully overcome in practice, in terms of efficient coordination and delivery of outputs, but they had an influence on the effectiveness of activities due to the factors outlined below. - Insufficient synergies between Components. The implementing organizations/ structures remained rather compartmentalized within their scope of work. Despite the several cross-sector activities (especially in organizing joint events for HEALTH and EDU), there were not enough opportunities for communication and exchange of ideas among implementing organizations that would have facilitated strong synergies between Components. For example, implementing partners clearly communicated the intention to link the work of MTs with SGs by ensuring that monitoring reports (MTs’ deliverables) feed into the advocacy activities of SGs. According to KIs (7 of 12 Network CSO members), CSOs focused on their own deliverables, in accordance with the sub-grant agreements, without much insight into other Activity components and into the larger picture. - Effects dispersed across numerous stakeholders. While striving to roll out activities over the broadest possible scale, numerous target groups––institutions, individuals, and structures––were involved in and affected by activities. The resulting institutional map augments the complexities of the initial organizational structure from the M&E Plan, while the changes induced at the level of target groups and beneficiaries are dispersed. 20 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 12. ACCOUNT Institutional Map Finding 6: Progress has been made in all sectors, as confirmed by KII’s, the online survey, and Activity reports. However, results are not comparable in certain instances due to overlapping activities across sectors. In relation to advocacy activities, sector-specific achievements have been evident from Activity documentation. ACCOUNT made significant progress at: - State and entity level. Law-making activities (design of new and amendments to existing laws) in the two sectors (WBP and PP). - Local level. Roll out of by-laws (rulebooks) in the three sectors (HEALTH3, EMPL, EDU) and technical assistance for developing AC plans in municipalities/ cities and cantons. Figure 13 presents the most important achievements per sector and shows progress against the baseline described in the Program Description. 3 Activities in the HEALTH sector were implemented at the cantonal level and were classified as local-level activities in the Project reports. 21 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 13. ACCOUNT Achievements in Component 1, per sector Sector Baseline (Component 1) Achievements (Component 1) Whistleblower Protection Law (WBPL) State-level WBP Law passed in 2013, but no experience with implementation. No entity WBP Laws: Federation of Bosnia and Herzegovina (FBiH) drafted the WBP Law; RS no activity. Signed MoUs with 4 institutions; 5 amendments to the BiH WBPL adopted by the BiH House of Representatives (HoR); 14 (of 30) recommendations accepted in the adopted RS WBP Law; ACCOUNT involved in development of the first draft of the FBiH WBP Law, adopted by FBiH HoR Public Procurement Law (PPL) ACCOUNT Phase 1 drafted 18 amendments to the former PP Law and advocated for their adoption in 2012–2014. The new PP Law adopted in 2014; ACCOUNT organized training for 12 CSOs to build capacity for monitoring of PP Law implementation. Monitoring report published. MoU signed with PP Agency; 18 amendments to PP Law jointly developed by ACCOUNT and counterparts; Working Group (WG) adopted recommendations on 11 articles (out of 16). ACCOUNT’s recommendations were officially included in the Draft Law on Changes of PP, which was submitted to the Parliamentary procedure. HEALTH Drafted a model anti-corruption Rulebook for healthcare institutions. Around 20 healthcare institutions in the Eastern Herzegovina, Una￾Sana Canton and Doboj region signed MoUs, committing to introduce the Rulebook. Only a few healthcare institutions adopted it and developed internal procedures. MoU signed with AC Team of Sarajevo Canton for HEALTH and EDU and tripartite MoU between ACCOUNT, APIK, and Zenica-Doboj (ZeDo) AC-Team. Anti-corruption rulebook model rolled out to 31 institutions in 3 cantons, out of which 11 adopted. EMPL No public institution had procedures in place to ensure transparent recruitment. Some CSOs had experience with advocacy campaigns (e.g., Justicia ran a successful campaign to reveal a series of irregularities in employment in the local Health Centre). Developed template rulebooks for employment for 2 municipalities (accepted) and 3 cantons (pending). 22 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV EDU Through Education (EDU) SG, ACCOUNT worked on introduction of anti-corruption as a subject in primary/ high schools and universities. In the European Union funded project (in line with ACCOUNT Phase 1), implementing partners provided support to the state Agency APOSO and prepared draft curricula for anti￾corruption education. Education curricula rolled out to 48 primary schools and 24 high schools and involved more than 1,000 students. Organized 3 youth forums, 34 workshops for students in primary/high schools and training of teachers. Local-level activities Successful cooperation established between ACCOUNT and 6 municipalities in developing anti-corruption plans. Conducted monitoring of the municipalities’ anti-corruption procedures by watchdog organizations, in line with state-level AC Strategy (2014–2015). Signed MoUs with 29 municipalities and supported development of local AC strategies, most of which were adopted. Provided training on development of integrity plans in 1,027 public institutions and 64 municipalities in the RS; INFOHOUSE involved in the review process of these plans, pending their adoption. Supported development of AC strategies in 2 cantons (Sarajevo; ZeDo). In relation to the indicators in the Activity documentation, based on data segregated by sector, ACCOUNT shows progress against targets in each sector (Figure 14), as outlined below. • Number of policy proposals. The most significant achievement is noted in the WBP sector, where the RS Law on WBP was adopted and two more proposals were submitted and are still pending. The local-level activities and the HEALTH sector had the highest number of adopted and pending policies. However, these achievements are not yet on target. • Number of policy recommendations. The WBP sector was the most active, with the highest number of submitted recommendations for legislative changes, and that is a significant overachievement in comparison to the target. In the first three years, targets have been met in PP and EMPL, but not in HEALTH and EDU. • Number of monitoring reports. Monitoring teams in PP and HEALTH produced at least one monitoring report per year, in accordance with the targets, while underachievement has been registered in WBP, EMPL, and EDU. 23 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 14. ACCOUNT Progress in Achieving Targets for Indicators in the M&E Plan, per sector About one-half of KIs recognize that ACCOUNT made achievements in all five sectors. Cumulative results of CSO and Media online surveys indicate respondents’ perception that the most significant achievements were made in local-level activities (50% support this statement) and WBP (48%), while the lowest score was given to the PP sector (34% think that the most significant achievements were made in PP). Some achievements cannot be segregated by sector due to the cross-sector nature of the activities. For example, Activity reports indicate that there were many activities implemented in parallel in the sectors of HEALTH, EMPL, EDU and local/cantonal level activities. The same events were organized and the same stakeholders were engaged in several activities: Cantons (Sarajevo, Zenica-Doboj, Bosnia-Podrinje) worked in parallel on the introduction of the rulebook in healthcare and on creation of an Anti-corruption Strategy/ Action Plan; Sarajevo Canton also worked in the sector of EDU; Lukavac municipality developed an anti-corruption strategy and adopted an anti￾corruption rulebook for employment; ACCOUNT cross-linked its activities by organizing joint events for stakeholders in the sectors of HEALTH and EDU (e.g., a conference on anti-corruption measures in EDU and HEALTH in Year 3). In terms of geographic coverage, effects in WBP and PP are country-wide as ACCOUNT influenced the state- and entity-level laws. Activities with cantonal institutions in the sectors of HEALTH, EMPL, and EDU and the local￾level activities are rather dispersed in various parts of the country due to a broad roll out of activities (Figure 15). 24 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 15. ACCOUNT Geographic Distribution of Institutions Involved in Activities Finding 7: According to KII’s and Activity reports, the demand-driven approach was effective because it facilitated partnerships with government bodies that expressed a need for assistance, leading to visible results. ACCOUNT and its implementing partners were ready to respond to the demands of institutions that expressed the need for technical assistance for implementation of anti-corruption strategies and action plans. Several activities during the follow-on phase were specifically designed according to a demand put forward by several institutions: the Sarajevo Canton Ministry for Education and Science invited ACCOUNT to take an active role in a working group for improving regulations for employment within the educational institutions in Sarajevo Canton; the same Ministry also asked for expert support in drafting an anti-corruption strategy for the Ministry, while APIK also expressed independent interest in supporting the roll-out of the activity for anti-corruption strategy-making to other cantonal education ministries. Hence, the follow-on Activity took up these opportunities to continue cooperation with interested institutions. During the follow-on phase, ACCOUNT continued with demand-driven approach. Most characteristic examples are: - The cooperation established with RS Ministry of Justice created opportunities for more intensive activities and more effective results in the design of integrity plans in the RS public institutions than was conceptualized in the Program Description. - The proactive approach of the Sarajevo Cantonal Anti-Corruption Team as well as Zenica-Doboj and Bosnia￾Podrinje Cantons marked the start of the ACCOUNT activities at cantonal level. 25 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV - BiH PP Agency opened to cooperation with CSOs after the change of Director, a breakthrough for ACCOUNT’s work on design of new policy proposals in the PP sector. About one third of KIs (13) recognize Sarajevo Canton, Ze￾Do Canton and/ or municipalities (Bijeljina, Goražde, Lukavac, Travnik, Živinice) as good examples of openness and political will for institutionalization of anti-corruption policies. Furthermore, educational institutions were also open to cooperation on anti-corruption topics; 40% of teaching personnel surveyed assessed the trainings to be of very high quality. Almost all teachers surveyed (96%) think that students have gained a very good understanding about identifying corruption and what they can do to prevent it; they agree that all educational institutions should teach anti-corruption in the future. Finding 8: According to KII’s, the political will of government officials in support of anti-corruption work was critical for achievements within ACCOUNT, while institutionalization of the anti￾corruption structures and policies contributed to the higher effectiveness of Activity implementation. Most KIs claim that the overall political environment is not conducive to meaningful participation; 20 KIs were adamant in saying that the political will of the governments is the critical factor for inducing changes and that the effects of CSOs’ input depend on this political will. Once political will has been established, the governments welcome technical assistance from CSOs. Fifteen KIs (7 of them governmental representatives) claim that technical assistance provided by ACCOUNT CSOs was the right assistance and further agree that the CSOs had the expertise required for the task and the provision of quality assistance. This was stressed by KIs from APIK, the BiH Agency for PP, and the RS Ministry of Justice who cooperated with ACCOUNT on the design of new legislation. Based on KIIs it was possible to identify several types of motivations behind political support, as presented below. - Change of management in institutions triggered political will to start cooperation with CSOs and then invited ACCOUNT to cooperate on AC issues. - Urge to implement anti-corruption strategies (approaching deadlines) or complete tasks for which they lack adequate capacity (e.g., entity Ministries of Justice, PP Agency). - Demand for new skills or additional capacity due to professionalization of tasks related to anti-corruption (e.g., appointment of the Sarajevo Canton Anti-Corruption Team). Political will was a prerequisite for the signing of MoUs with ACCOUNT that marked the start of formal and systematic cooperation between ACCOUNT and authorities. Also, political will led to institutionalization of anti- “Management changed and as soon as the new Director came, he opened all these things. We continued communication that we had established before, but now we had a counterpart who is responsive to our invitations, reports, joint organization of events, etc. Such cooperation culminated in the informal engagement of CSOs in the working group for changes and amendments to the law.” -KII with CSO representative “... we simply did not have that expertise and we invited the ACCOUNT Network to discuss things and check if there are options, since we have limited finances … We asked them to consider if there is space in the Activity, in line with the signed MoU, to get the support to establish the team and the IT application. They welcomed our invite and they are now an active stakeholder in this application …” Interview statement by a public institution representative 26 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV corruption structures (e.g., anti-corruption teams) and adoption of AC policies that were often milestones for cooperation between CSOs and governments. 4.2.2 CONCLUSIONS Based on Findings 4 and 5, it is difficult to track achievements per sector area as the Program Description defined anticipated results against which achievements can be measured for only two sectors (public procurement and whistleblower protection), while for the remaining sectors results were defined on an ongoing basis during the implementation process. The Network management has been complicated in design, primarily in connections between Activity Components and in mechanisms of implementation. However, at the activity level, clear expectations were communicated to CSOs and the Media, and beneficiaries see it as practical. ACCOUNT implementation partners and other partners successfully mitigated the risks associated with overly complex lines of communication and coordination that could have negatively affected implementation. Furthermore, the absence of clear theory of change and causality between activities, as well as a very complex map of implementation responsibilities among stakeholders affected the possibility to document and isolate mechanisms at play behind achievements fulfilled within the five sectors. Based on Findings 5–7, ACCOUNT made progress in all five sector areas. Due to high variation in scope, nature, and level of these interventions, it is not possible to compare successes across sectors. The input is most visible in the processes of drafting legislation and putting forward policy recommendations for the state- and entity-level governments in the sectors of public procurement and whistleblower protection, with the RS law on whistleblower protections being adopted during ACCOUNT implementation. In the sectors of HEALTH, EMPL, and EDU, ACCOUNT rolled out template policies and organized monitoring of anti-corruption processes in numerous institutions. Thus, the Activity facilitated broad outreach in terms of themes, geographic reach, and stakeholder engagement. While this high level of dissemination and promotion of anti-corruption themes is a desirable outcome, the available project resources were spread among many themes and actors, leaving limited space for in-depth work in each sector and institution. Finally, as discussed in Findings 7 and 8, the demand-driven approach taken by ACCOUNT’s implementing partners was effective, in particular in terms of seizing opportunities to change the environment of the political landscape and reacting against the backdrop of political will to introduce new policies/regulations. Implementing partners and Network members were ready to respond to the demands of institutions that expressed a need for technical assistance in the development of anti-corruption strategies and action plans, as well as implementation in several instances. Even though it is more complex to monitor and document results with this approach, such an approach does increase the sense of ownership by local stakeholders, as well as increasing the relevance of the overall Activity. 27 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 4.3. EVALUATION QUESTION 3: TO WHAT EXTENT HAS ACCOUNT INCREASED THE QUALITY AND QUANTITY OF INVESTIGATIVE JOURNALISM TARGETING PUBLIC CORRUPTION IN BIH? 4.3.1 FINDINGS Finding 9: ACCOUNT has exceeded targets for production of investigative reports in online portals and is gaining readership. A general increase in media reporting on corruption cannot be determined based on available data. Based on the ACCOUNT M&E Plan, ACCOUNT has almost reached its life-of-activity target of 1,350 on the indicator “Number of investigative reports prepared by media partners,” with total of 1,257 reports published to date and an annual over-performance (for details on ACCOUNT Indicators please see Annex IV). Zurnal.info is producing the largest number of reports, with an average of six (6) investigative reports per week. Surveyed Media Pool members report producing an average of two (2) reports per week in one of the five sectors addressed by ACCOUNT, and within ACCOUNT they produced a total of 217 reports. Figure 16 provides an overview. Although data are not available for all media outlets––Google Analytics data, Zurnal.info, and Interview.ba–– indicate a continuous increase in visitors to their respective websites. Zurnal.info reports an overall increase of 40%, while interview.ba, which had just begun operations, also indicated steady annual growth. A detailed presentation of the Google Analytics is available in Annex VI. With respect to quantity, 12 out of 14 interviewed CSOs, several Media Pool KIs, and over 50% of surveyed CSOs and Media representatives indicate an increase in the quantity of media reporting on corruption issues. KIs from the government institutions did not notice any difference with respect to quantity but held the opinion that the media space is generally oversaturated with reporting on corruption. The evaluation team could not access any data on activity and productions levels of individual media outlets pre-ACCOUNT with respect to investigative reporting that could confirm any of the perceptions expressed during the KIIs and Panel Discussion. Finding 10: A majority of respondents are familiar with Zurnal.info and, to a lesser extent, with Interview.ba and the Media Pool. Citizens are mainly consuming traditional media sources over the internet. Deeper knowledge about the Network’s work is mainly reserved to beneficiary CSOs and Media representatives. Although the Activity made a significant effort to increase outreach to the general population, most respondents agree that competing with public broadcasters still remains challenging in a context where citizens heavily rely on traditional media sources for information. 317 409 308 132 85 Y1 Y2 Y3 Number of investigative reports produced by ACCOUNT annually Total Zurnal.info Total Media Pool per year Figure 16. Investigative Reports Produced within ACCOUNT 28 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV One-third of ACCOUNT CSOs and Media representatives interviewed and surveyed (see Figure 17) frequently followed Zurnal.info and Interview.ba, while another one-third mentioned one of the Media Pool members as a source they frequently consult. Outside of the Network membership, only two governmental institutions stated that they are closely familiar with the content produced by ACCOUNT, while most remained reserved when asked about Zurnal.info or Interview.ba (8 out of 14), stating they are not familiar enough to comment. This may be because most of published content includes government institutions and their representatives. Figure 17. Most Frequently Mentioned Media Sources among Surveyed CSO and Media Representatives Surveyed CSOs and Media Representatives – frequency of mentioning: Zurnal.Info 21 Centre for Investigative Reporting (CIN) 9 6yka.com, Capital.ba, N1-tv 6 Klix.ba 5 Face-tv 4 BDC TV, FTV, HIT TV&Radio, InfoBrcko.com, Interview.ba 3 Al Jazeera, Gerila.info, MojaHercegiovina, Mreža Mira, Nezavisne Novine 2 AKTA.BA, Bljesak.info, BN TV, e-Trafika.net, Fokus.ba, Frontal.ba, FRVT-RTV USK, Impuls portal, List Krajina, Oslobođenje, RTBiH, Sprska info, Stampa, Vaskovic, Puhalo 1 Furthermore, based on NSCP 2015–2018 data, no significant change is noted with respect to citizens’ familiarity with ACCOUNT, as shown in Figure 18. Unfortunately, the NSCP survey does not inquire about individual ACCOUNT-related media outlets but does provide some insight into how informed citizens are about anti￾corruption CSOs. There is a consistent trend regarding citizens’ awareness of anti-corruption CSOs when looking at a broader picture that includes the Centre for Investigative Reporting (CIN) and Transparency International (TI) that shows a slight decrease for all organizations between 2017 and 2018. The NSCP data also indicate that citizens who inform themselves through online and social media are twice as likely to be informed about all three of these organizations; however, only 20 percent of surveyed citizens state that they inform themselves through these media sources for political news. The remaining 80 percent inform themselves via the traditional media outlets. This remains particularly challenging for ACCOUNT Network members, as almost all of them fall within the category of online media portals. Although ACCOUNT is working to encourage citizens to use online platforms and trying to increase outreach to the general public by combining online and print versions of newspapers, the majority of KIs from all stakeholder groups confirmed that for now, the outreach of ACCOUNT is limited to a small fraction of the population. Most members of the panel discussion also expressed concerns about their ability to compete against the traditional media, in particular public broadcasters, that citizens are accustomed to viewing. 29 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 18. Citizens’ Awareness of Anti-corruption Organizations in General and Based on Media Source Finding 11: CSO and Media Panel and KIs perceive the quality of ACCOUNT anti-corruption reporting as much better than it was previously and that it has improved; the content evaluation shows that 1 in 4 reports are very good or excellent. Based on ACCOUNT documentation, KIIs, and evaluation of the content, ACCOUNT investigative reporting has been primarily directed at increasing the quantity of reporting to draw citizens’ attention to the issue of corruption rather than at improving the quality of reporting. Figure 19 presents the average scores per evaluation category for Zurnal.Info and the Media Pool. In general, the components on Ethical Conduct received the highest scores, followed by the Public Interest and Advocacy, with the lower scores relating to Accuracy, Investigative Approach, and Fairness (see overall scores for ACCOUNT in Figure 19). However, based on experts evaluations, in general all rating categories are interconnected and the slight differences between the categories do not affect the overall assessment of the quality per investigative report. 31% 30% 27% 22% 10% 9% 13% 11% 22% 31% 28% 26% 2015 2016 2017 2018 Have you heard of ...: (YES) TI ACCOUNT CIN 46% 47% 38% 20% 21% 20% 50% 45% 42% 2016 2017 2018 of the 20% of individuals that inform themselves via online/social media 30 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 19. ACCOUNT Investigative Reporting, average scores ACCOUNT INVESTIGATIVE REPORTING AVERAGE Excellent [5]; Very good [4]; Good [3]; Fair [2]; Poor [1] ZURNAL MEDIA POOL TOTAL ACCOUNT Number of reports evaluated 16 14 30 Ethical conduct 3.81 4.25 4.02 Public interest 3.84 3.89 3.87 Advocacy 3.47 3.86 3.65 Respect 3.41 3.89 3.63 Rights 3.38 3.82 3.59 Overall average score 3.27 3.77 3.50 Clarity 3.31 3.54 3.42 Accuracy 3.00 3.71 3.33 Investigative approach 2.59 3.54 3.03 Fairness 2.63 3.43 3.00 In general, respondents in the interviews only reluctantly assessed ACCOUNT content. One-half of KIs refused to provide any characterization, in particular those among the governmental/public institutions interviewed (10 out of 14), stating that although they are familiar with ACCOUNT media outlets, mainly Zurnal.info, they are not sufficiently familiar with their content and do not follow it. While CSOs were also reluctant to comment, more than one-third of them (6 out of 14), along with members of the media panel, had opposite views, claiming that Zurnal’s greatest impact is in uncovering and reporting corruption which, according to many, affects public officials’ perception that they can become subjects of public scrutiny. Some panel members and few other KIs singled out as the greatest benefit of ACCOUNT media reporting its timely and quick reporting, which immediately directs attention to the issue at hand. Others noted that real-time reporting comes at the expense of in-depth analysis and collection of evidence, which all agreed requires a significant investment of time. Furthermore, panel members elaborated that the governmental/public institutions are unwilling to provide information, despite an apparent increase in transparency in publishing certain documentation on governmental/public institution websites. Panel discussion members explained that, in essence, governmental/public institutions have actually been more closed and reluctant to cooperate with the media by hiding behind the “transparency” as an excuse. Furthermore, while a strong analytical approach is desirable, several panel members noted that it is not feasible to wait for all evidence as timely public scrutiny and simple questioning may affect governments’ reactions. „…. Thanks to ACCOUNT assistance to the media, there definitely seems to be increased media reporting on corruption, both in volume and quality, because they [ACCOUNT] have tasked us with a new level of seriousness; they grade our stories, we can consult with them on our stories, so that this peer-review process certainly contributed to quality“ - Panel discussion member 31 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ACCOUNT, in its annual reports, provides examples of seven successful stories that have resulted in actions undertaken by governmental/public institutions or officials on investigating the circumstances/cases reported on or changing the course of action. Based on ACCOUNT documentation and internal evaluation of the quality of investigative reports produced by Media Pool members, the primary focus of the investigative reporting was to direct attention to corruption, while quality has not been the significant subject of evaluation. However, there is an overarching perception among the CSOs (55%) and media (77%) surveyed that investigative journalism has improved over the past three years. Based on the evaluation of the quality of investigative reports on the selected sample of ACCOUNT reports, on average 1 in 4 reports produced are of very good or excellent quality, with a higher frequency of high-quality reports produced among the Media Pool members (1 in 2 reports) than by Zurnal.info (1 in 5 reports). However, it must also be noted that Zurnal.info produced five times the volume of investigative reports as compared to the Media Pool members. Finding 12: Journalists operate in a difficult and risky environment, particularly those that report on corruption, furthermore the sources for sustainability of investigative reporting are highly susceptible to political influence. ACCOUNT is one of those sources that enable investigative journalism. Based on the online surveys, one-third of respondents (CSOs and Media representatives) claim that overall media reporting is biased, with 68% of all respondents agreeing that there are only a few media outlets that are unbiased. The majority of KIs hold the same opinion about the media, claiming that it is politicized in general, and that editorial policy is influenced by political parties/elites. The majority of KIs (19) think that the media is under strong influence by political parties and used as tool to propagate their messages or attack other parties. One-third (16) also explained that media reporting in general is focused on sensationalism and negative reporting, a view strongly represented by governmental/public institutions (8 out of 14). One-third of all KIs (13) elaborate that reporting is lacking in “educative” topics and good examples of the systems mechanisms and their application. Media Panel members, elaborated that sources of financing for independent investigative journalism are primarily available through various donor funded projects, and even fewer with the focus on corruption as was the case with ACCOUNT. The private sector, as a potential domestic source of finance, is also under political influence, according to some also through public procurement mechanism (2 KIIs). Media representatives in the panel discussion and few KIIs and IPs, also elaborated that there are few journalist ready to take on the risks related to reporting on corrupt behavior of the government/political leaders, as they faced with continuous types of public ‘I believe that there is a big bang sensation that they achieved in the past few years. It’s unbelievable. People are reading them, they are copying them, talking about them. Faith returned to people regarding the investigative journalism. There is barely any [investigative journalism]. We have Žurnal and CIN, and maybe few more, the rest of them you just can’t trust.’ - KI from CSO ‘For good or for bad, ACCOUNT has reached the effect that people [in public institutions] fear they will be detected. ACCOUNT is persistent and that creates some complications for the government with respect to misconduct they engage in. They [ACCOUNT] are pretty capable of gathering information, processing and presenting it to citizens - that itself creates a spectacle – Boom! However, that’s like a firecracker - it bangs, people scare jump, and then - nothing. - KI from governmental/public institution 32 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV scrutiny, law-suits and event threats to life. All of this, according to KIIs affects both the individual journalist sustainability and motivation on reporting about corruption as well as media outlets sustainability in the face of incurring costs to defend themselves against the financial implications of the lawsuits in the face of inexistent private sector support to increase their funding. The Media Panel members, stated that investigative reporting, in particular on corruption would be hardly possible without donor support. Some of the explanations relate to high level of political influence on media itself, but also political influence over the private sector, as the only other possible sources of finances for media, only one KI form the government/public sector held a similar opinion. ACCOUNT’s financial support, along with the requirement for certain professional and quality standard motivated journalist to write a story that would meet those standards. The same type of motivation is related to application for the journalism award. Finding 13: The media reports indicate correlation with advocacy and monitoring activities. However, two-way communication between CSOs and Media representatives is sub-optimal. Based on the Program Description and the ACCOUNT Award Modification for the media component, investigative reporting was envisioned to focus on one of the five sectors and feed into the advocacy activities of Component 1 as well as integrate the monitoring activities findings from Component 2. The implementing partners elaborated that all inputs received from the CSO organizations, in particular those that form the Monitoring Teams, have been a significant source of information for the investigative reports. This is to a large extent confirmed by CSOs, all of whom noted that the media role in promoting ACCOUNT activities has been a strong element during implementation, with several of them remarking on an increased interest by the media in their work. Furthermore, most of them (9) also stated that media reporting has been well connected to the overarching advocacy initiatives, with few of them providing examples that their work „Given that we are writing on corruption, I am primarily talking about public procurement and public employment, we always need additional interpretations and support in understanding the issues, NGOs that are dealing with those issues can provide accurate information, or provide direction, what we should pay attention to, they have experts for various specific sub-fields in all sectors, so I frequently reach out to them. “ - Panel discussion member ‘The legislation on defamation and libel is particularly misused. An example of that is the weekly magazine Slobodna Bosna, maybe the last political magazine that dealt with corruption was forced to shut down as a result of lawsuits, by politicians, on defamation. Those that sue the media have advantages in the judiciary over media.’ - Panel discussion member ‘… we recently had a corruption story on a large local company… . And we shared it with all outlets to republish for free. No one did. They didn’t because that company is one of their largest advertising clients, no one is going to cut off that source of funding for the sake of truth. That is a big problem for the journalism profession, if you want to work you constantly have to balance not to find yourself in a conflict – that can cause financial consequences. It is difficult to be independent.’ - Panel discussion member ‘The financial means for media are less and less; today the sources from marketing are 10–20% smaller than they were 5–10 years ago. The marketing firms themselves are controlled by political parties.’ - Panel discussion member 33 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV and findings have been, to a large degree, well translated in the media reporting. Only one CSO noted that messaging based on monitoring focused more on the negative or wrong findings. Survey results from the media also confirm that there has been interconnection between CSOs and Media pool members in several forms, and in media through either direct interaction on their activities, individual meetings with CSOs, or coordination meeting within sectors for information on the sector area topics (see Figure 20 for details). Furthermore, 50% of the articles randomly selected for review show that all sector areas covered by ACCOUNT were discussed in the reports, which indicates that the investigative report topics were focused on the sectors. However, the ACCOUNT reports for this component neither elaborate on nor establish a clear connection between this component and the remaining components, but rather focus on the output in terms of investigative reports produced. It is not clearly explained to what degree each sector was covered, nor do the ACCOUNT reports elaborate on the degree to which the CSOs findings and investigations have been integrated/sourced for reporting. One of the reasons why this may be the case, is that the Program Description itself does not stipulate any anticipated results in relation to this interconnection between reporting and advocacy, even if the Program Description and the fourth Award modification state that two-way CSO-Media communication is expected to occur. Figure 20. Surveyed Media Pool Members’ Frequency of Reporting on ACCOUNT’s Five Sector Areas If you are a member of the ACCOUNT media pool, please answer if and how often you have participated in the following activities of ACCOUNT (on average): Published corruption/anti-corruption investigative reports once (….) on: weekly monthly quarterly annually DN Frequency Whistleblower protection issues 11% 11% 56% 11% 11% 9 Public procurement 33% 22% 33% 11% 1% 9 Public employment 0% 22% 33% 33% 11% 9 Healthcare institutions 33% 33% 22% 0% 11% 9 Education 11% 44% 22% 11% 11% 9 Prepared a specialized analysis on corruption in public institutions/officials. 22% 44% 11% 22% 0% 9 4.3.2. CONCLUSIONS Based on Findings 9 and 10, in the absence of baseline data, the evaluation team was unable to identify a change in the quantity of investigative reporting on corruption, either in the aggregate or in any specific sector areas. However, with respect to planned output, ACCOUNT has overachieved in terms of quantity. As generally observed in relation to quantity, there is an oversaturation of anti-corruption content in the media focusing on corruption; however, little of it focuses on positive or educative content for citizens. With respect to ACCOUNT media sources, Zurnal.info is the strongest media outlet, showing a promising annual increase in readership, while interview.ba and Media Pool are primarily known among ACCOUNT beneficiary CSOs and Media representatives. ACCOUNT Media outlets are generally recognized as reliable sources of information. However, their outreach 34 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV to a wider population remains challenging, as close to 80% of citizens inform themselves primarily through traditional media outlets, as opposed to online and social media. Based on Findings 11 and 13, ACCOUNT reporting has been directed at drawing citizens’ attention to current corruption issues by covering topics that are most relevant to citizens, with a high prevalence of advocacy elements in the reporting. The general quality of ACCOUNT’s media reporting is good and occasionally excellent, but there is still much room for improvement. The trade-off between “exclusivity” and quality should be further investigated, as the current approach taken by ACCOUNT media is perceived to be effective. This effectiveness presents itself through ACCOUNT’s media image as a credible watchdog, causing anxiety among governmental/public institution representatives about being detected in misconduct. The two-way communication between CSOs and Media representatives, although existing within ACCOUNT, has not been central focus of the intervention. Finally, based on Finding 12, support to independent and investigative reporting is primarily donor funded. It seems unlikely, in the current economic and political environment, that media outlets detecting and report on corruption issues would find sustainable resources to keep them focused on investigative reporting about corruption. The lack of financial resources is further amplified by the fact that they are competing with public broadcasters that are far better resourced, yet laboring under strong political influence. 4.4. EVALUATION QUESTION 4: TO WHAT EXTENT HAVE INTERVENTIONS UNDER ACCOUNT INFLUENCED PUBLIC AWARENESS OF CORRUPTION? 4.4.1. FINDINGS Finding 14: 1 in 5 citizens have heard of ACCOUNT. However, there is no change in public perception of corruption. Based on the Program Description and M&E Plan, this change was expected in response to (1) increased media reporting on corruption (discussed in EQ3) and (2) overall ACCOUNT initiatives. As illustrated in Finding 10, Figure 18, one in four citizens have heard of ACCOUNT, in particular citizens that inform themselves through online media sources (portals and social media). Furthermore, CSO KIs––particularly those who have worked on the lower levels of governmental/public institutions in the sectors of EDU, HEALTH, and local-level strategies––explained that they noted changes in awareness among their beneficiaries. For example, municipal-level government representatives increased their understanding and support for the development of Action Plans and recognized their significance in fighting corruption, as most of those Action Plans envisioned definitions of various types of procedures to be applied in that fight. For EDU, noted was that education workers and students’ had increased interest in topics relevant to corruption issues, while in HEALTH they noted an openness to establishing procedures whereby individuals can anonymously report corruption. However, an improvement in the awareness has not been sufficient to change citizens’ general perceptions. Based on the National Survey of Citizens’ Perceptions Report, citizens’ perceptions about the existence of petty, grand, and systemic corruption remains unchanged, with 90% of surveyed citizens believing it exists. The same is true for citizens’ perceptions of corruption in two of the sector areas covered by ACCOUNT, namely, EMPL and PP (see Figure 21 for details). 35 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 21. Citizens’ Perceptions on Corruption in Public Employment and Public Procurement NSCP 2016–2018 results How prevalent do you believe corruption to be in: Public Employment Public Procurement 2016 2017 2018 2016 2017 2018 Not at all 2% 1% 2% 2% 2% 2% Somewhat 7% 7% 10% 9% 8% 11% Moderately 18% 18% 22% 28% 25% 26% Extremely 70% 71% 64% 55% 60% 59% Does not know/Refuses to answer 2% 3% 2% 6% 6% 3% Total for Yes 96% 96% 96% 92% 93% 95% Furthermore, the ACCOUNT M&E Plan and project-level indicator Percentage of citizens informed of local CSOs’ anti￾corruption activities remains below 3% throughout the ACCOUNT follow-on phase. The percentage is somewhat higher where citizens are asked if they have heard of any Activities that ACCOUNT has organized in their town (see Figure 22). However, there is a decreasing trend from 2015 onward. Based on the data, neither the ACCOUNT investigative reporting (which only a small portion of citizens access) nor activities on the local level have thus far affected citizens’ perceptions. 14% 11% 8% 9% 2015 2016 2017 2018 Have you heard of any events ACCOUNT organized in your town? Yes (1) Figure 22. Citizens Informed about ACCOUNT Local Anti-corruption Initiatives 36 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Finding 15: There is a perception among ACCOUNT CSOs that citizens are more ready to report corruption. However, available statistics do not indicate this change. Among ACCOUNT-surveyed CSOs and Media representatives, 50 percent believe that citizens are more frequently reporting corruption today than three years ago. Of the legal-aid providers interviewed, only one noted an increase in the number of people seeking legal aid and without being able to specify if that was related to corruption. Of 27 KIs who discussed citizens’ readiness to report corruption, most (16) across all stakeholder KIs did not think that people are more ready today than previously. Higher Judicial and Prosecutorial Council (HJPC) administrative data on corruption reports filed with the Prosecutors’ Offices, as well as ACCOUNT data on corruption reports filed with their Legal Aid team, both show a slight downward trend in the later periods (see Figure 23). In many cases, KIs noted a large gap between citizens’ perceptions of what corruption is and how the legislation treats corruption. Based on data from implementing partners and other KIs, most of the reported cases are either unfounded or not corruption. Another reason given by most KIs (26 of 38) is the ineffectiveness of the judicial system that primarily affects citizens’ decisions to report corruption. With regard to the judicial system, most frequently mentioned shortcomings are the ineffective work of the Prosecutors’ Offices and the mild or complete absence of sanctions. Finally, a third explanation provided by most KIs (26) is fear of reporting corruption on the part of citizens. 4.4.2. CONCLUSIONS Based on Findings 14 and 15, available data are insufficient to detect any changes in citizens’ awareness of corruption, even if ACCOUNT CSOs have noted some changes among those they have interacted with. Citizens’ perceptions remain the same, in terms of both corruption issues and the anti-corruption CSOs. The theory of change for ACCOUNT relied on a strong assumption that citizens will consume investigative reports produced within ACCOUNT, and that assumption has not been fulfilled to the extent expected by results. Anticipated awareness change as reflected in change on the part of citizens taking action against corruption by reporting it shows a decreasing trend from last year due to citizens’ fear of reporting corruption. This is further amplified by the fact that corruption remains unsanctioned, or very weakly sanctioned, by responsible authorities, affecting citizens’ pessimistic views. The primary culprit for this is most frequently seen to be the weak judiciary, or more precisely, the prosecutors. 10 20 30 40 0 22 20 0 20 40 60 2016 2017 2018 2019 Number of corruption cases reported to the Legal Team of ACCOUNT Target Actual 1435 1713 1805 1647 0 500 1000 1500 2000 Y2014 Y2015 Y2016 Y2017 Newly received criminal reports by Prosecutors’ Offices in BiH Figure 23. Corruption Cases Reported to Prosecutors’ Offices; and to ACCOUNT 37 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 4.5. EVALUATION QUESTION 5: IS THE ACCOUNT LEGAL-AID MODEL TO PROTECT INDIVIDUALS REPORTING CORRUPTION AND WHISTLEBLOWERS PERCEIVED AS EFFECTIVE BY ACCOUNT BENEFICIARIES? 4.5.1 FINDINGS Finding 16: There is familiarity with the existence of ACCOUNT’s legal-aid provision. A majority of KIs (21) and 63 percent of the CSOs and Media representatives surveyed believe that ACCOUNT is recognized as an address where corruption can be reported (see Figure 24). However, only two KIs noted that they directed individuals to ACCOUNT for consultation on a given case they are facing. In fact, implementing partners elaborated that a majority of the reports are received directly though their system, and in some cases via telephone. In general, one-third of KIs (13) believed that citizens can easily identify free legal-aid providers when in need of one, but that a general citizen if not in need is less likely to know about them. Yet the few KIs with a deeper understanding of free legal aid explained that the legal-aid model should be further expanded, in particular to provide legal assistance throughout the procedures in the judiciary. Finding 17: ACCOUNT legal-aid assistance is perceived as professional and contributing to important cases. Surveyed CSOs and Media representatives to a large extent believe that the ACCOUNT Legal Team and Assistance is professional and that it assisted in important cases and stopped corruption (50%), see Figure 24. All (4) interviewed recipients of ACCOUNT legal aid were satisfied with the assistance they received and pointed to a high level of professionalism in the ACCOUNT Legal Aid Team. They explained that they were well informed about the assistance they can receive though ACCOUNT; however, they also experienced difficulties in navigating their way through the judiciary system. One legal-aid recipient noted that monitoring of the court proceedings by ACCOUNT seems to have an effect on the judges, who appear to be more careful about their case than before. Based on ACCOUNT data, thus far 8 of 49 cases for which legal aid was provided had a positive outcome for the victim, meaning that an indictment on corruption, a court verdict on corruption, and/or any type of administrative decision in an institution in favor of a whistleblower, and/or disciplinary punishment of the perpetrator has been achieved (for further detail on the number of cases assisted by ACCOUNT on an annual basis please consult Annex III). Figure 24. Perceptions of Surveyed CSOs and Media Representatives on ACCOUNT Legal-Aid Provision CSO and Media online survey on Legal Aid Provision On a Scale from 1 to 4, strongly disagree to strongly agree with the statement ACCOUNT Legal Aid Team is: Frequency Strongly agree … professional in their work 38 61% … has provided assistance to important cases and stopped further corruption in public institutions 38 55% ACCOUNT is recognized by citizens as an address to report corruption 38 63% 38 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Finding 18: Citizens trust non-governmental legal-aid provision over public-sector legal-aid centers. One-third of KIs believes that CSOs are still not sufficiently promoted as legal-aid providers; however, they still believe that citizens have greater trust in CSOs than in public institutions. Interviewed KIs (15 of 38) expressed several concerns with respect to public institutions’ free legal-aid provision. One concern was that this mechanism may be subject to government influence, either through direct political influence on the management of the institutions or through budgetary constraints and the mere fact that “government” would not investigate “government.” A second concern involved a probable lack of expertise, particularly for complex cases. Some legal￾aid providers interviewed further elaborated that developing the expertise was a long process and that in general corruption cases are very complex, requiring diverse types of expertise unlikely to exist among the free legal-aid providers. A third concern was that legal aid may only be available to those facing dire social circumstances, since the qualification criteria for legal aid is said to be very limiting. Furthermore, a KI from an international organization expressed concerns that legal-aid provision by CSOs, although they have built up expertise and some capacity, is in danger of shrinking with donor withdrawals in the wake of the establishment of free public legal-aid centers being developed to show that the domestic government is assuming its obligation with respect to legal-aid provision. Finding 19: Coordination of legal-aid providers has been tightly linked to the C1 and C2 whistleblower-protection legislation. The ACCOUNT Program Description elaborates on the establishment of effective coordination among the free legal-aid providers. Survey respondents in 53% of the cases believe that whistleblower-protection mechanisms and legal-aid services for whistleblowers have improved since 2015. However, none of the KIs interviewed were aware of efforts related to coordinate free legal aid, in particular the free legal-aid providers. Few (4) KIs knew about bpp.ba, which was envisioned as an online registry for reporting corruption cases, and one noted that the website itself should be further developed and promoted. ACCOUNT reports also provided assistance through the RS Ministry of Justice by delivering training to the Public Free Legal Aid Centers in RS, which has been well received, and the majority of KIs consider this an important aspect. However, KIs in most cases related this component to the work on whistleblower protection at the state and entity levels through the SGs and MTs. In fact, based on ACCOUNT documentation, the legal-aid team has provided input relevant for this sector, including analysis of existing legislative frameworks and draft recommendations submitted to the ACCOUNT SG and implementing partners coordinating activities within this sector. 4.5.2. CONCLUSIONS Based on Findings 16 and 17, ACCOUNT legal-aid provision is important and welcomed. However, the current structure is insufficient, as legal representation before the courts remains a primary challenge for many who would pursue that step. Coupled with still weak protection mechanisms for individuals reporting corruption, fear is the most frequently mentioned limiting factor. As noted under the conclusions for EQ4, additional factors relate to citizens’ perceptions of judiciary effectiveness and apathy related to the absence of sanctions for perpetrators. Based on Finding 18, although the BiH government(s) are assuming responsibility for the provision of free legal aid, citizens are pessimistic that public institutions will effectively fight corruption of the government that has founded them. Likely withdrawal of donors from programs like ACCOUNT, for non-governmental providers, as 39 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV more free legal-aid centers are established may mean that fewer people will receive free legal aid, in particular those reporting corruption. Finally, based on Finding 19, the evaluation team could not determine whether there has been improved coordination of legal-aid providers, as attention has been primarily focused on the legislative framework for WBP within Component 1. The improved coordination is also perceived to rely on outcomes form that component. 4.6. RECOMMENDATIONS Recommendation 1: Consider further consolidation of the ACCOUNT Network organized around CSOs with expertise in specific sectors and expanding their anti-corruption expertise to build on current results and credibility gained. Coordination among members should be systematized in line with CSOs’ strengths/expertise and along clear rules and division of responsibilities (e.g., per sector type of activity) so as to ensure continual in-depth engagement of expert CSOs. Recommendation 2: Consider streamlining support for activities focusing on in-depth implementation of policies and procedures initially developed within all five ACCOUNT sector areas, in cooperation with governmental/public institutions, in particular those on the lower level of governance, rather than rolling out in new geographic/administrative areas. A technical-assistance approach to implementation may influence stronger higher long-term effects, as well as building trust relationships between the CSOs and government institutions. Recommendation 3: Consider increasing cohesion between the different intervention mechanisms and coordinating between the different roles of CSOs in the Network. For example, strengthen the links between sub-grantees involved in advocacy and monitoring activities; ensure CSOs’ continual engagement in the monitoring of implementation of procedures/ policies that were developed within the Activity and that were adopted by the government(s); create spaces for communication between CSOs and the media. Such efforts would lead to a programmatic approach with greater cohesion in implementation of all lines of action. Furthermore, a more coherent outcome-focused reporting would ensure better communication of future achievements, particularly for complex intervention designs. Recommendation 4: Maintain flexibility in adjusting the program design to build on the opportunities of a changing political landscape to achieve better results. The demand-driven approach should also be more intentionally reflected in the program design and accounted for with respect to expected results. Recommendation 5: Continue support to independent media in providing evidence-based reporting on corruption, as the current political environment and funding sources for independent media are highly volatile and subject to political influence. Recommendation 6: Improve communication between media and CSOs to streamline high-quality reporting that can strengthen both the journalists’ expertise in sector areas as well as the CSOs’ visibility in advocacy processes relevant to anti-corruption and the introduction of positive and educative contents to media reporting. Recommendation 7: Consider placing greater focus on increasing the level of quality of media reporting in general. At the same time, along with quality, consider maintaining the current level of quantity to 40 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV balance the sensationalistic media outputs and messages in order to provide alternative views to the mainstream media that are under strong political control. Recommendation 8: Emphasize delivering positive, informative anti-corruption content through media reporting and improve communication of the results achieved by directing citizens’ attention to issues uncovered through investigative reporting. Furthermore, consider increased coordination between other Activities related to the justice sector and the rule of law, in particular as citizens’ perceptions are highly dependent on other contextual factors and not exclusively on media reporting. Recommendation 9: Continue providing support to free legal-aid providers, especially non￾governmental providers, until public institutions can improve their public image. Also, consider expanding this type of assistance to include representation of cases before the courts. 41 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEXES - LIST ANNEX I: EVALUATION STATEMENT OFWORK ANNEX II:ACCOUNT SUB-AWARDS AND ROLES OF CSOS ANNEX III:ACCOUNT M&E PLAN INDICATORS ANNEX IV: LIST OF KEY INFORMANTS ANNEXV: QUALITY OF INVESTIGATIVE REPORTS PRODUCEDWITHIN ACCOUNT EVALUATION ANNEXVI:ACCOUNT GOOGLE ANALYTICS FOR INVESTIGATIVE REPORTING ANNEXVII: MAIN DATA COLLECTION INSTRUMENTS ANNEX VIII: COMMENTS FROM THE IMPLEMENTING PARTNER AND USAID/BIH MISSION ON THE ACCOUNT EVALUATION ANNEX IX: THE EVALUATION TEAM RESPONSES TO COMMENTS FROM THE IMPLEMENTING PARTNER AND USAID/BIH MISSION 42 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX I: EVALUATION STATEMENT OF WORK USAID/BiH Democracy Office STATEMENT OF WORK Performance Evaluation Anti-Corruption Civic Organizations’ Unified Network Follow-on activity (ACCOUNT) I. PURPOSE OF THE EVALUATION The purpose of the performance evaluation of the USAID/BiH Anti-Corruption Civic Organizations’ Unified Network Follow-on activity (ACCOUNT) is threefold: iv. Assess the Activity’s implementation performance against the stated deliverables in the Cooperative Agreement. v. Learn from local partners about the best approaches for civil society organizations (CSOs) to fight corruption, gain insights about ACCOUNT implementation mechanisms for strengthening BiH CSOs’ ability to advocate based on evidence-based arguments and influence policy to improve the anti-corruption regulation through a coordinated approach. vi. Provide recommendations based on the lessons learned from ACCOUNT to inform the design of a potential new USAID/BiH anti-corruption intervention, as well as provide information to implementers and other international development organizations working in the same area. II. PROGRAM INFORMATION Program Name Anti-Corruption Civic Organizations’ Unified Network Follow-on activity (ACCOUNT) Contractor 1 Contractor II Center for Media Development and Analysis (CRMA) and INFOHOUSE (IH) Cooperative Agreement # AID-168-A-15-00001 Total Estimated Cost (TEC) $ 2,499,474 ($ 2,563,823 inclusive IP contribution) Life of Program July 15th, 2015 – July 14th, 2019 Active Geographic Regions Throughout BiH USAID/BiH Project 1.2. More functional and accountable institutions and actors that meet citizens’ needs N 43 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV III. BACKGROUND The four-year USAID Anti-Corruption Civic Organizations' Unified Network Activity (ACCOUNT) started its implementation in July 2015 as a follow-on Activity, building on the successes of the previous USAID anti￾corruption intervention. The overall objective of the follow-on intervention was to create an environment that increases civil society participation and reforms through strong collaboration and cooperation in anti-corruption initiatives. Furthermore, it intended to align the priorities and activities in accordance with the anti-corruption obligations required for Euro-Atlantic integration. ACCOUNT envisioned providing sub-awards for up to 50 ACCOUNT CSO members to undertake public advocacy and awareness-raising activities. Through this sub-award mechanism, ACCOUNT was to support the investigative research and production of more than 100 media and CSO reports intended to stimulate initiatives on public anti-corruption policies that respond to concerns raised by citizens. ACCOUNT envisioned ensuring evidence-based research, civic monitoring and investigative reporting on corruption cases in five sectors, 1) public procurement, 2) whistleblowing, 3) public sector employment, 4) healthcare, and 5) education. Furthermore, Network members engaged at the local level to develop and/or implement sound anti-corruption strategies (at the community and cantonal level). Additional effort was invested in increasing the volume, quality and outreach of sector-based monitoring reports and ensuring they are accompanied by appropriate and timely investigative media reports to raise public awareness and stimulate civic participation. The Activity’s sub-awards/grants model was intended to stimulate and support the ACCOUNT Network members that includes over 170 members through the following mechanisms: • Sub-awards are provided to ACCOUNT Network members engaged under the relevant sector group to lead efforts in the improvement of anti-corruption regulation and strategies, utilizing evidence-based intervention approaches. • Sub-grants are provided for public advocacy to lead organizations in the sector groups and their partners (five grants of approximately $15,000 each per year) as well as to the lead organizations for the anti-corruption strategies in the project municipalities & cantons (7 grants of approximately $5,000 each per year). • Sub-awards are also implemented in the form of contracts with non-governmental organizations (NGOs) and media outlets or journalists involved in the following project components: “Monitoring and reporting on corruption in the selected sectors” and “Public awareness through strategically targeted media campaigns.” The monitoring component will involve five sectoral monitoring teams, each comprised of up to five CSOs. Up to $8,000 will be awarded to these teams annually as a contribution towards the costs of monitoring and reporting. • To increase the volume, quality, number and outreach of investigative reports, ACCOUNT will support around 120 investigative stories over the four project years (all media formats), equivalent to $300,000. ACCOUNT intended to achieve this through an operational model that is based on: • Strengthening civil society grassroots organizations: capacity building was used as an essential component of the bottom-up approach and involved awareness raising, training, analysis, and setting criteria for activities. ACCOUNT expected that this method would contribute to the RFA goal of fostering civil sector sustainability. 44 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV • Research/surveys, monitoring and reporting: this was envisaged as continuous and cumulative work based on empirical data, fact gathering, public reports/surveys on sector-based corruption, and “human features stories.” This method was expected to ensure public awareness of corruption and its consequences, including its impact on social and economic development. • Anti-corruption initiatives and work on corruption cases: it was expected that the project would engage in advocacy initiatives for anti-corruption policies, backed up by actual cases and examples and the experiences of whistle-blowers and other individuals reporting corruption, to demonstrate how to prevent or suppress corruption in practice. ACCOUNT aimed to achieve four main results as expressed in the ACCOUNT M&E Plan: 5) Increased CSO involvement and input regarding anticorruption legislation. 6) Strengthened CSO capacity for in-depth monitoring, data processing, synthesis and analysis of corruption in selected sectors. 7) Strengthened public awareness of the problem of corruption and public engagement in anti￾corruption initiatives. 8) An appropriate and effective legal aid model to protect individuals reporting corruption and whistleblowers established The ACCOUNT assistance is organized in four main Components each focused on one of the main results; however, as a whole they complement and reinforce each other to achieve the outlined results. An illustrative presentation of the Activity results and sub-results as well as the corresponding components is presented in the Figure 1 (on the next page). 45 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 1. ACCOUNT Results Framework and Components I Activity Purpose: An environment that increases civil society participation in anti-corruption initiatives and reforms through strong collaboration and cooperation Activity Purpose: An environment that increases civil society participation in anti-corruption initiatives and reforms through strong collaboration and cooperation Activity Sub￾purpose 1: Increased CSO involvement and input regarding anticorruption legislation Activity Sub￾purpose 1: Increased CSO involvement and input regarding anticorruption legislation Activity Sub￾purpose 2: Strengthened CSO capacity for in-depth monitoring, data processing, synthesis and analysis of corruption in selected sectors Activity Sub￾purpose 2: Strengthened CSO capacity for in-depth monitoring, data processing, synthesis and analysis of corruption in selected sectors Activity Sub￾purpose 3: Strengthened public awareness of the problem of corruption and public engagement in anti-corruption initiatives Activity Sub￾purpose 3: Strengthened public awareness of the problem of corruption and public engagement in anti-corruption initiatives Activity Sub￾purpose 4: An appropriate and effective legal-aid model to protect individuals reporting corruption and whistleblowers Activity Sub￾purpose 4: An appropriate and effective legal-aid model to protect individuals reporting corruption and whistleblowers Component 1: Public Advocacy on selected anti￾corruption legislation Component 1: Public Advocacy on selected anti￾corruption legislation Component 2: Monitoring and reporting on corruption in the selected sectors Component 2: Monitoring and reporting on corruption in the selected sectors Component 4: Free legal assistance to victims of corruption and whistleblowers Component 4: Free legal assistance to victims of corruption and whistleblowers Component 3: Public awareness through strategically targeted media campaigns Component 3: Public awareness through strategically targeted media campaigns Anticipated Result 1.1: CSO recommendations to improve implementation of the state￾level Whistleblower Protection Law taken into consideration by government Anticipated Result 1.2: CSO inputs for passage of the entity￾level whistleblower protection laws taken into consideration by government Anticipated Result 1.3: CSO recommendations to improve implementation of the Public Procurement Law taken into consideration by government Anticipated Result 1.1: CSO recommendations to improve implementation of the state￾level Whistleblower Protection Law taken into consideration by government Anticipated Result 1.2: CSO inputs for passage of the entity￾level whistleblower protection Anticipated Result 2.1: CSOs effectively analyze, monitor and publish reports on corruption in specified sectors Anticipated Result 2.2: Increased recognition of local CSOs within their communities Anticipated Result 2.1: CSOs effectively analyze, monitor and publish reports on corruption in specified sectors Anticipated Result 2.2: Increased recognition of local CSOs within their communities Anticipated Result 3.1: Increased media reporting on corruption Anticipated Result 3.2: Increased public awareness of corruption cases in the specified sectors Anticipated Result 3.1: Increased media reporting on corruption Anticipated Result 3.2: Increased public awareness of corruption cases in the specified sectors Anticipated Result 4.1: Victims of corruption and whistleblowers to receive free legal aid to make them more secure about reporting corruption Anticipated Result 4.1: Victims of corruption and whistleblowers to receive free legal aid to make them more secure about reporting corruption 46 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Within Component 1 and Component 2, ACCOUNT worked with the network organizations to improve anti-corruption strategies and practices in the five selected sectors. Within these two components, ACCOUNT has supported 19 CSOs (17 sub-awards) through the sub-award mechanisms related to: 1) sector-area advocacy; and 2) establishment of watchdog teams and their monitoring and analysis (24 sub-awards). For both of these mechanisms, ACCOUNT has selected sector-leads. Some organizations were selected to play that role through several mechanisms, while others were selected once. The full ACCOUNT network supported the lead organizations. Figure 2 below illustrates the sub-award mechanisms and grantees. Based on the ACCOUNT M&E plan, 73% of all of their policy proposals submitted to executive governments have been accepted by the government to date. Furthermore, ACCOUNT watchdog teams conduct annual monitoring in each sector-area, which is used to inform ACCOUNT advocacy efforts and proposals to the government. With regard to the sectors, ACCOUNT has established cooperation with the state-level and entity-level governments, more specifically, within: The Sector for Whistleblower Protection, ACCOUNT signed a memorandum of understanding (MoU) with the Agency for Prevention of Corruption and Coordination of the Fight against Corruption (APIK) on the implementation of the national Anti-Corruption Strategy. ACCOUNT also carried out assessments on whistleblower protection including a risk assessment of the existing law in coordination with the USAID/SGIP Activity. Finally, in coordination with the US Embassy, Transparency International, and the EU Delegation, ACCOUNT submitted 5 amendments to the BiH Law on Whistleblower Protection to the BiH parliamentary bodies for consideration. The amendments have been cleared by the Constitutional Legal Committee; however, the law has not been adopted in the second reading. On the Entity Level, ACCOUNT has established close cooperation with the Ministries of Justice (MoJ) both in RS and in FBiH. Neither of the Entities had adopted a Whistleblower Protection Law (WBPL) at the time of ACCOUNT initiation. In the RS, ACCOUNT assisted the MoJ in drafting the WBPL, whereby the MoJ accepted 14 out of 30 ACCOUNT proposed recommendations. The draft was sent to public consultations with ACCOUNT sector members participating in 4 public discussions. In parallel, ACCOUNT worked with the MoJ on the improvement of Legal Aid Protection by the RS Centers for Free Legal Aid in RS; delivering trainings to the centers on how to improve protection of whistle-blowers. In FBiH, ACCOUNT worked with the MoJ on the development of the FBiH WBPL, whereby they facilitated cooperation between RS and FBiH on the process. Furthermore, ACCOUNT was involved in the working group for development of the law, and provided assistance in its drafting. The draft was adopted by the FBiH Government and sent to the FBiH Parliament. Within the Sector for Public Procurement, ACCOUNT resumed activities covered within the previous iteration of ACCOUNT by proposing a set of recommendations for amendments to the Public Procurement Law (PPL). The recommendations, initially not accepted by BiH institutions, led to ACCOUNT’s reorganization in its advocacy activities including organization of the annual conference on issues related to the PPL and communication of ACCOUNT recommendations. ACCOUNT soon established cooperation with the Public Procurement Agency with whom they organized a workshop on the PPL. Also among the participants were the members of Council of Ministers (CoM) Working Group tasked to prepare amendments to PPL. Subsequently, ACCOUNT, in coordination with Association Tender, Association of Employers of BiH, Open Society Fund, Transparency International, and Analitika, presented 20 amendments to 16 articles to the WG. Amendments were accepted to 5 of those articles in the final draft PPL and sent for e-consultations. Within the healthcare sector, ACCOUNT sector leads (ICVA and Stop Mobbing) worked intensively to establish connections with various levels of government (RS Ministry of Health and Social Welfare, the Anti- 47 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Corruption Teams of Sarajevo Canton and Zenica-Doboj Canton, and Bosnia Podrinje Canton government), and public institutions (University of Sarajevo, 9 healthcare institutions in Sarajevo and Zenica). ACCOUNT members developed a harmonized anti-corruption rulebook to be used in the selected healthcare institutions in Sarajevo Canton and in the project regions of RS, while in later stages they expanded to Zenica-Doboj. Finally, the rulebook was piloted in the 9 healthcare institutions. In other areas, in cooperation with the government institutions such as Bosnia Podrinje Canton and RS, discussions with healthcare institutions and other stakeholders have been organized in an effort to raise awareness. Within the education sector, ACCOUNT focused on preventative activities with efforts to introduce anti￾corruption education into the curriculum of elementary and high schools. In these efforts, ACCOUNT and the sector lead in particular, have established cooperation with the ministries of education in Sarajevo Canton, Hercegovina-Neretva Canton and the RS, and piloted the curriculum in 35 schools in both of the mentioned cantons, as well as in Bijeljina and Trebinje. This included capacity building of teachers via Training of Teachers. Additionally, within the raising awareness activities, ACCOUNT reports organizing campaigns across the cities where member COs operate to support conferences with students, teachers, and professors discussing the elimination of corruption in education and corruption in general. Within the sector on public employment, the ACCOUNT sector lead (primarily HCHR) has primarily coordinated with sector groups to strengthen their capacity for monitoring public sector employment. ACCOUNT engaged with cantonal anti-corruption teams in 10 cantons, presenting them with the Model of Employment Regulations in Public Administration Rulebook as a starting point of activities. ACCOUNT continues to engage with the anti-corruption teams and the CSO sector to further refine the rulebook. In parallel, cooperation was also established with the Zivinice municipality, with respect to public employment, where ACCOUNT members were invited to participate in the interview process of candidates being selected for the employment in this municipality’s institutions. 48 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 2. ACCOUNT supported organizations within the sub-award mechanism Name of the Organization Implementation Year Sub-Award Type CI - Sector-based Leading Organizations - C2: Activity 2.2. Monitoring Groups for Sectors C2: Activity 2.3. Anti￾corruption Strategies on Local Level Leads Total # or Grants per Organiztaion Whistleblower Protection Public Procurement Healthcare Employment Education Whistleblower Public Procurement Healthcare Employment Education Youth Center Trebinje 1 1 1 2 3 Tender/Banja Luka 1 1 1 2 3 Stop Mobbing/Trebinje 1 1 1 2 3 Initiative and Civic Action (ICVA)/Sarajevo 1 1 3 2 1 3 Helsinki Committee for Human Rights of RS/Bijeljina 1 1 1 6 2 1 1 1 3 Center for Development of Youth Activism (CROA)/Sarajevo 1 1 1 4 2 1 3 JUSTICIA/Srebrenik 1 1 1 5 2 1 3 1 Centar za građansku saradnju/ Livno 1 1 1 3 2 1 3 Centar za razvoj civilnog društva (CRCD)/Doboj 1 1 4 2 1 3 1 1 Tolerancijom protiv različitosti (Topper)/ Doboj 1 1 1 2 3 49 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV SOS/Brčko 1 2 1 1 3 Vermont/Brčko 1 2 1 1 3 Foundation for Development and Democracy (FRD), Sarajevo 1 2 3 1 1 Center of Woman’s Rights, Zenica 1 2 3 1 1 ALDI/Gorazde 1 1 3 2 1 3 Topeer/Doboj 1 1 2 2 3 KAM/Zenica 1 2 1 1 3 TOTAL 2 1 3 2 2 3 2 3 3 3 9 39 With respect to the local level and assistance to the municipal and cantonal level governments, ACCOUNT has provided assistance to six CSOs to work on the development of local anti-corruption strategies (9 Sub￾awards). ACCOUNT has signed MOUs with 22 municipalities (Foca, Pale, Citluk, Lopare, Capljina, Ljubuski, Teslic, Usora, Lukavac, Ugljevik, Odzak, Ilijas, Breza, Neum, Domaljevac-Samac, Zavidovic, Banovic, Travnik, Visoko, Zepce, Vogosca, Bihac). Lead organizations have assisted the municipalities in developing anti-corruption strategies/action plans with at least 7 of those adopting action plans and 12 municipalities monitoring their implementation. Furthermore, ACCOUNT worked on the exchange of experiences among these municipalities by organizing annual Anti-Corruption Forums of Local Initiatives (2 Forums so far held). ACCOUNT also established cooperation on the cantonal level, most notably through tripartite MOUs between APIK, ACCOUNT and 8 cantonal anti-corruption teams to assist with the anti-corruption initiatives in those cantons. ACCOUNT organized a major conference for these cantons where sector-specific consultations were provided (also discussed above). The most intensive assistance, however, took place in the Sarajevo and Zenica￾Doboj Cantons, in particular, in relation to the implementation of initiatives undertaken by the healthcare and education sector leads. This assistance included implementation of anti-corruption rulebooks, developed within ACCOUNT, in a total of 12 healthcare institutions and the already mentioned implementation of the anti￾corruption classes in 14 elementary and high schools. The close collaboration with the anti-corruption team was additionally established in two respects: 1) assisting the Sarajevo and Zenica-Doboj Cantons in the development of an Anti-Corruption Strategy (initial or new); and 2) providing assistance to the cantonal public institutions in the development and implementation of Integrity Plans (in Sarajevo analyzing existing integrity plans and organizing 50 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV workshops on their implementation, while in Zenica-Doboj Canton providing trainings on both development and implementation of integrity plans). Finally, an additional activity implemented by ACCOUNT in cooperation with the RS Ministry of Justice relates to the implementation of the Integrity Plans for all public institutions in the RS. ACCOUNT provided technical assistance to the RS MoJ in developing training on how to prepare and implement Integrity Plans. The training was delivered to 950 representatives of 885 public institutions. Finally, ACCOUNT has been invited by the RS MoJ to participate in the new iteration of the RS Anti-Corruption Strategy (2018-2022). Within Component 3, which worked on raising public awareness through strategically targeted media campaigns, ACCOUNT’s activities focused primarily on production of investigative reports that would disclose issues of corruption; both on corrupt behavior in government and public institutions as well as through discussions on policy issues. Within this component, based on the ACCOUNT M&E Plan, they have produced and published over 765 investigative reports (in the first two years). Those include investigative stories published by Zournal.info as well as those prepared by the ACCOUNT established Media Pool. The ACCOUNT media pool has received financial support to produce investigative stories on corruption and, based on the ACCOUNT reports, a total of 14 media4 outlets participated at one point in the media pool and produced over 280 stories. In addition, ACCOUNT established the portal Interview.ba, which provides information on the prevention and fight against corruption in the format of interviews. The portal topics and interviews are based on the findings from the sector groups and project teams involved in monitoring and investigative reporting on anti-corruption. About 400 interviews were published by the third quarter of the third year of implementation. However, while the ACCOUNT M&E plan indicates that ACCOUNT output is on target, the Citizens’ awareness on the CSOs anti￾corruption initiatives has not changed. The National Survey of Citizens’ Perceptions 2015–2017, shows a negative result, with 2.7% of citizens being informed of CSOs anti-corruption initiatives in 2015 and only 1.8% in 2017. Furthermore, with respect to the number of citizens actually reporting corruption to ACCOUNT, the expected target in their M&E Plan is 100 reported cases and in the first 2 years, ACCOUNT achieved 1/5 of their target (22%). Based on ACCOUNT reports, there are no elements of corruption among the complaints received by citizens in a majority of these cases. Finally, within Component 4, relating to free legal aid assistance to victims of corruption, ACCOUNT has established a legal team that provides assistance to the victims and represents them in courts. Within the first two years, ACCOUNT provided assistance to the 30 cases reported to them and monitored the proceedings of those cases before the court. ACCOUNT has achieved a positive outcome in 13% of those cases. This component has direct links to C1 and C2, which are related to the whistleblower protection (WBP) sector activities. ACCOUNT reports that the legal team was involved in analyzing the existing legislation for the WBP, drafting of ACCOUNT recommendations and suggestions to the state and entity laws, as well as trainings for the sector group CSOs. In particular, the legal team trained those CSOs that provide free legal aid. Furthermore, ACCOUNT reports that they have improved the Legal Aid Registry functions and established cooperation with APIK on development of education courses for civil servants that would relate to WBP rules and procedures on reporting corruption and integrity plans. Finally, a series of open door sessions about ACCOUNT’s free legal aid assistance have been held primarily in partner municipalities. 4 (Moja Hercegovina, Tacno.net, Captial.ba, BUKA, e-trafica.net, Direkt-portal.com, Gerila.info, Frontal.ba, InfoRadar.ba, B1Infor.ba, slobodanvaskovicblog, Bljesak.info, RTV Zenica, and Analiziraj.ba) 51 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV IV. EVALUATION QUESTIONS The evaluation questions include: 1. To what extent has ACCOUNT increased CSO involvement and input regarding anti-corruption legislation/regulation? 2. What are ACCOUNT’s major achievements in selected five sector areas (public procurement, whistleblower protection, education, health and public employment) and on the local level? 3. To what extent has ACCOUNT increased quality and quantity of investigative journalism targeting public corruption in BiH? 4. To what extent have interventions under ACCOUNT influenced public awareness on corruption? 5. Is ACCOUNT legal aid model to protect individuals reporting corruption and whistleblowers perceived as effective by ACCOUNT beneficiaries (e.g. legal-aid recipients, beneficiaries of joint registry, and beneficiaries of the ACCOUNT legal team)? V. EVALUATION DESIGN AND METHODOLOGY The Evaluation team is expected to rely on a mixed-methods approach for the design of the ACCOUNT Performance Evaluation and draw on quantitative and qualitative data to inform and answer each evaluation question. The expected data sources should include the following: 9. ACCOUNT documents and ACCOUNT products (e.g., Investigative reports and Sector Group Monitoring Reports) 10. Semi-structured Key Informant Interviews 11. Online mini-survey among the ACCOUNT network members 12. Online mini survey with schools/teachers where anti-corruption curriculum was delivered 13. Panel discussion with media representatives (both ACCOUNT beneficiary and non-beneficiary) 14. National Survey of Citizens’ Perceptions 2015–2018. The data from above-listed sources will be triangulated and address the same evaluation questions from multiple perspectives. Comparing and contrasting data will help us to gain a more complete understanding of the issue and provide more confidence in the findings. The evaluation team should consult with up to 45 KIs from the following main stakeholder groups: a. ACCOUNT lead CSOs (10) b. Beneficiary BiH Government and Public institutions (7 total: 5 government and 2 public institutions) c. Beneficiary Municipalities and Cantons (2 cantons and 5 municipalities) d. ACCOUNT media pool members and non-assisted media (8 -12 panel discussion) 52 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV e. ACCOUNT assisted Whistle Blowers (4) f. USAID Agencies and USAID IPs (2) g. Other donors and international organizations active in the area of Anti-corruption support to BiH (3) The evaluation team should consider the evaluation matrix in Figure 3 for the data-sources and evaluation methods. 53 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 3. Evaluation Matrix EVALUATION QUESTIONS DATA SOURCES METHODOLOGY 1. To what extent has ACCOUNT increased CSO involvement and input regarding anti￾corruption legislation/regulation? 5) ACCOUNT reports 6) 36 Semi-structured KII of ACCOUNT implementers and beneficiaries (CSO and Government/Public institutions including municipalities, and donors) 7) Online mini-survey of ACCOUNT network members Mixed method triangulation based on: ▪ Desk research of ACCOUNT reports ▪ Qualitative analysis of semi￾structured KIIs ▪ Descriptive analysis on Online Survey findings 6. What are ACCOUNT’s major achievements in selected five sector areas (public procurement, whistleblower protection, education, health and public employment) and on the local level? 1) ACCOUNT reports 2) 48 KIs, with all stakeholder groups 3) Online mini-survey of up to 170 ACCOUNT network members (grantees and non-grantees) 4) Online mini survey with schools/teachers Qualitative analysis ▪ Desk research of ACCOUNT reports ▪ Qualitative analysis of semi￾structured KIIs Summary findings will be presented by sector areas. 7. To what extent has ACCOUNT increased quality and quantity of investigative journalism targeting public corruption in BiH? 1) ACCOUNT Reports 2) 48 KIs, with all stakeholder groups including Panel Discussion with up to 12 media representatives 3) Online mini-survey of up to 170 ACCOUNT network members (grantees and non-grantees) Mixed method triangulation based on: ▪ Desk research of ACCOUNT reports ▪ Qualitative analysis of semi￾structured KIIs and Panel Discussion KIs ▪ Descriptive analysis of online mini￾survey data 8. To what extent have interventions under ACCOUNT influenced public awareness on corruption? 7) ACCOUNT Reports, including investigative reports 8) 48 KIs, with all stakeholder groups including Panel Discussion with up to 12 media representatives 9) Online mini-survey of up to 170 ACCOUNT network members (grantees and non-grantees) 10) 4. National Survey of Citizens’ Perceptions data 2015–2018 Mixed method triangulation based on: ▪ Desk research of ACCOUNT reports ▪ Qualitative analysis of semi￾structured KIIs and Panel Discussion KIs ▪ Descriptive analysis of online mini￾survey data ▪ Descriptive analysis of National Survey of Citizens’ Perceptions data . 9. Is ACCOUNT legal aid model to protect individuals reporting corruption and whistleblowers perceived as effective by ACCOUNT beneficiaries (e.g., legal-aid recipients, beneficiaries of joint registry, and beneficiaries of the ACCOUNT legal team)? II. ACCOUNT Reports III. 48 KIs with all stakeholder groups including up to 4 individuals that received legal aid assistance form ACCOUNT IV. Online mini-survey of up to 170 ACCOUNT network members (grantees and non-grantees) Mixed method triangulation based on: ▪ Desk research of ACCOUNT reports; qualitative analysis of ACCOUNT intervention efforts as per Component 1 and Component 4 relating to whistleblower legislation (and perception of its quality) and direct legal aid assistance to individuals reporting corruption ▪ Qualitative analysis of semi￾structured KIIs ▪ Descriptive analysis of online mini￾survey data 54 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 2. DELIVERABLES, SCHEDULE,AND REPORTING REQUIREMENTS 1. Evaluation Design and Work Plan: A draft work plan and evaluation design document for the evaluation shall be submitted to USAID/BiH two weeks after SOW approval. The evaluation design will include: (1) a detailed evaluation design matrix (including the key questions, methods, and data sources used to address each question and the data analysis plan for each question); (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited; (4) known limitations to the evaluation design; and (5) a dissemination plan. The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the evaluation team delineated by roles and responsibilities. USAID offices and relevant stakeholders are asked to take up to one week to review and consolidate comments through the AOR/COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 3 days. 2. Data Collection: Key informant interviews are anticipated to last for five weeks and commence on October 23, 2018 and complete by November 20, 2018. The online survey and focus group discussions will be conducted during the same period. 3. In-briefing: Prior to conducting key informant interviews, the Evaluation team will have an in-briefing with the USAID/BiH Democracy Office to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan. 4. Evaluation Presentation: The evaluation team is expected to hold a presentation with a preliminary evaluation of their findings, conclusions and recommendations to USAID/BiH and discuss the summary of findings and recommendations with USAID pending an agreement with USAID. This is anticipated to occur two weeks after the completion of the fieldwork (December 7, 2018). 5. Draft Evaluation Report: The draft evaluation report will be submitted 7 weeks after the start of key informant interviews (December 15, 2018). The report shall be consistent with the USAID Evaluation Report Requirements provided in ADS REFERENCE 201MAH (USAID Evaluation Report Requirements https://www.usaid.gov/ads/policy/200/201mah) and take into account criteria to ensure the quality of the evaluation report specified in ADS REFERENCE 201MAA (https://www.usaid.gov/ads/policy/200/201maa). Once the initial draft evaluation report is submitted, USAID/BiH will have 10 calendar days in which to review and comment on the initial draft, and submit the consolidated comments to the evaluation team. The Evaluation Team will then be asked to submit a revised final draft report in 10 calendar days hence and, again, the USAID/BiH will review and send comments on this final draft report within 5 calendar days of its submission. 6. Final Evaluation Report: The Evaluation Team will be asked to take no more than 10 calendar days to respond/incorporate the final comments from USAID/BiH. The evaluation team leader will then submit the final report. All data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the activity or evaluation, and owned by USAID. 55 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX II: ACCOUNT SUB-AWARDS AND ROLES OF CSOS ACCOUNT Small Grants Awards by type and per Activity Components ACCOUNT Small Grants Awards by type and per Activity Components Small Grant Recipient Implementation Year CI - Sector-base Leading Organizations C2: Activity 2.2. Monitoring LEAD for sectors CI - Sector-member Organizations C2: Activity 2.2. Monitoring member for sectors C2: Activity 2.3. Anti￾corruption Strategies on local level. Total # of grants received per CSO Name of the Organization Whistleblower Public Procurement Healthcare Employment Education Whistleblower Public Procurement Healthcare Employment Education Whistleblower Public Procurement Healthcare Employment Education Whistleblower Public Procurement Healthcare Employment Education Youth Center Trebinje 1 1 1 1 1 1 5 2 0 3 0 Tender/Banja Luka 1 1 1 2 2 0 3 0 Stop Mobbing/Trebinj e 1 1 1 2 2 1 1 2 3 1 1 Initiative and Civic Action (ICVA)/Sarajevo 1 1 1 2 1 1 3 0 Helsinki Committee for Human Rights of RS/Bijeljina 1 1 1 1 1 4 2 1 1 2 3 1 1 1 1 1 1 3 56 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW -ON ACTIVITY USAID.GOV Center for Development of Youth Activism (CROA)/Sarajevo 2 1 1 1 1 1 5 3 1 1 JUSTICIA/Srebre nik 1 1 1 1 3 2 1 1 1 1 1 1 6 3 1 1 2 Centar za građansku suradnju/ Livno 1 1 1 1 1 4 2 1 1 1 1 4 3 1 1 Centar za razvoj civilnog društva (CRCD)/Doboj 1 1 1 1 1 4 2 1 1 1 1 4 3 1 1 2 UG "Za bolji Sanski Most" 1 1 1 2 1 1 2 3 0 SOS/Brčko 1 0 2 1 1 3 1 1 Vermont/Brčko 1 1 1 2 1 1 3 0 Foundation for Development and Democracy (FRD), Sarajevo 1 0 2 0 3 1 1 2 Center of Woman’s Rights, Zenica 1 0 2 1 1 3 1 1 57 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW -ON ACTIVITY USAID.GOV ALDI/Gorazde 1 1 1 1 3 2 1 1 1 1 1 5 3 1 1 Topeer/Doboj 1 1 1 1 1 4 2 1 1 3 1 1 KAM/Zenica 1 0 2 1 1 1 1 1 5 3 1 1 Međunarodni forum Bosna - Regionalni centar Tuzla 1 1 1 2 0 3 0 Udruženje roditlja - Ruka Ruci 1 1 1 2 0 3 0 UG Lovor Mostar 1 1 1 2 0 3 0 UG Oaza Trebinje 1 1 1 2 0 3 0 UG Inicijator, Sarajevo 1 1 1 2 2 0 3 0 UG REZ Tešanj 1 0 2 1 1 3 0 Prizma Brčko 1 1 1 58 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 2 0 3 0 Centar za mlade Kvart, Prijedor 1 0 2 1 1 3 1 1 Helsinški parlament građana Banja Luka 1 0 2 1 1 3 0 UG Resursni centar Brčko 1 0 2 1 1 3 1 1 2 Perpetuum Mobile, Banja Luka 1 0 2 1 1 3 0 GUŽ Duvanjke, Tomislavgrad 1 0 2 0 3 1 1 TOTAL 2 1 3 3 2 3 3 3 3 4 7 6 7 4 1 2 11 8 9 9 8 108 59 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX III: ACCOUNT M&E PLAN INDICATORS DETAILED PRESENTATION OF ACCOUNT MEL INDICATORS (ACTUALS AND TARGETS) BASED ON PIRS DOCUMENTATION Figure 1: Indicator I Indicator 1 Indicator Name Year Target Actual Total 1. Percentage of citizens participating in decision-making process 2016 40.60% 38.40% 2017 42.60% 34.00% 2018 44.70% 37.50% 2019 Source: ACCOUNT PIRS Figure 28: Indicator 1 - Total Figure 2: Indicator 2 Indicator 2 Indicator Name Year Target Actual Total 2. Percentage of citizens who believe that CSOs have influence over government decision￾making 2016 59% 60.60% 2017 61% 57.70% 2018 63% 59.16% 2019 65% Source: ACCOUNT PIRS 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 2016 2017 2018 2019 1. Percentage of citizens participating in decision-making process Target Actual 60 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 3: Indicator 2 Figure 4: Indicator 3 Indicator 3 Indicator Name and Disaggregation Categories Year Target Actual Total 3. Number of CSOs receiving USG assistance engaged in advocacy interventions for sectoral issues 2016 17 15 2017 21 19 2018 21 17 2019 21 Disaggregation I Protection of whistleblowers 2016 3 3 2017 3 3 2018 3 4 2019 3 Disaggregation II Public procurement 2016 2 3 2017 4 2 2018 4 4 2019 4 Disaggregation III Healthcare 2016 2 4 2017 2 3 2018 2 2 2019 2 Disaggregation IV Employment in public institutions 2016 3 2 2017 5 3 2018 5 5 2019 5 Disaggregation V Education 2016 3 3 2017 3 5 2018 3 2 2019 3 Disaggregation VI Local Level Anti-Corruption 2016 4 0 2017 4 3 2018 4 0 2019 4 Source: ACCOUNT BIHPERFORM Tracking Table 54% 56% 58% 60% 62% 64% 66% 2016 2017 2018 2019 2. Percentage of citizens who believe that CSOs have influence over government decision-making Target Actual 61 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 5: Indicator 3 - Total Figure 6: Indicator 3 - Disaggregation I Figure 7: Indicator 3 - Disaggregation II 0 5 10 15 20 25 2016 2017 2018 2019 3. Number of CSOs receiving USG assistance engaged in advocacy interventions for sectoral issues (Total) Target Actual 0 2 4 6 2016 2017 2018 2019 Disaggregation of 3. Number of CSOs receiving USG assistance engaged in advocacy interventions for sectoral issues (Protection of whistleblowers) Target Actual 0 1 2 3 4 5 2016 2017 2018 2019 Disaggregation of 3. Number of CSOs receiving USG assistance engaged in advocacy interventions for sectoral issues (Public procurement) Target Actual 62 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 8: Indicator 3 - Disaggregation III Figure 9: Indicator 3 - Disaggregation IV Figure 10: Indicator 3 - Disaggregation V 0 1 2 3 4 5 2016 2017 2018 2019 Disaggregation of 3. Number of CSOs receiving USG assistance engaged in advocacy interventions for sectoral issues (Healthcare) Target Actual 0 2 4 6 2016 2017 2018 2019 Disaggregation of 3. Number of CSOs receiving USG assistance engaged in advocacy interventions for sectoral issues (Employment in public institutions) Target Actual 0 2 4 6 2016 2017 2018 2019 Disaggregation of 3. Number of CSOs receiving USG assistance engaged in advocacy interventions for sectoral issues (Education) Target Actual 63 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 11: Indicator 3 - Disaggregation VI Figure 12: Indicator 4 Indicator 4 Indicator Name and Disaggregation Categories Year Target Actual Total 4. Number of policy proposals submitted to executive governments and parliaments with citizens’ input 2016 12 1 2017 23 25 2018 30 33 2019 40 Disaggregation I Accepted by government(s) 2016 1 0 2017 11 19 2018 15 14 2019 20 Disaggregation II Pending acceptance by government(s) 2016 11 1 2017 12 6 2018 15 19 2019 20 Source: ACCOUNT BIHPERFORM Tracking Table 0 2 4 6 2016 2017 2018 2019 Disaggregation of 3. Number of CSOs receiving USG assistance engaged in advocacy interventions for sectoral issues (Local Level Anti￾Corruption) Target Actual 64 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 13: Indicator 4 - Total Figure 14: Indicator 4 – Disaggregation I Figure 15: Indicator 4 – Disaggregation II 0 10 20 30 40 50 2016 2017 2018 2019 4. Number of policy proposals submitted to executive governments and parliaments with citizens’ input (Total) Target Actual 0 10 20 30 2016 2017 2018 2019 Disaggregation of 4. Number of policy proposals submitted to executive governments and parliaments with citizens’ input (Accepted by government(s)) Target Actual 0 10 20 30 2016 2017 2018 2019 Disaggregation of 4. Number of policy proposals submitted to executive governments and parliaments with citizens’ input (Pending acceptance by government(s)) Target Actual 65 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 16: Indicator 5 Indicator 5 Indicator Name and Disaggregation Categories Year Target Actual Total 5. Number of policy recommendations provided by Watchdog teams, Sector groups and Monitoring teams 2016 20 35 2017 20 19 2018 20 22 2019 20 Disaggregation I Protection of whistleblowers 2016 4 30 2017 4 4 2018 4 5 2019 4 Disaggregation II Public procurement 2016 4 0 2017 4 6 2018 4 6 2019 4 Disaggregation III Healthcare 2016 4 0 2017 4 3 2018 4 4 2019 4 Disaggregation IV Employment in public institutions 2016 4 5 2017 4 4 2018 4 4 2019 4 Disaggregation V Education 2016 4 0 2017 4 2 2018 4 3 2019 4 Source: ACCOUNT BIHPERFORM Tracking Table Figure 17: Indicator 5 - Total 0 10 20 30 40 2016 2017 2018 2019 5. Number of policy recommendations provided by Watchdog teams, Sector groups and Monitoring teams (Total) Target Actual 66 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 18: Indicator 5 – Disaggregation I Figure 19: Indicator 5 – Disaggregation II Figure 20: Indicator 5 – Disaggregation III 0 10 20 30 40 2016 2017 2018 2019 Disaggregation of 5. Number of policy recommendations provided by Watchdog teams, Sector groups and Monitoring teams (Protection of whistleblowers) Target Actual 0 2 4 6 8 2016 2017 2018 2019 Disaggregation of 5. Number of policy recommendations provided by Watchdog teams, Sector groups and Monitoring teams (Public procurement) Target Actual 0 2 4 6 2016 2017 2018 2019 Disaggregation of 5. Number of policy recommendations provided by Watchdog teams, Sector groups and Monitoring teams (Healthcare) Target Actual 67 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 21: Indicator 5 – Disaggregation IV Figure 22: Indicator 5 – Disaggregation V 0 2 4 6 2016 2017 2018 2019 Disaggregation of 5. Number of policy recommendations provided by Watchdog teams, Sector groups and Monitoring teams (Employment in public institutions) Target Actual 0 2 4 6 2016 2017 2018 2019 Disaggregation of 5. Number of policy recommendations provided by Watchdog teams, Sector groups and Monitoring teams (Education) Target Actual 68 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 23: Indicator 6 Indicator 6 Indicator Name and Disaggregation Categories Year Target Actual Total 6. Number of monitoring reports on sector￾based corruption 2016 5 0 2017 5 6 2018 5 7 2019 5 Disaggregation I Protection of whistleblowers 2016 1 0 2017 1 1 2018 1 1 2019 1 Disaggregation II Public procurement 2016 1 0 2017 1 2 2018 1 2 2019 1 Disaggregation III Healthcare 2016 1 0 2017 1 1 2018 1 2 2019 1 Disaggregation IV Employment in public institutions 2016 1 0 2017 1 1 2018 1 1 2019 1 Disaggregation V Education 2016 1 0 2017 1 1 2018 1 1 2019 1 Source: ACCOUNT BIHPERFORM Tracking Table Figure 24: Indicator 6 - Total 0 2 4 6 8 2016 2017 2018 2019 6. Number of monitoring reports on sector￾based corruption (Total) Target Actual 69 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 25: Indicator 6 – Disaggregation I Figure 26: Indicator 6 – Disaggregation II Figure 27: Indicator 6 – Disaggregation III 0 0.5 1 1.5 2016 2017 2018 2019 Disaggregation of 6. Number of monitoring reports on sector-based corruption (Protection of whistleblowers) Target Actual 0 0.5 1 1.5 2 2.5 2016 2017 2018 2019 Disaggregation of 6. Number of monitoring reports on sector-based corruption (Public procurement) Target Actual 0 0.5 1 1.5 2 2.5 2016 2017 2018 2019 Disaggregation of 6. Number of monitoring reports on sector-based corruption (Healthcare) Target Actual 70 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 28: Indicator 6 – Disaggregation IV Figure 29: Indicator 6 – Disaggregation V 0 0.5 1 1.5 2016 2017 2018 2019 Disaggregation of 6. Number of monitoring reports on sector-based corruption (Employment in public institutions) Target Actual 0 0.5 1 1.5 2016 2017 2018 2019 Disaggregation of 6. Number of monitoring reports on sector-based corruption (Education) Target Actual 71 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 30: Indicator 7 Indicator 7 Indicator Name Year Target Actual Total 7. Percentage of citizens informed of local CSOs anti-corruption activities 2016 5% 2.00% 2017 8% 1.80% 2018 10% 2.68% 2019 15% Source: ACCOUNT PIRS Figure 31: Indicator 7 - Total Figure 32: Indicator 8 Indicator 8 Indicator Name Year Target Actual Total 8. Number of investigative reports prepared by media partners (mechanisms for external oversight of public resource use supported by USG assistance) 2016 250 321 2017 300 544 2018 385 392 2019 415 Source: ACCOUNT BIHPERFORM Tracking Table 0% 5% 10% 15% 20% 2016 2017 2018 2019 7. Percentage of citizens informed of local CSOs anti-corruption activities Target Actual 72 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 33: Indicator 8 - Total Figure 34: Indicator 9 Indicator 9 Indicator Name Year Target Actual Total 9. Number of corruption cases reported to the Legal Team of ACCOUNT 2016 10 0 2017 20 22 2018 30 20 2019 40 Source: ACCOUNT BIHPERFORM Tracking Table Figure 35: Indicator 9 - Total Figure 36: Indicator 10 Indicator 10 Indicator Name Year Target Actual Total 10. Percentage of corruption cases with positive outcome 2016 20 13 2017 20 13 2018 20 21 2019 20 Source: ACCOUNT BIHPERFORM Tracking Table 0 200 400 600 2016 2017 2018 2019 8. Number of investigative reports prepared by media partners (mechanisms for external oversight of public resource use supported by USG assistance) Target Actual 0 10 20 30 40 50 2016 2017 2018 2019 9. Number of corruption cases reported to the Legal Team of ACCOUNT Target Actual 73 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 37: Indicator 10 - Total Figure 38: Indicator 11 Indicator 11 Indicator Name and Disaggregation Categories Year Target Actual Total 11. Number of corruption cases reported to the Legal Team for which legal assistance was provided 2016 5 15 2017 10 15 2018 15 19 2019 20 Disaggregation I Cases reported by men 2016 3 2017 6 13 2018 9 10 2019 12 Disaggregation II Cases reported by women 2016 2 2017 4 2 2018 6 9 2019 8 Source: ACCOUNT BIHPERFORM Tracking Table Figure 39: Indicator 11 - Total 0 5 10 15 20 25 2016 2017 2018 2019 10. Percentage of corruption cases with positive outcome Target Actual 0 5 10 15 20 25 2016 2017 2018 2019 11. Number of corruption cases reported to the Legal Team for which legal assistance was provided (Total) Target Actual 74 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 40: Indicator II – Disaggregation I Figure 41: Indicator II – Disaggregation II 0 5 10 15 2016 2017 2018 2019 Disaggregation of 11. Number of corruption cases reported to the Legal Team for which legal assistance was provided (Cases reported by men) Target Actual 0 5 10 2016 2017 2018 2019 Disaggregation of 11. Number of corruption cases reported to the Legal Team for which legal assistance was provided (Cases reported by women) Target Actual 75 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX IV: LIST OF KEY INFORMANTS 76 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX V: QUALITY OF INVESTIGATIVE REPORTS PRODUCED WITHIN ACCOUNT - EVALUATION Based on ACCOUNT Documentation and list of investigative reports produced during the first three years of implementation a random sampling has been conducted and a total of 30 Articles evaluated against nine quality criteria, as presented in Table 1 of this Annex. Each criteria was scored on a scale of 1 to 5, with 1 being poor quality and 5 excellent quality. The evaluators were also asked to provide brief justification for each score (see the Figure 1). Finally Each Article is awarded an overall score which represents the average of scores for each criteria. Figure 1: Criteria and scoring for quality of investigative reports SCORING CARD Criteria Score: Excellent [5]; Very good [4]; Good [3]; Fair [2]; Poor [1]; Missing [0] Comments on score (Explain why you gave a specific score) Fairness (Story includes many viewpoints, it is thorough, moderate in tone, includes all facts or details that would influence readers, even those that tend to disprove the thesis of the story) Accuracy (Story provides not only right and accurate facts but also deep and complete context. The author seeks to accurately reflect the tone of interviews and quotes. There is no usage of hearsay, innuendo or rumor) Clarity (Story is written in simple words and language, well organized with beginning, middle and end, respecting the importance of context in a way that even the most casual reader should be able to follow and understand the complex issues of the story) Public Interest (Story seeks to represent interests of readers/listeners and to express their voice. Story honors the belief that people have a right to know) Respect (Civil tone of respect is evident in the produced story. There is no usage of clever, funny, sarcastic or bitter comments. Story is told unvarnished and supported by documents and interviews, uses moderate tones and does not incite people to violent emotions) Investigative Approach (How author combines different elements of investigative journalism, such as facts, sources, background, analysis, ethical standards; Is there anything news in the story; Any part of the story that is unexplored ) 77 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Advocacy (Story is written in a manner that respects the advocacy approach; it provides fair and accurate information allowing the mechanisms of civil society to act whether it is the people, the courts, the police or the government. The author does not take a stand of prosecutor, judge or the jury) Ethical Conduct (There is no conflict of interest, not even the appearance of a conflict of interest. The story pursues highest ethical standards as defined by the Press Code BiH) Rights (Story honors and defends the rights of free speech and open access to public information, open and transparent government and public institutions, right to a free and open press) Average Score (the sum of all criteria scores/9) 0 Provide general comment on the journalistic piece The sampling of reports is based on the list of investigative reports provided by ACCOUNT with links to each report. The list included Zurnal.info investigative reports produced within Year (Y) 1, Y2, and Y3, and investigative articles produced by the Media Pool member outlets, through different types of financial support mechanisms awarded by ACCOUNT. In the first three years of the implementation, ACCOUNT has awarded financial support in in four cycles (C1 – C4). The data presented is aligned with the US Fiscal year period (October to September), according to which three rounds of awards have been distributed in the second Year of implementation. For the number of articles produced by zurnal.info and each of the Media Pool member outlet within the awarding cycles, please see the Figure 2: Number of investigative reports produced within ACCOUNT. 78 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 2: Number of investigative reports produced within ACCOUNT ACCOUNT Y1-Y3 and C1-C4 number of investigative reports Total number of investigative reports per year Only for the Media Pool Cumulative Y1-Y3 Occurring in Y2 19 Awards total Occurring in Y3 11 Awards total Media Outlet Y1 Y2 Y3 C1 C2 C3 C4 Zurnal.info 316 409 308 N/A N/A N/A N/A 1033 Analiziraj.ba 75 0 55 20 75 B1 0 5 5 5 Bljesak Info 6 0 3 3 6 Blog Slobodan Vaskovic 0 7 7 7 Buka 11 7 5 5 1 7 18 Capital.ba 10 6 4 5 1 6 16 Direkt 0 13 13 13 E Trafika 8 5 4 4 5 13 Frontal 0 10 10 10 Gerila 0 10 10 10 Inforadar 0 12 12 12 Moja Hercegovina 13 5 2 8 3 5 18 RTV Zenica 5 0 2 3 5 Tacno.net 4 5 2 2 5 9 Total Media Pool per Y/C N/A 132 85 77 46 9 85 1250 ACCOUNT TOTAL 316 541 393 N/A N/A N/A 85 Source for data: List of investigative reports submitted to the Evaluation Team by ACCOUNT IPs on November 19, 2018. The sampling has been conducted in the following manner: 1. For Zurnal.info which has produced the highest number of investigative reports on annual basis, a random sample of five (5) written reports from annual population of the reports has been extracted. Zurnal.info also produced a 20 investigative videos in the three year period published on YouTube channel. We extracted all 2 videos from the annual reports on and made randomly selected one for review. 2. For the Media Pool outlets, the overall cumulative production of reporting is five (5) times smaller than that of Zurnal.info alone, and with a high level differences (the minimum is 5 reports while the maximum is 75), however one randomly sampled article has been extracted for each media outlet. Please see the Figure 3 on sampling and reports population. 79 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 3: ACCOUNT Media Report Sampling SAMPLING Zurnal.info Reporting Period Report format Population Sample Y1 Written Reports 302 5 Y2 Written Reports 403 5 Y3 Written Reports 308 5 Y1 -Y3 Video Reports 20 1 Media Pool Reporting Period Media outlet - any format Population Sample Y1- Y3 Analiziraj.ba 75 1 B1 5 1 Bljesak Info 6 1 Blog Slobodan Vaskovic 7 1 Buka 18 1 Capital.ba 16 1 Direkt 13 1 e-Trafika 13 1 Frontal 10 1 Gerila 10 1 Inforadar 12 1 Moja Hercegovina 18 1 RTV Zenica 5 1 Tacno.net 9 1 Table 4 presents the 30 investigative reports that have been extracted through the sampling technique for evaluation on the nine scoring criteria presented in Figure 1, along with the overall average score per each report assigned by two evaluators in Figure 4 80 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 4: List of sampled investigative reports and scores LIST OF SAMPLED INVESTIGATIVE REPORTS EVALUATED FOR QUALITY AND OVERALL AVARAGE SCORE PER ARTICLE BY EACH EXPERT ZURNAL.INFO Date publish ed Report title Link to report Evalua tior 1 Evalu ator 2 Avar age 4-Aug￾2016 POZADINA SUKOBA SALIHOVIĆ - MEKTIĆ: Kako je televizijski voditelj prekinuo istragu protiv Milorada Dodika http://zurnal.info/novost/ 19943/kako-je￾televizijski-voditelj￾prekinuo-istragu-protiv￾milorada-dodika 1.8 3.9 2.8 5-Sep￾2015 DUG OD 33 MILIONA EURA: Bankrotirao strateški partner "Bosnalijeka" u Rusiji! http://zurnal.info/novost/ 19250/dug-od-33- miliona-eura￾bankrotirao-strateski￾partner-bosnalijeka-u￾rusiji 0.6 3.2 1.9 11-Jul￾2016 21. GODIŠNJICA U SREBRENICI: Negiranje genocida uprkos više od 600 godina zatvora https://www.zurnal.info/n ovost/19908/negiranje￾genocida-uprkos-vise-od￾600-godina-zatvora 3.4 4.2 3.8 25-Mar￾2016 0ČITAONICA ŽURNAL KRVNIKOV TRAG: Nepoznati detalji iz lova na Radovana Karadžića http://www.zurnal.info/no vost/19731/krvnikov￾trag-nepoznati-detalji-iz￾lova-na-radovana￾karadzica 3.8 0.0 1.9 27-Mar￾2016 OBRAZOVANJE ODRASLIH U RS: Sa profesorima iz Širokog brijega trogodišnju školu završićete za tri mjeseca http://www.zurnal.info/no vost/19733/obrazovanje￾odraslih-u-rs-sa￾profesorima-iz-sirokog￾brijega-trogodisnju-skolu￾zavrsicete-za-tri-mjeseca 1.8 3.7 2.7 10-Aug￾2017 PRINUDNE NAPLATE U PRAVOSUĐU: Sudski vještaci tužbama naplaćuju svoje honorare http://zurnal.info/novost/ 20620/sudski-vjestaci￾tuzbama-naplacuju-svoje￾honorare 2.3 3.2 2.8 15-Jun￾2017 SLUČAJ DOKTORICE GORDANE ŠOPALOVIĆ: Patološke igre oko specijalizacije za patologe http://www.zurnal.info/no vost/20526/patoloske￾igre-oko-specijalizacije￾za-patologe 3.7 3.7 3.7 3-Dec￾2016 RASPRAVA U PARLAMENTU: Da li je sirotom poslaniku malo 1.000 maraka za telefonski račun? http://www.zurnal.info/no vost/20173/da-li-je￾sirotom-poslaniku-malo￾1.000-maraka-za￾telefonski-racun 5.0 4.3 4.7 81 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 14-Jun￾2017 KADA SRETNETE PATRIOTU: Stavite novčanik u unutrašnji džep http://www.zurnal.info/no vost/20521/stavite￾novcanik-u-unutrasnji￾dzep 2.3 3.7 3.0 14-Feb￾2017 ELEKTROPRIVREDA PO STAROM: Javna nabavka na kojoj može pobijediti samo Bekto Precisa http://www.zurnal.info/no vost/20293/javna￾nabavka-na-kojoj-moze￾pobijediti-samo-bekto￾precisa 4.3 5.0 4.7 23-Oct￾2017 OPLJAČKANA ARHIVA OPĆINSKOG SUDA: Ukrali 2000 spisa i prodali ih za reciklažu http://zurnal.info/novost/ 20749/ukrali-2000-spisa￾i-prodali-ih-za-reciklazu 3.0 3.3 3.2 1-Mar￾2018 GRADONAČELNIK ZENICE ŽELI SAMO POZITIVNE PRIČE: Fuad Kasumović namjerava bojkotovati plaćanje RTV takse http://zurnal.info/novost/ 20997/fuad-kasumovic￾namjerava-bojkotovati￾placanje-rtv-takse 2.1 3.3 2.7 10-Mar￾2018 KUKIĆEV OBRAČUN SA ELEKTROPRIVREDOM BIH: Banovići smanjili isporuku uglja, ugašen Blok 6 Termoelektrane Tuzla http://zurnal.info/novost/ 21013/banovici-smanjili￾isporuku-uglja-ugasen￾blok-6-termoelektrane￾tuzla￾2.4 3.4 2.9 13-Mar￾2018 PREMIJER NOVALIĆ NEMA NIŠTA PROTIV: Direktor Mahmutović privatizirao Financijsko-informatičku agenciju http://zurnal.info/novost/ 20992/direktor￾mahmutovic-privatizirao￾financijsko-informaticku￾agenciju 2.1 3.9 3.0 16-Nov￾2017 SOCIJALNI SLUČAJ MINISTRA MIHALJEVIĆA: Pored stipendija,na prevaru obezbijedili i mjesta u domu! http://zurnal.info/novost/ 20794/pored￾stipendijana-prevaru￾obezbijedili-i-mjesta-u￾domu￾2.7 4.6 3.6 12-Dec￾2016 AFERA - TV ŽURNAL: Srebrenica - Video https://www.youtube.co m/watch?v=TVEDQWFX pR8&t=638s 4.9 5.0 4.9 MEDIA POOL Date publish ed Title Media outlet Link to report Evalua tior 1 Evalu ator 2 Avar age 16-Sep￾2016 RTV USK I NAŠA TV: DALEKO OD STVARNIH PROBLEMA Analizirajba http://analiziraj.ba/20 16/09/16/rt-usk￾nasa-tv-daleko-od￾stvarnih-problema/ 2.3 3.8 3.1 9-Aug￾2018 JELŠINGRAD – STEČAJ I PRODAJA PO SVAKU CIJENU B1 http://b1info.ba/jelsin grad-stecaj-i￾prodaja-po-svaku￾cijenu/ 2.7 5.0 3.8 82 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 23-Feb￾2017 Liječnici koriste i strah da pacijentima naplate što bi u bolnici bilo besplatno Bljesak.info http://bljesak.info/r ubrika/vijesti/clanak /lijecnici-koriste-i￾strah-da￾pacijentima-naplate￾sto-bi-u-bolnici-bilo￾besplatno/188046 5.0 4.9 4.9 11-Jun￾2018 EIB protiv IRB/Dosje: Kako su Tegeltija, Džombić, Vujnić... oteli i oprali kreditna sredstva dobijena od EIB-a preko “Energolinije” i kako su pokušali zataškati pljačku Slobodan Vaskovic Blog http://slobodanvask ovic.blogspot.com/2 018/06/eib-protiv￾irbdosje-kako-su￾tegeltija.html 2.6 4.4 3.5 11-Oct￾2016 ZA SVE NEPODOBNE RADNIKE: Prijaviš korupciju, pa te vozaju u gepeku! Buka http://www.6yka.co m/novost/114944/z a-sve-nepodobne￾radnike-prijavis￾korupciju-pa-te￾vozaju-u-gepeku￾5.0 4.6 4.8 23-Feb￾2017 Kome zvižde zviždači – Otkaz za otkrivanje kriminala u Zaštitnom fondu RS Capital http://www.capital. ba/kome-zvizde￾zvizdaci-otkaz-za￾otkrivanje-kriminala￾u-zastitnom-fondu￾rs/ 5.0 4.7 4.8 10-Jul￾2018 HET daje više od 35.000 maraka za medijsku promociju Direkt-portal https://www.direkt￾portal.com/het-daje￾vise-od-35-000- maraka-za-medijsku￾promociju/ 3.3 4.4 3.9 2-Mar￾2017 Kako su sistematski obesmišljeni školski konkursi u RS? eTrafika.net http://www.etrafika.n et/drustvo/50586/ka ko-su-sistematski￾obesmisljeni-skolski￾konkursi-u-rs/ 5.0 4.1 4.6 8-May￾2018 Kako su sistematski obesmišljeni školski konkursi u RS? Frontal.ba http://www.frontal.b a/novost/91967/prop ast-banjalucke￾privrede-1.-dio￾fabrike-propale￾nekretnine￾razgrabljene 3.3 4.4 3.9 15-May￾2018 Politika u zdravstvu Republike Srpske (prvi dio) Gerila.info http://www.gerila.inf o/drustvo/politika-u￾zdravstvu-republike￾srpske/ 2.4 4.1 3.3 3-Apr￾2018 IZETBEGOVIĆ, ČOVIĆ I IVANIĆ U STRAHU OD IZBORNOG PORAZA: Predsjedništvo BiH vrši “sječu” ambasadora! Inforadar.ba http://inforadar.ba/i zetbegovic-covic-i￾ivanic-u-strahu-od￾izbornog-poraza￾predsjednistvo-bih￾vrsi-sjecu￾ambasadora/ 1.3 2.8 2.1 83 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV 21-Feb￾2017 Zapošljavanje u Opštini Gacko: Procedure spotakle konkurs Moja Hercegovina http://mojahercegov ina.com/zaposljavan je-u-opstini-gacko￾procedure-spotakle￾konkurs/ 2.9 3.6 3.2 7-Oct￾2016 BORBA PROTIV KORUPCIJE RTV Zenica https://www.youtube .com/watch?v=Sg59 mqy5MlA 2.9 3.8 3.3 23-Jul￾2018 Vranica – Ratni zločin u ratu, pravosudni u miru Tacno.net http://www.tacno.n et/mostar/vranica￾ratni-zlocin-u-ratu￾pravosudni-u-miru/ 2.8 4.4 3.6 Figure 5: ACCOUNT Investigative reporting scores per category ACCOUNT INVESTIGATIVE REPORTING AVERAGE SCORES PER CATEGORY Excellent [5]; Very good [4]; Good [3]; Fair [2]; Poor [1] ZURNAL MEDIA POOL ACCOUNT TOTAL Expert 1 Expert 2 Average Expert 1 Expert 2 Average Expert 1 Expert 2 Average Number of reports evaluated 16 16 16 14 14 14 30 30 30 Fairness 1.94 3.31 2.63 2.79 4.07 3.43 2.33 3.67 3.00 Accuracy 2.69 3.31 3.00 3.57 3.86 3.71 3.10 3.57 3.33 Clarity 3.25 3.38 3.31 3.21 3.86 3.54 3.23 3.60 3.42 Public interest 3.50 4.19 3.84 3.36 4.43 3.89 3.43 4.30 3.87 Respect 2.88 3.94 3.41 3.43 4.36 3.89 3.13 4.13 3.63 Investigative approach 2.63 2.56 2.59 3.36 3.71 3.54 2.97 3.10 3.03 Ethical Conduct 3.25 4.38 3.81 3.50 5.00 4.25 3.37 4.67 4.02 Rights 3.00 3.75 3.38 3.36 4.29 3.82 3.17 4.00 3.59 Advocacy 2.88 4.06 3.47 3.36 4.36 3.86 3.10 4.20 3.65 Overall average score 2.89 3.65 3.27 3.33 4.21 3.77 3.09 3.91 3.50 84 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX VI: ACCOUNT GOOGLE ANALYTICS FOR INVESTIGATIVE REPORTING ZURNAL.INFO AND INTERVIEW.BA DATA BASED ON THE ACCOUNT GOOGLE ANALYTICS - AUDIENCE REPORTS Note: Google Analytics audience reports are not always reliable for the user flow analysis of the specific website. According to the Analytics Edge website and published article “Misunderstood Metrics: New vs Returning Visitors” there are certain calculation problems in Google Analytics methodology to determine who are the real New or Returning Visitors to the website (Source: https://help.analyticsedge.com/article/misunderstood-metrics￾new-vs-returning-visitors/ ). Also the Google Analytics has announced a proclamation in February 2018 which defines their point of view on the reported calculation problems, and sets a 2-year expiration date on New Visitors: “If someone has visited our website within the past two years and returns from the same device, they are marked as a Returning Visitor in our Google Analytics. If it has been more than two years since someone has visited our site, the next time they return they will be counted as a New Visitor again.” (Source: https://www.dbswebsite.com/blog/2016/10/04/google-analytics-made-easy-new-visitors-vs-returning-visitors/ ). Although there are some reliability issues with the data presented thogh google analytics, the data is still sufficiently confident to estimate the increase of visits to the website, and the readership. Following is the presentation of data for Zurnal.Info and Interview.ba based on the data reported within ther Y1-Y3 Annual Reports. Figure 1: Zurnal.info Audience Report Data I Web Site Year Sessions Users Page Views Pages/Sessions www.zurnal.info 2016 1,191,917 671,798 1,822,590 1.53 2017 2,235,453 879,051 3,640,535 1.63 2018 2,910,771 941,842 4,772,485 1.64 Figure 2: Zurnal.info Audience Report Data II Web Site Year Avg. Session Duration Bounce Rate New Sessions New Visitor Returning Visitor www.zurnal.info 2016 0:01:17 79.87% 54.84% 54.9% 45.1% 2017 0:01:23 75.43% 37.11% 37.1% 62.9% 2018 0:01:23 74.67% n/a 74.5% 25.5% Figure 3: Interview.ba Audience Report Data Web Site Year Sessions Users Page Views Pages/Sessions Avg. Session Duration Bounce Rate Interview.ba 2016 9,812 8,253 15,415 1.57 0:01:19 84.98% 2017 n/a 63,189 90,942 n/a n/a n/a 2018 n/a 198,000 342,000 n/a n/a n/a 85 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 4: Audience Report Data Definitions (Source: The Ultimate Google Analytics Glossary) Sessions Total number of visits to a website, consisting of one or more page views Users Total number of persons browsing a website Page Views Total number of pages which have been viewed by users on a website Pages/Sessions Pages per session. Average number of page views in each session Avg. Session Duration Average session duration. Provides a top-level view of how long users are spending on Interview.ba (hrs, min, sec) Bounce Rate The percentage of visitors to a particular website who navigate away from the site after viewing only one page New Sessions The percentage of new sessions on a website New Visitor The percentage of new visitors on a website Returning Visitor The percentage of returning visitors on a website Figure 5: Zurnal.info Audience Report Data 0 1,000,000 2,000,000 3,000,000 4,000,000 5,000,000 6,000,000 2016 2017 2018 Zurnal.info - # of ssesions, users and page views Sessions Users Page Views 86 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Figure 6: Zurnal.info Audience Report Data Figure 7: Interview.ba Audience Report Data 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 2016 2017 2018 Zurnal.info - bounce rate and % of new/returnig vistors Bounce Rate New Visitor Returning Visitor 0 100,000 200,000 300,000 400,000 2016 2017 2018 Interview.ba - # of ssesions, users and page views Users Page Views 87 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX VII: MAIN DATA COLLECTION INSTRUMENTS INTERVIEW GUIDE FOR USAID and IMPLEMENTING PARTNERS My name is and these are my colleagues . We are the team of researchers working for MEASURE-BiH. First of all, we want to thank you for setting aside the time for this conversation. As you know, USAID/BiH has tasked MEASURE-BiH with conducting an independent evaluation of the USAID/BiH Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) Follow-on Activity. The four-year ACCOUNT program started implementation across Bosnia and Herzegovina in July 2015 and is anticipated to end in July 2019. The aim of the program is to create an environment that increases civil society participation and reforms through strong collaboration and cooperation in anti-corruption initiatives. Within this evaluation, we will conduct semi-structured interviews with around 45 ACCOUNT stakeholders. Your views and opinions on the Activity are profoundly important for this evaluation. Through these KIIs, we are looking to gain insights into the program implementation, the challenges faced along the way and the influence of specific interventions. We would also like to hear about stakeholders’ perceptions, lessons learned and recommendations for any possible future donor/government interventions relevant to civil society participation in anti-corruption initiatives and reforms. The information you provide will be combined with information provided from other stakeholders. Your comments are confidential and you will not be identified by name in any report. will be taking notes while we talk. With your permission, we would also like to record this session. The reason we are recording is that it is quite hard to actively participate in a conversation and take notes at the same time. Another reason is that we want to analyze the interviews using objective methods and avoid any biasrelated to quality of notes and the capacity of interviewers’ memory. Do we have your permission to begin recording? Please do not hesitate to mention anything that you find important or that you might find relevant to ask. Do you have any questions before we start? CSO INVOLVEMENT AND INPUT REGARDING ANTI-CORRUPTION LEGISLATION/REGULATION? Background: Can you give us a brief overview of the environment three years ago when ACCOUNT started to operate? o PROBE: What was the role of civil society in shaping the anti-corruption legislation/regulation? o PROBE: Could you please elaborate on the distinction between the ACCOUNT network and the ACCOUNT Activity, and the timeframe for both? o PROBE: To your knowledge, is the ACCOUNT network supported by other donors? If so, do you know what that support is or was? Could you please elaborate on how the design of the Activity was developed? 88 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV o PROBE: What are the main objectives and theory of change? What is the interaction between the four main objectives with the four components of the projects? o PROBE: The program description designates for each objective one implementation component with specific measures to be taken. However, ACCOUNT reports indicate high interaction of the components with all objectives. Could you please elaborate on this interaction? Could you please elaborate how the implementation of the Activity was organized? o PROBE: What are the responsibilities of the Implementing Partners toward achieving the stated objectives? o PROBE: What are the responsibilities of the sub-awarded organizations toward achieving the Activity objectives? Which main activities and mechanisms were envisioned to increase CSOs’ involvement and input to the government on anti-corruption legislation/regulation? o PROBE: How Does the expected increase of CSOs’ capacities to conduct higher quality of evidence based research, policy proposals, and monitoring in the ACCOUNT selected sectors relate to CSOs’ increased participation in decision making process? o PROBE: How did the information sharing on research and monitoring results among the CSOs occur? How did the different “lead” interact with one another? o PROBE: What would you say was achieved over the past three years in the capacity of CSOs to conduct evidence-based research on anti-corruption strategies/legislation/regulation/policy? What were the main challenges of the implementation? The ACCOUNT program description states that they will provide sub-awards to 50 CSOs. However, based on the ACCOUNT-adopted MEL Plan, their target is to provide sub-award to 21 CSOs? Was this change approved by USAID and in which format? THE MAIN ACHIEVEMENTS IN SELECTED FIVE SECTOR AREAS (PUBLIC PROCUREMENT, WHISTLEBLOWER PROTECTION, EDUCATION, HEALTH AND PUBLIC EMPLOYMENT AND LOCAL LEVEL ANTI-CORRUPTION INITIATIVES)? ACCOUNT has worked in five sector areas providing input on anti-corruption legislation/regulation and providing information on monitoring of implementation of existing policies and processes relevant to the five sectors. How were the five sectors for intervention defined? What was the status in those sectors prior to ACCOUNT? Provided that ACCOUNT worked in five sectors and the results are primarily expected in two sectors, whistleblower protection and public procurement legislation, could you please explain the purpose for this focus? To what degree, if at all, did this affect the implementation process with regards to all five sectors? Has that been intentional by design? What were the main implementation challenges in each sector area? How were those addressed? The ACCOUNT program description describes assistance on the local level. ACCOUNT worked with 22 municipalities assisting them in development of anti-corruption strategies and action plans and worked with 8 cantons on their anti-corruption initiatives. Could you please explain if this was originally envisioned by the Activity and what mechanisms for implementation have been used for these two levels? 89 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ACCOUNT activities relating to the development and implementation of integrity plans in RS, Canton Sarajevo and Canton Zenica’s public institutions were not originally envisioned by the program description. Please elaborate how assistance in this area evolved? o PROBE: What is or was ACCOUNT’s role in this process? o PROBE: How does this Activity relate to the sector-network’s objectives? o PROBE: How did the ACCOUNT network participate in this process? o PROBE: To what extent did ACCOUNT coordinate with other international donors (including USAID) on this matter? QUALITY OF INVESTIGATIVE JOURNALISM ABOUT CORRUPTION What was the main motivation for the work related to investigative reports? What was the main expectation from these activities? How do they relate to the five sectors and the CSOs’ input to government institutions? What was the status of the quantity and quality of media reporting prior to ACCOUNT? What has been achieved so far? How would you rate the current quality of reporting produced by zurnal.info, ACCOUNT Media Pool, and interview.ba? How many media outlets participate in the ACCOUNT media pool? What type of technical assistance did they receive? o PROBE: The ACCOUNT reports mention the following media outlets as the ACCOUNT media pool: Moja Hercegovina, Tacno.net, Captial.ba, BUKA, e￾trafica.net, Direkt-portal.com, Gerila.info, Frontal.ba, InfoRadar.ba, B1Infor.ba, slobodanvaskovicblog, Bljesak.info, RTV Zenica, and Analiziraj.ba. However, not all of these same media outlet are mentioned every year? Why is this the case? PUBLIC AWARENESS OF CORRUPTION Could you please elaborate for us what is the guiding definition of “public awareness of corruption” that the Activity applied? What was the motivation for this definition? What was the public’s awareness prior to ACCOUNT? o PROBE: What was the focus: increasing the general perception that corruption exists or increasing the understanding of corruption and its mechanisms? o PROBE: What would you say influences public perception (e.g. media, CSOs, individual experiences)? What is your impression of the implementation of these activities and the results they achieved? Is the public informed about or aware of corruption in ACCOUNT’s five-sector areas? What were the main challenges during the implementation? Are you aware of specific cases or circumstances where ACCOUNT media has reported on a corruption issue that has reached a wide audience? What was the outcome of that awareness? Public perception is that corruption in the public sector is very high. However public perception on anti-corruption initiatives undertaken by CSOs are very low. How would you explain the relationship between these two issues? 90 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Based on your experience in the implementation of ACCOUNT, did you observe any change in the public’s readiness to report corruption or engage in anti-corruption initiatives? Please elaborate on your opinion. Why or why not? What, in your opinion, are the main factors influencing the public’s readiness and reaction to corruption? LEGAL AID MODEL TO PROTECT INDIVIDUALS REPORTING CORRUPTION AND WHISTLEBLOWERS The ACCOUNT Component on establishing a free legal aid mechanism for victims of corruption and whistleblowers protection is a separate component yet it seems very closely linked to results related to component 1 and 2 of the program description. Could you please elaborate on this interconnection and what type of activities the ACCOUNT Legal Aid team had envisioned carrying out? What were the main challenges in implementing the free legal aid mechanism and the work of the ACCOUNT legal aid team? Could you please also elaborate on what changes were envisioned for the free legal aid online registry www.bpp.ba (Besplatna Pravna Pomoc)? 91 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV INTERVIEW GUIDE FOR ACCOUNT NETWORK MEMBERS AND SMALL GRANTS RECIPIENTS/LEAD CSOs My name is and these are my colleagues . We are the team of researchers working for the MEASURE-BiH. First of all, we want to thank you for setting aside the time for this conversation. As you know, USAID/BiH has tasked MEASURE-BiH with conducing an independent evaluation of the USAID/BiH Anti-Corruption Civic Organizations’’ Unified Network (ACCOUNT) Follow-on Activity. The four-year ACCOUNT Program started implementation across Bosnia and Herzegovina in July 2015 and is anticipated to end in July 2019. The aim of the program is to create an environment that increases civil society participation and reforms through strong collaboration and cooperation in anti-corruption initiatives. Within this evaluation, we will conduct semi-structured interviews with around 45 ACCOUNT stakeholders. Your views and opinions on the Activity are profoundly important for this evaluation. Through these KIIs, we are looking to gain insights into the program implementation, the challenges faced along the way, and the influence of specific interventions. We would also like to hear about stakeholders’ perceptions, lessons learned and recommendations for any possible future donor/government interventions relevant to civil society participation in anti-corruption initiatives and reforms. The information you provide will be combined with information provided from other stakeholders. Your comments are confidential and you will not be identified by name in any report. will be taking notes while we talk. With your permission, we would also like to record this session. The reason we are recording is that it is quite hard to actively participate in a conversation and take notes at the same time. Another reason is that we want to analyze the interviews using objective methods and avoid any biasrelated to quality of notes and the capacity of interviewers’ memory. Do we have your permission to begin recording? Please do not hesitate to mention anything that you find important or that you might find relevant to ask. Do you have any questions before we start? CSO INVOLVEMENT AND INPUT REGARDING ANTI-CORRUPTION LEGISLATION/REGULATION? Background: Can you give us a brief overview of how civil society shaped anti-corruption legislation or regulation three years ago before ACCOUNT? o PROBE: Did your organization provide input to any anti-corruption policy, legislation, or regulation? Could you please provide an example? What was your role in the ACCOUNT network? o PROBE: What type of assistance did you receive through ACCOUNT; small grant for advocacy, monitoring, anti-corruption strategies at the local level or other? o PROBE: Could you please elaborate on your main responsibilities within ACCOUNT? What type of assistance did you receive to fulfill those responsibilities (other than financial)? 92 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV o PROBE: What is the role of non-lead organizations that participate in: 1) sector￾group; and 2) watch-dog teams? Could you please describe your involvement within ACCOUNT in providing input to the government in anti-corruption initiatives, legislation, or regulation? Has your involvement in scope and effort changed since being a part of ACCOUNT? Please elaborate? o PROBE: Has evidence-based research and monitoring in the ACCOUNT selected sectors increased CSOs’ activity and input to government for anti-corruption strategies remained the same, increased, or decreased? To what extent would you say it is different than three years ago? If not, why not? o PROBE: Is your organization more involved in providing input to the government on anti-corruption legislation or regulation? Could you please explain what are the most frequently used methods to do so (public consultations, meetings, cooperation with the institutions, lobbying activities with the Members of the Parliament, etc). o PROBE: How would you say ACCOUNT contributed to your involvement with government? Is it in any way different than it was before? What is different? How did this change come about? How would you describe your interaction with the government or public institutions on the State, Entity, Cantonal, or City/Municipality level? o PROBE: Would you say that the government is taking into account CSOs’ recommendations in their development of policy (including legislation, regulation, by-laws, rules, or practices)? o PROBE: When you think back three years ago, what would you say is the main reason for that, e.g. change of how your organization approaches the government, more organizations advocating for the same issue, etc.? o PROBE: Is the government more responsive to CSOs? If so, would you say that ACCOUNT as a network contributed to it? Please elaborate on how this is the case? How did media facilitate ACCOUNT’s efforts to draw government(‘s) attention to their policy/regulatory recommendations? o PROBE: What were the major activities of ACCOUNT in relation to CSOs’ engagement with the media and the public in communicating the findings? o PROBE: From your experience, has the coordination between ACCOUNT and media contributed to adoption/acceptance of anti-corruption policy? If so, could you please explain in which cases this occurred? THE MAIN ACHIEVEMENTS IN SELECTED FIVE SECTOR AREAS (PUBLIC PROCUREMENT, WHISTLEBLOWER PROTECTION, EDUCATION, HEALTH AND PUBLIC EMPLOYMENT? Could you please explain what were the main objectives and main activities in your organization’s active sector (public procurement, whistleblower protection, education, health and public employment)? 93 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV o PROBE: What were the main objectives in your sector-network at the onset of the ACCOUNT Activity? o PROBE: What were the main types of activities that the sector network and your organization had undertaken in reaching those objectives? o PROBE: How were other ACCOUNT-network organizations and other sector networks informed about your sector-network activities? How, if at all, were they involved in them? In relation to the above described main objectives and activities, what is your assessment on reaching them during the implementation of ACCOUNT? Please provide examples of: 1) when you managed to reach your objectives; and 2) if you failed to do so, when did this happen and why do you think this occurred? o PROBE: What changes to the whistleblower protection have been accepted by the government? How would you rate those changes? o PROBE: What changes have been accepted with respect to the public procurement legislation and practice? o PROBE: What changes have been accepted with respect to the public employment practice? o PROBE: What are the outcomes of the anti-corruption curriculum in primary and secondary schools? o PROBE: What are the outcomes after introducing the rulebooks and procedures in the Sarajevo Canton and Zenica Canton Healthcare institutions? o PROBE: What would you say was the main driving factor for adopting those changes? (e.g., the quality of input provided by CSOs, the monitoring results, the media influence, political environment, something else, please explain…) To what extent would you say ACCOUNT’s proposals are addressing the main issues that your sector￾network/organization set out to address (both existing proposals and those adopted by the government)? o PROBE: How would you rate those achievements in terms of quality and scope of the legislative/policy/implementation issue they addressed? o PROBE: What did the government take into account? Please specify the policy/legislation/implementation changes that have occurred as a result of your activities? Could you please explain what was achieved with the local level governments (22 municipalities) with respect to anti-corruption initiatives (strategies and action plans development and implementation)? How would you rate local level activities? Is the government more transparent and accountable to the public as a result? o PROBE: What were the main objectives of ACCOUNT on the local level at the onset of ACCOUNT Activity? o PROBE: What were the main types of activities that the sector network has undertaken in reaching those objectives? o PROBE: How did the sector networks from the five main areas interact with the local anti-corruption initiatives? 94 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV o PROBE: What would you say was achieved? How would you rate those achievements in terms of quality (both adopted and implemented government changes)? o PROBE: To what extent would you say ACCOUNT’s proposals are addressing the main issues within the authority of the local level governments (both existing proposals and those adopted by the government)? o PROBE: What did the government take into account? o PROBE: What would you say was the main driving factor for adopting those changes? (e.g. the quality of input provided by CSOs, the monitoring results, the media influence, political environment, something else, please explain…) Could you please explain what was achieved within the 8 cantonal level governments with respect to the anti-corruption initiatives (including the work with the cantonal anti-corruption teams)? How would you rate local level activities? Is the government more transparent and accountable to the public as a result? o PROBE: Please explain the differences in work intensity with different cantons (e.g. Sarajevo and Zenica-Doboj Cantons versus other cantons)? o PROBE: How was the work with cantonal level governments organized? o PROBE: Who was the lead on coordinating the work related to the cantonal level anti-corruption initiatives? o PROBE: What type of implementing assistance was provided by the ACCOUNT implementers (CRMA/INFOHOUSE) to the lead organizations? Could you please explain the main activities/tasks of sector network leads (advocacy) versus the watchdog teams and how these two have interacted? What are the outcomes of this interaction? Are you aware of ACCOUNT activities relating to the development and implementation of integrity plans in RS, Canton Sarajevo and Canton Zenica’s public institutions? How would you rate this process? o PROBE: What is was ACCOUNT’s role in this process? o PROBE: How does this Activity relate to the sector-networks objectives? o PROBE: How did ACCOUNT network participate in this process? o PROBE: To what extent did ACCOUNT coordinate with other international donor interventions (including USAID) on this matter? QUALITY OF INVESTIGATIVE JOURNALISM ABOUT CORRUPTION How would you assess media reporting on corruption of public officials as well as in the public sector and government institutions? o PROBE: How would you rate the quality of media reporting on corruption in general? o PROBE: Is there a difference between the type of media outlets that are reporting on corruption in terms of frequency, scope, focus, and quality of reporting? Is the reporting objective and evidence based? 95 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV How would you rate the quality of reporting produced by zurnal.info, ACCOUNT Media Pool, interview.ba? (Under quality, considering the amount of sources consulted, analytic ability, objectivity, tone, impartiality, elaboration/presentation of the reporting, evidence presented, Etc.)? o PROBE: Are you familiar with zurnal.info, how would you rate the quality of their investigative reports? How frequently do you visit that website (in comparison to other sources of information that you visit frequently)? o PROBE: Are you familiar with the Interview.ba? How would you rate the content of that website with respect to reporting on corruption? o PROBE: Are you aware of the ACCOUNT media pool that over the period of ACCOUNT implementation established cooperation with 14 media outlets? Do you know which media outlets participate in it? How would you rate their reporting on corruption? Would you say that: 1) it is more frequent and 2) it has a better quality than before? - ACCOUNT Media-Pool consists of: Moja Hercegovina, Tacno.net, Captial.ba, BUKA, e-trafica.net, Direkt-portal.com, Gerila.info, Frontal.ba, InfoRadar.ba, B1Infor.ba, slobodanvaskovicblog, Bljesak.info, RTV Zenica, and Analiziraj.ba PUBLIC AWARENESS OF CORRUPTION Is the public aware of corruption in the public/government sector? To what extent? o PROBE: What would you say influences public perception (media, CSOs, individual experiences?)? Is the public informed about and aware of corruption in the ACCOUNT five sector areas? To what extent? Please explain? o PROBE: What is the difference between general perception that corruption is everywhere versus specific understanding of corruption and its mechanisms? The public perception is that corruption in the government/public sector is very high. However, the public perception of anti-corruption initiatives conducted by CSOs is very low. How would you explain the relationship between these two issues? In your opinion is the public more ready today than 3 years ago to report corruption or engage in anti￾corruption initiatives? Please elaborate on your opinion. Why or why not? In your opinion, what are the main factors that influence the publics’ readiness and reaction to corruption? LEGAL AID MODEL TO PROTECT INDIVIDUALS REPORTING CORRUPTION AND WHISTLEBLOWERS Are you familiar with ACCOUNT’s legal aid to victims of corruption and whistleblowers? How would you rate it? 96 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV o PROBE: Do people feel safer to report corruption now? Are you aware of any improvement to the whistleblower protection over the past three years? What role did ACCOUNT play in that? What kind of actions would you recommend to improve the protections of corruption victims and whistleblowers (e.g., in relation to the ACCOUNT’s legal aid model and other mechanisms)? o PROBE: What kind of role should your institution have in future improvements to mechanisms that protect victims of corruption and whistleblowers? What kind of role should ACCOUNT have? Which measures can ensure sustainability of ACCOUNT’s legal aid model? 97 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV INTERVIEW GUIDE FOR BENEFICIARY GOVERNMENTAL AND PUBLIC INSTITUTIONS (STATE, ENTITY, CANTONS AND MUNICIPALITIES) My name is and these are my colleagues . We are the team of researchers working for MEASURE-BiH. First of all, we want to thank you for setting aside the time for this conversation. As you know, USAID/BiH has tasked MEASURE-BiH with conducting an independent evaluation of the USAID/BiH Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) Follow-on Activity. The four-year ACCOUNT Program started implementation across Bosnia and Herzegovina in July 2015 and is anticipated to end in July 2019. The aim of the program is to create an environment that increases civil society participation and reforms through strong collaboration and cooperation in anti-corruption initiatives. Within this evaluation, we will conduct semi-structured interviews with around 45 ACCOUNT stakeholders. Your views and opinions on the Activity are profoundly important for this evaluation. Through these KIIs, we are looking to gain insights into the program implementation, the challenges faced along the way and the influence of specific interventions. We would also like to hear about stakeholders’ perceptions, lessons learned, and recommendations for any possible future donor/government interventions relevant to civil society participation in anti-corruption initiatives and reforms. The information you provide will be combined with information provided from other stakeholders. Your comments are confidential and you will not be identified by name in any report. will be taking notes while we talk. With your permission, we would also like to record this session. The reason we are recording is that it is quite hard to actively participate in a conversation and take notes at the same time. Another reason is that we want to analyze the interviews using objective methods and avoid any biasrelated to quality of notes and the capacity of interviewers’ memory. Do we have your permission to begin recording? Please do not hesitate to mention anything that you find important or that you might find relevant to ask. Do you have any questions before we start? CSO INVOLVEMENT AND INPUT REGARDING ANTI-CORRUPTION LEGISLATION/REGULATION? How did you first hear about ACCOUNT? What kind of communication and cooperation did you have with ACCOUNT network members? o PROBE: What type of assistance did you receive through ACCOUNT? For example, assistance in preparing draft legislation, strategic plans, rulebooks and procedures, or analysis and monitoring results presented to you. o PROBE: Which specific organization did you work with? Have you worked with them before ACCOUNT? o PROBE: Could you please elaborate what was agreed between you and ACCOUNT? What were the main responsibilities between you and ACCOUNT? What were the main activities that you cooperated on? 98 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV What do you know about the ACCOUNT network? Are you aware of the main objectives and membership of the ACCOUNT network? o PROBE: Do you know how many organizations participate in the ACCOUNT network? Which ones did you cooperate with? How would you assess or rate this experience? o PROBE: Which sectors or policies are ACCOUNT members focused on? Please elaborate. How would you assess the role or involvement of CSOs in providing input regarding anti-corruption legislation or regulation? o PROBE: Do you think that the CSOs involvement changed in the last three years? If yes, in which way? What are the main reasons for the change(s)? o PROBE: Please illustrate examples? What were your own experience with CSO involvement and inputs to the anti-corruption legislation or regulation (i.e., how constructive was their involvement and how useful was their input). Did you notice any difference in quality of CSOs’ input, in particular, in the five sectors addressed by ACCOUNT (public procurement, whistleblower protection, public employment, health and education)? What would you say is different today versus before the implementation of ACCOUNT (three years ago)? o PROBE: How would you rate the capacity of CSOs to conduct evidence-based research on anti-corruption strategies/legislation/regulation/policy? If better, would you say that it better reflects the interest of the public today than three years ago? Please elaborate and offer examples from one of the ACCOUNT sectors (in which you have hands-on experience). o PROBE: How did the ACCOUNT organization, with which you cooperated, communicate with you about the research and monitoring results? Have those findings been useful and relevant to you and your institution? How did you use that information in your work? How would you describe your or your institution’s interaction with the CSOs? Has it changed (worsened or improved) during the implementation of ACCOUNT? o PROBE: Could you please explain what are the most frequent methods used to engage with CSOs? o PROBE: Are CSOs more involved in providing input to the government on anti￾corruption legislation or regulation? o PROBE: Has ACCOUNT in any way affected your cooperation with CSOs? Did you sign a Memorandum of Understanding/Memorandum of Cooperation (MoU/MoC) with ACCOUNT? o PROBE: What role did the MoU/MoC play in your cooperation with ACCOUNT? What was agreed on? Were all elements of that MoU/MoC respected and implemented according to the agreement? o PROBE: Would you say that issues relating to corruption in the five sectors have been more frequently reported on in the media in the last three years than before? 99 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV THE MAIN ACHIEVEMENTS IN SELECTED FIVE SECTOR AREAS (PUBLIC PROCUREMENT, WHISTLEBLOWER PROTECTION, EDUCATION, HEALTH AND PUBLIC EMPLOYMENT)? Based on your experience, how would you assess the main achievements within the five sector areas that ACCOUNT operates? o PROBE: What changes related to whistleblower protection have been proposed by ACCOUNT and have been accepted by your institution and government? How would you rate those changes? Has the mechanism for whistleblower protection, established with ACCOUNT, been completed? If not, to what extent has it been completed? o PROBE: What changes proposed by ACCOUNT to public procurement legislation and practice have been accepted? In particular, those changes related to BiH public procurement law? What other changes in practice have been adopted by your institution or government/public institutions, and at what level? Please provide examples. o PROBE: What changes proposed by ACCOUNT have been accepted with respect to the public employment practices? What other changes in practice have been adopted by government/public institutions, and at what level? Please provide examples. o PROBE: What are the implementation outcomes of the anti-corruption curriculum delivered by ACCOUNT in primary and secondary schools? Is that a good model to further expand to all schools? o PROBE: What are the outcomes of the introduction of the rulebooks and procedures in the Sarajevo Canton and Zenica Canton healthcare institutions? o PROBE: What would you say were the main factors that led to the success in cooperating with ACCOUNT and these changes being adopted by your institution or the government (e.g., the quality of input provided by CSOs, the monitoring results, the media influence, the political environment, or something else)? Please explain. To what extent would you say the accepted/adopted changes proposed by ACCOUNT have addressed the main or priority issues within that sector? o PROBE: How were the addressed issues selected or prioritized? Did you cooperate with CSOs during that process? o PROBE: Are there other more important issues that were not addressed by ACCOUNT? ACCOUNT worked with 22 municipalities assisting them in development of anti-corruption strategies and action plans. Could you please elaborate on the process of providing assistance to your municipality? o PROBE: Who and how was your municipality approached to inquire about options for cooperation? Who was representing the municipality’s interests and negotiating the type of assistance and cooperation with ACCOUNT (e.g., the mayor or president of the municipal council)? How would you describe the implementation process of ACCOUNT assistance to your municipality? Did ACCOUNT provide input that your municipality needed for developing a strategic plan or action plan for anti-corruption. Are these action plans being implemented? 100 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV o PROBE: How would you describe the implementation process of ACCOUNT assistance to your municipality? Did ACCOUNT provide the relevant input (i.e., the type of assistance that your municipality needed for developing a strategic plan or action plan for anti-corruption)? o PROBE: Were there any other suggestions form ACCOUNT that government took into account? Please specify the exact changes that have occurred as a result of ACCOUNT activities o PROBE: How would you assess the overall cooperation with ACCOUNT? Was it useful for your municipality? If yes, in what way? What was the most useful? o PROBE: Which aspects of cooperation with ACCOUNT were not helpful for your municipality? What would you change in future cooperation with similar initiatives? What would you suggest to ACCOUNT to improve the conduct or effects? Are you familiar with the achievements within the 8 cantonal governments with respect to anti￾corruption initiatives (including the work of the cantonal anti-corruption teams)? How would you rate cantonal activities, are there any changes introduced relating to cantonal governments transparency and accountability to the public? o PROBE: Please explain the different levels of work intensity within the different cantons (e.g. Sarajevo and Zenica-Doboj Cantons vs other cantons), if you are familiar? o PROBE: How was the work with cantonal governments organized? o PROBE: Who was the lead on coordinating the work related to the cantonal anti￾corruption initiatives? Are you aware of the ACCOUNT Activities relating to the development and implementation of integrity plans in RS, Canton Sarajevo and Canton Zenica’s public institutions? How would you rate this process? o PROBE: What was ACCOUNT’s role in this process? o PROBE: How does this Activity relate to the sector network’s objectives? o PROBE: How did the ACCOUNT network participate in this process? o PROBE: To what extent did ACCOUNT coordinate with other international donor interventions (including USAID) on this matter? QUALITY OF INVESTIGATIVE JOURNALISM ABOUT CORRUPTION How would you assess media reporting on corruption of public officials as well as in the public sector and government institution? o PROBE: How would you rate the quality of media reporting on corruption in general? Is the reporting objective and evidence based? o PROBE: Is there a difference between the type of media outlets that are reporting on corruption in terms of frequency, scope, focus, and quality of reporting? Can you distinguish between a more objective/evidence-based media outlet from those whose reports are of lower quality? 101 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV If you are familiar, how would you rate the quality of reporting produced by zurnal.info, the ACCOUNT media pool, and interview.ba? o PROBE: How frequently do you visit that zurnal.info website (in comparison to other sources of information that you visit frequently)? o PROBE: How would you rate the content of interview.ba? o PROBE: Do you know which media outlets participate in ACCOUNT Media Pool? How would you rate their reporting on corruption? - ACCOUNT Media-Pool consists of: Moja Hercegovina, Tacno.net, Captial.ba, BUKA, e-trafica.net, Direkt-portal.com, Gerila.info, Frontal.ba, InfoRadar.ba, B1Infor.ba, slobodanvaskovicblog, Bljesak.info, RTV Zenica, and Analiziraj.ba. o PROBE: To what extent would you agree with statements that: 1) reporting is more frequent and 2) it has a better quality than three years ago? PUBLIC AWARENESS OF CORRUPTION How would you assess public perception of corruption in the public/government sector? How is it different from the public’s awareness of corruption? Has the public perception or awareness changed in the last three years? o PROBE: What would you say influences public perception of corruption most? o PROBE: In what way has the public’s perception changed in the last three years? Do you think that ACCOUNT activities affected these changes? Please elaborate (illustrate with examples, if possible). Please assess the public awareness of corruption in the ACCOUNT five-sector areas. Provide examples, if possible. o PROBE: To what extent has public awareness influenced changes in a particular sector area? What was the contribution of ACCOUNT in such changes? One of ACCOUNT’s objectives is to raise awareness about corruption through increased media reporting? Are you aware of a specific case or circumstance where this objective materialized? What was the outcome of that increased awareness? o PROBE: Would you say that people are more ready now, than 3 years ago to report corruption? Please elaborate on your opinion. Why or why not? What, in your opinion, are the main factors that influence the public’s readiness and reaction? o PROBE: To what extent are institutions engaging in a public dialogue on the issue of corruption? Based on public opinion survey and ACCOUNT data, the public perception that corruption in the government/public sector is very high (over 90% of surveyed citizens believe there is systemic corruption). However, the public’s perception of anti-corruption initiatives conducted by CSOs is very low (less than 3% of surveyed citizens know about local CSOs anti-corruption initiatives). Do you 102 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV agree with this finding? To what extent do you agree? How would you explain the relationship between these two issues? LEGAL AID MODEL TO PROTECT INDIVIDUALS REPORTING CORRUPTION AND WHISTLEBLOWERS Are you familiar with ACCOUNT legal aid assistance to victims of corruption and whistleblowers? If so, how would you rate it? o PROBE: If you are familiar with this mechanism, how useful is the legal aid to the victims? Does it have any side effects or unwanted consequences? o PROBE: Has your institution played a role in the legal aid assistance? What kind of role should institutions have in protecting the victims? Is there a way to institutionalize the legal aid mechanism or do you think it should be independent (e.g., for the sake of objectivity)? Did your institution collaborate with ACCOUNT on improvement of legal aid assistance? What type of assistance did your institution receive? What changes have been introduced and how would you rate them? Have they improved free legal aid provisions? o PROBE: What role did your institution play in the free legal aid assistance model? What role should institutions have in protecting the victims of corruption and whistleblowers? o PROBE: Which institutions should be responsible for ensuring legal aid assistance to victims of corruption and whistleblowers? Are you familiar with the online registry for free legal aid www.bpp.ba (Besplatna Pravna Pomoc)? Are you aware of any changes to that registry? o PROBE: If you are familiar with the online tool, what changes or improvements would you suggest in order to ensure better support or protection for the victims? o PROBE: Are you aware of any positive or negative examples when victims used this mechanism? Are you aware of any improvement to whistleblower protection over the past three years? What role did ACCOUNT play in that? o PROBE: Are whistleblowers better protected today than three years ago? Can you illustrate your answer with examples? What is, in your opinion, a guarantee of a higher degree of protection? What was, or is, the role of ACCOUNT and how did it contribute to strengthening the mechanisms for protecting whistleblowers? What kind of actions would you recommend for future improved protections of corruption victims and whistleblowers (e.g., in relation to the ACCOUNT’s legal aid model and other mechanisms)? o PROBE: What kind of role should your institution have in the future improvements of mechanisms that protect corruption victims and whistleblowers? What kind of role should ACCOUNT have? Which measures can ensure sustainability of the ACCOUNT’s legal aid model? 103 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV INTERVIEW PROTOCOL FOR ACCOUNT ASSISTED WHISTLE BLOWERS My name is and these are my colleagues . We are the team of researchers working for the MEASURE-BiH. First of all, we want to thank you for setting aside the time for this conversation. As you know, USAID/BiH has tasked MEASURE-BiH with conducing an independent evaluation of the USAID/BiH Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) Follow-on Activity. The four-year ACCOUNT Program started implementation across Bosnia and Herzegovina in July 2015 and is anticipated to end in July 2019. The aim of the program is to create an environment that increases civil society participation and reforms through strong collaboration and cooperation in anti-corruption initiatives. Within this evaluation, we will conduct semi-structured interviews with around 45 ACCOUNT stakeholders. Your views and opinions on the Activity are profoundly important for this evaluation. Through these KIIs, we would like to gain insights into the program implementation, the challenges faced along the way and the influence of specific interventions. We would also like to hear about stakeholders’ perceptions, lessons learned, and recommendations for any possible future donor/government interventions relevant to civil society participation in anti￾corruption initiatives and reforms. The information you provide will be combined with information provided from other stakeholders. Your comments are confidential and you will not be identified by name in any report. will be taking notes while we talk. With your permission, we would also like to record this session. The reason we are recording is that it is quite hard to actively participate in a conversation and take notes at the same time. Another reason is that we want to analyze the interviews using objective methods and avoid any bias related to quality of notes and the capacity of interviewers’ memory. Do we have your permission to begin recording? Please do not hesitate to mention anything that you find important or that you might find relevant to ask. Do you have any questions before we start? LEGAL AID MODEL TO PROTECT INDIVIDUALS REPORTING CORRUPTION AND WHISTLEBLOWERS Could you please describe how you first got in touch with the ACCOUNT legal team? What were the circumstances that led to you reaching out to the legal team? o PROBE: Did you report corruption in a government or public institutions? o PROBE: What was the process of reporting that corruption to ACCOUNT? How did you find the ACCOUNT legal team (e.g., through a personal recommendation or through an online search)? Please elaborate. o PROBE: Are you familiar with the online registry for free legal aid www.bpp.ba (Besplatna Pravna Pomoc). Did you use that website? Has it been helpful? o PROBE: Have you tried in any other way to report corruption before you came to the ACCOUNT legal team? What type(s) of assistance was/were provided to you by the ACCOUNT legal team (e.g., legal advice, legal representation before a court, assistance in taking legal action, general assistance in navigating the justice system, or execution of legal documents)? 104 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV o PROBE: Please describe your cooperation with ACCOUNT after you reported corruption to them? How was your claim of corruption treated? How would you describe that process? Could you please describe if ACCOUNT legal assistance was what you expected? What are the results from that assistance? o PROBE: Did you receive a sufficient amount of information on the process that would take place if you received free legal assistance from ACCOUNT? Was that information clear to you? Have you been informed of challenges or potential risks? o PROBE: What are the expected and unexpected outcomes of ACCOUNT legal assistance? Do you think that individuals reporting corruption and whistleblowers are protected from external influences (e.g., pressure, threats etc.)? Please elaborate why you think yes or no? Did the ACCOUNT legal team provide protection to you? What were your initial expectations and to what degree have those expectations occurred? Please elaborate. Was your story or case also covered by the media? Could you please describe that process? o Was it in cooperation with ACCOUNT’s media outlets (zurnal.info, intervju.ba, Moja Hercegovina, Tacno.net, Captial.ba, BUKA, e-trafica.net, Direkt-portal.com, Gerila.info, Frontal.ba, InfoRadar.ba, B1Infor.ba, slobodanvaskovicblog, Bljesak.info, RTV Zenica, and Analiziraj.ba)? o PROBE: Did you work with ACCOUNT to have your story or case covered by the media? How did this cooperation evolve? What are the intended and unintended outcomes of that? o PROBE: Did media outlets not affiliated with ACCOUNT cover your story or case? Did you expect that to happen? How was your story or case covered? o PROBE: Did you require or receive assistance from ACCOUNT or the ACCOUNT legal team with respect to public attention to your story or case? How would you describe that assistance? What do you see as the largest barrier for people when reporting corruption? To what extent do you think legal aid is available to victims of corruption and whistleblowers, in general? Who should provide that type of assistance? Are you familiar with the work of ACCOUNT’s legal aid team other than the work they have provided for your particular story or case? o PROBE: Are you aware of any changes to whistleblower protection legislation or regulation? o PROBE: If yes, could you please explain what your view is on the current or draft legislation? Does it address the main issues relevant to whistleblower protection and providing legal aid to individuals reporting corruption? o PROBE: Has the ACCOUNT legal team consulted with you regarding your recommendations to that legislation? If so, are you aware of your opinions being introduced in that legislation? o PROBE: Is there anything that needs further improvement? Do you think that the government will adopt and/or ensure implementation of that legislation? 105 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV GUIDE FOR PANEL DISCUSSION WITH MEDIA REPRESENTATIVES (ACCOUNT BENEFICIARIES AND NON-BENEFICIARIES) The panel discussion guides are intended to serve as semi-structured guides for your conversations with ACCOUNT beneficiaries. Do not read the questions or probes word for word. Instead, adapt the wording to match the phrasing used by the respondents. Take notes on key terms or phrases used by the respondents that may be helpful in coding the focus group data. Ask for clarification and definitions as needed. Familiarize yourself with the discussion guides in advance of your meeting. Skip questions that are not relevant given the discussion specificities. Highlight the questions you will prioritize if the discussion time is limited. Be respectful of the group’s time and keep the discussion to the agreed length of time. We can follow up by phone or email for more information as needed. In addition: ▪ Take notes during the discussion to ensure we accurately report what is discussed during the panel discussion. We will record this session as well. No one except the research team will have access to this recording. ▪ As necessary, tailor all questions to fit the focus group’s relationship with ACCOUNT or investigative journalism. ▪ Keep the discussion under ninety minutes. ▪ The evaluation team will ensure that the information shared through these focus groups remain strictly confidential. ACCOUNT EVALUATION PANEL DISCUSSION WITH MEDIA REPRESENTATIVES Thank you for agreeing to speak with us today. My name is and this is , and we are researchers from USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH). We are here today because USAID/BiH has commissioned MEASURE-BiH to conduct an independent evaluation of the USAID/BiH ACCOUNT Activity. Our aim is to learn from your experiences, not to audit or judge your or the ACCOUNT implementers work in any way. The information you provide to us will be used in combination with what we learn from others to produce an overview of ACCOUNT’s performance. Your comments are confidential and you will not be identified by name in any report. will be taking notes while we talk. With your permission, we would also like to record this session so that we can refer to the audio and clarify our notes later, if necessary. Do we have your permission to begin recording? Do you have any questions before we start? 106 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV BACKGROUND INFORMATION First, I would like to provide you with some background information on ACCOUNT in order to assist with the focus of today’s conversation. The Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) follow-on activity is a USAID/BiH intervention that has been implemented since July 2015 and will continue until mid-2019. ACCOUNT works with civil society organizations as well as media outlets and representatives to ensure evidence-based research, civic monitoring, and investigative reporting on corruption cases in five sectors: 1) public procurement, 2) whistleblowing, 3) public sector employment, 4) healthcare, and 5) education. Within this intervention, ACCOUNT envisioned establishing an effective collaboration between CSOs and the media to better inform the public on corruption in the public sector. Considerable efforts have been invested in increasing the volume, quality, and outreach of sector-based CSOs’ monitoring reports and ensuring they are accompanied by appropriate and timely investigative media reports to raise public awareness and stimulate civic participation. Specifically, ACCOUNT has within one of their intervention components aimed to strengthened public awareness of corruption and public engagement in anti-corruption initiatives though strategically targeted media campaigns. The expected results within this component are: 1) Increased media reporting on corruption and 2) Increased public awareness of corruption cases in the specified sectors. ACCOUNT worked to increase the quality of media investigative reporting in the five-sector areas through various approaches. One of these approaches was to directly work with Zurnal.info. Zurnal.info designated special attention to investigate corruption stories as well as informing the public on the activities related to anti￾corruption (ACCOUNT media-pool consists of, among others: Moja Hercegovina, Tacno.net, Captial.ba, BUKA, e-trafica.net, Direkt-portal.com, Gerila.info, Frontal.ba, InfoRadar.ba, B1Infor.ba, slobodanvaskovicblog, Bljesak.info, RTV Zenica, and Analiziraj.ba). A separate website, Interview.ba, was established to provide individuals’ stories and views related to fighting corruption. This website uses the interview format by various types of individuals either telling their story or giving their opinions on the topic. Finally, ACCOUNT has established a media pool, which has provided financial assistance to engage in these same types of activities and to network them with the CSOs in order to increase the quality of their reporting. Provided this brief background of the ACCOUNT intervention, we would like to gain a greater understanding of the quality and quantity of investigative journalism targeting public corruption. We would like to discuss the following main questions or sections with you. In the first section we would like to discuss the quality and quantity of media reporting on corruption, while in the second section we would like to discuss with you the public awareness of corruption. For both of the sections, we have prepared some guiding questions that we will keep on the screens during the whole discussion. SECTION 1: THE QUANTITY AND QUALITY OF INVESTIGATIVE JOURNALISM ON CORRUPTION In general how would you assess the quality of investigative reporting on corruption of public officials and within the public sector and government institutions? What are the main quality criteria that are present or absent from that reporting? For example, is the reporting objective and evidence based; are there enough media resources allocated to the reporting; to what degree of depth; and is there variety of views in this reporting? Is there a difference between the types of media outlets that are reporting on corruption? What are the differences and reasons behind them? 107 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Is there any change in the quantity of media reports about corruption in the past three years? If so, what do you think are the main reasons for that change? What are the most frequent topics of coverage? If you are familiar with the reporting of zurnal.info or interview.info, how would you compare them to the average or general circumstances that you described previously with respect to quality and quantity of reporting? And what about the ACCOUNT assisted media outlets such as: Moja Hercegovina, Tacno.net, Captial.ba, BUKA, e-trafica.net, Direkt-portal.com, Gerila.info, Frontal.ba, InfoRadar.ba, B1Infor.ba, slobodanvaskovicblog, Bljesak.info, RTV Zenica, and Analiziraj.ba SECTION 2: PUBLIC AWARENESS OF CORRUPTION Based on the data that we have collected though the four annual National Surveys of Citizens Perceptions, our data indicate that public perception that corruption is prominent in the government/public sector is very high. However, public perception of anti-corruption initiatives conducted by CSOs is very low. How would you explain the relationship between these two findings? One of ACCOUNT’s objectives is to raise awareness of corruption through increased media reporting? Are you aware of a specific case or circumstance where this objective materialized? What was the outcome of that increased awareness? Have you observed any change in the public’s readiness to report corruption in the past three years? Are they more or less likely to report corruption? Why? Did, in your opinion, media have any influence over that change? 108 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV DATA COLLECTION INSTRUMENT FOR ONLINE SURVEY FOR ACCOUNT NETWORK CSOs Dear ACCOUNT network members, This survey has 18 main questions and is estimated to take you around 25 minutes to complete. Your input is very important as it will inform USAID’s external performance evaluation of the Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) follow-on activity. This, in turn, will provide recommendations to USAID/BiH for possible programing related to civil societies’ engagement in fighting corruption. We greatly appreciate you taking the time to answer this mini-survey! Your answers are confidential and you will not be identified by name or organization in any report. The survey asks for your name only for the purpose of our own verification and better understanding of the context of your responses. IF YOU HAVE ANY QUESTIONS OR REQUIRE ANY CLARIFICATIONS ON THE SURVEY, PLEASE CONTACT USAID’S MONITORING AND EVALUATION PROJECT (MEASURE-BiH), WHICH IS CONDUCTING THIS EVALUATION AT ecosic@measurebih.com. Please tell us your name/name your CSO. ________________________________________________ Have you been involved in anti-corruption projects prior to ACCOUNT (please note that it started in July 2015). a) Yes b) No Are you a member of the ACCOUNT network? i. Yes ii. No Since when were you or your organization a member of the ACCOUNT network? Please enter the year when you joined the network: ________________ If you have financial assistance from ACCOUNT, could you please select what type: a) Small grant/sub-award b) Contract for certain services. c) Nothing. 109 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV In what capacity are you a member of ACCOUNT, please select all that apply a) Member of the sector network for advocacy b) Member of the Watchdog team c) Lead of the sector network for advocacy activities d) Lead of the sector for anti-corruption initiatives e) Lead on cantonal/municipal level anti-corruption initiatives f) None of the above Are you a member of any of the following ACCOUNT network groups? Please select all that apply. g) Whistleblower Protection h) Public Procurement i) Public Employment/Recruitment j) Healthcare Sector Group k) Education Sector Group l) No one above Could you state if you have participated in any of the following Activities of ACCOUNT. Please select yes or on. Yes No Did you participated in activities related to Whistleblower Protection Participated in the Whistleblower Protection Sector Group’s coordination meetings Participated in defining the recommendations for the amendments to the BiH Whistleblower Protection Law Provided recommendations for the RS Whistleblower Protection Law to the RS Ministry of Justice or participated in the drafting of that law Participated in the recommendation process to the Ministry of Justice of FBiH Working Group in drafting the Whistleblower Protection Law Conducted monitoring or analysis to inform the effectiveness of existing whistleblower protection and communicated those finding to the Whistleblower Protection Group Participated in organization of street actions, campaigns, roundtables, or conferences to advocate for better whistleblower protection 110 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Provided content or information for the ACCOUNT media campaign regarding improvements to whistleblower protection Participated in meetings with government officials to advocate amendments or enact legislation to protect whistleblowers Did you participated in activities related to improvement of Public Procurement Participated in the Public Procurement Sector Group’s coordination meetings Conducted monitoring or analysis related to reducing corruption in public procurement and communicated those finding to the Public Procurement Sector Group Participated in organization of street actions, campaigns, roundtables, conferences, or exhibitions to advocate for better public procurement legislation Provided content or information for the media campaign regarding improvements to public procurement legislation Participated in meetings with government representatives to advocate for amendments to the Public Procurement Law. Did you participated in activities related to anti-corruption in Healthcare Participated in the Healthcare Sector Group’s coordination meetings Conducted monitoring or analysis related to corruption and fighting corruption in Healthcare institutions and communicated those finding to the Public Procurement Sector Group Participated in organizing street actions, campaigns, roundtables, conferences, or exhibitions to advocate for the elimination of corruption form healthcare institutions Participated in meetings with government or public-institution representatives to advocate for the improvement of procedures to reduce the risk of corruption in healthcare institutions. Assisted government and public Healthcare institutions to develop anti-corruption policies and procedures (e.g., the rulebooks, guidelines, strategic plans, or action plans) Did you participated in activities related to anti-corruption in Education Participated in the Education Sector Group’s coordination meetings Conducted monitoring or analysis related to the introduction of anti-corruption education content in BiH schools Participated in the organization of street-actions, campaigns, roundtables, conferences, and exhibitions to advocate for the introduction of anti-corruption education content into school curriculum 111 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Provided content or information for the media campaign related to the improvement of education practices on anti-corruption curriculum Participated in meetings with government or public-institution representatives to advocate for the introduction of anti-corruption content in education curriculum Participated in the presentation of the Curriculum on the fight against corruption in educational institution Did you participated in activities related to anti-corruption in Public Employment/Recruitment Participated in the Public Employment/Recruitment Sector Group’s coordination meetings Conducted monitoring or analysis related to introducing anti-corruption measures in the area of public sector employment or recruitment Participated in organizing street actions, campaigns, roundtables, conferences, and exhibitions to advocate for the introduction of anti-corruption measures in the area of public sector employment Provided content or information for the media campaign on corruption and fighting corruption in the area of public sector employment Participated in meetings with government or public-institution representatives to advocate for the introduction of anti-corruption measures in the area of public sector employment Assisted governments, public institutions, or public enterprises to implement fair and transparent employment procedures Did you participate in activities related to Anti-Corruption Strategies/Action Plans at the Local Level At ACCOUNT meetings aimed at coordinating between institutions and CSOs in the process of developing anti-corruption strategies or action plans at the level of municipalities or cantons Provide technical support to the municipalities in order to develop the anti-corruption strategy/action plan/integrity plan Provided technical assistance to cantonal anti-corruption teams to develop or implement anti￾corruption strategies Perform supervision or undertake analyzes related to the implementation of anti-corruption strategies at municipal level 112 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV For your organization, please note next to the following sector initiatives whether your organization has participated in them prior to 2015 (before the implementation of ACCOUNT) or as of the 2015 (during the implementation of ACCOUNT). In case you already worked in the sector with your organization’s own initiative, please indicate whether that engagement intensified. No engagement before 2015. Engagement as of 2015. Intensified engagement since 2015 Engagement is on same level before and after 2015 Anti-corruption initiatives related to whistleblower protection Initiatives related to anti-corruption in public procurement Initiatives related to anti-corruption in public employment Initiatives related to anti-corruption in health Initiatives related to anti-corruption in education Please state to what extent you agree with the following statement to be true for you/your organization since 2015. Strongly disagree Somewhat disagree Somewhat Agree Strongly Agree Don’t know Our organization is more frequently participating in public consultations organized by government in the procedure of adopting laws/regulations Our organization is more frequently invited for collaboration with government institutions to participate in the development of new laws/regulations/procedures/ to reduce corruption (e.g. in joint projects/initiatives, 113 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Working Groups formed by Ministries or Parliaments) Our organization is more frequently monitoring and reporting on government corruption in the public sector Please carefully read each of the following statements. Please indicate whether the Account network has contributed to the listed improvements since 2015 by rating them on a scale from 1 to 4, where 1 represents “Not at all” and 4 represents “Significantly. Not at all 1 2 3 Significantly 4 Don’t know Improved the Draft Law on the Protection of Persons Reporting Corruption in the Institutions of Bosnia and Herzegovina Contributed to drafting good whistleblower protection legislation in Republika Srpska Contributed to drafting good whistleblower protection legislation in Federation of BiH Improved whistleblower protection mechanism and free legal aid provisions to whistleblowers Contributed to drafting improved public procurement legislation. Improved public procurement procedures (e.g., through by-laws or instructions) in government/public institutions Increased CSOs involvement regarding improved procedures to suppress and punish corruption within healthcare institutions Contributed to the increase of anti-corruption education content in BiH schools Improved the actions taken by government(s)/public sector institutions to be 114 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV more transparent in their employment processes Significantly helped the institutions, at the municipal level, in undertaking activities aimed at reducing the level of corruption (e.g. development and implementation of anti￾corruption action plans / integrity plans) Significantly helped the institutions, at the cantonal level, to undertake activities aimed at reducing the level of corruption (e.g. development and implementation of anti￾corruption action plans / integrity plans) On a scale from 1 to 4, where 1 represents 1 “Totally disagree” and 4 “Totally agree”, to what extent do you agree with the following statements: Totally disagree 1 2 3 Totally agree 4 Don’t know Over the past three years, media has reported more frequently on corruption Investigative journalism in BiH has improved in the past three years Overall media reporting on corruption is unbiased There are few media outlets that are unbiased Zurnal.info produces good quality investigative reporting Intervju.ba selects very good interviewers and topics for the interview 115 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Please state at least 3 media sources that you trust or are fairly confident they provide accurate, unbiased and well-researched investigative reports on corruption. These media sources can be in any format (e.g. print, online, video, multi-media etc.)? • First: ______________________________________________________________ • Second: ______________________________________________________________ • Third: ______________________________________________________________ On a scale from 1 – 4, where 1 represents 1 “Not at all” and 4 “Significantly”, could you please state if public awareness of corruption has increased and to what extent: Not at all 1 2 3 Significantly 4 Don’t know Whistleblower Protection Issues Corruption in Public Procurement Corruption in Public Employment Corruption in Healthcare Institutions Corruption in Education Do you think citizens are more frequently reporting corruption today than three years ago? m) Yes n) No o) Don’t know 116 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ACCOUNT provides legal aid assistance to whistleblowers and victims of corruption. If you are familiar with these services, do you agree with the following statements: Totally disagree 1 2 3 Totally Agree 4 Don’t know The ACCOUNT Legal Aid Team is professional in their work The ACCOUNT Legal Aid Team has provided assistance to important cases and stopped further corruption in public institutions The ACCOUNT Legal Aid Team has provided adequate protection to victims of corruption and whistleblowers The ACCOUNT is recognized as an address that citizens can report corruption What would be your recommendations for future potential assistance to civil society sector’s engagement in anti-corruption in BiH? (If possible, please name specific strategies or instruments you think should be supported/developed) ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ______ Please mark your gender a) Male b) Female c) Other Thank you for participating. 117 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV DATA COLLECTION INSTRUMENT FOR ONLINE SURVEY FOR ACCOUNT NETWORK CSOs Dear journalists, This survey has 14 main questions and is estimated to take you around 20 minutes to complete. Your input is very important as it will inform USAID’s external performance evaluation of the Anti-Corruption Civic Organizations’ Unified Network (ACCOUNT) follow-on activity. This, in turn, will provide recommendations to USAID/BiH for possible programing related to civil societies’ engagement in fighting corruption. We greatly appreciate you taking the time to answer this mini-survey! Your answers are confidential and you will not be identified by name or organization in any report. The survey asks for your name only for the purpose of our own verification and better understanding of the context of your responses. IF YOU HAVE ANY QUESTIONS OR REQUIRE ANY CLARIFICATIONS ON THE SURVEY, PLEASE CONTACT USAID’S MONITORING AND EVALUATION PROJECT (MEASURE-BiH), WHICH IS CONDUCTING THIS EVALUATION AT ecosic@measurebih.com. Could you please tell your name/name of media you work for? _________________________________________________ Are you a member of the ACCOUNT network?. c) Yes d) No Since when were you or your media organization a member of the ACCOUNT network? Please enter the year when you joined the network: ________________ Are you a member of the ACCOUNT media-pool? a) Yes b) No If you have financial assistance from ACCOUNT, could you please select what type: a) Small grant/sub-award b) Contract for certain services c) Did not receive any kind of financial help 118 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Please carefully read each of the following statements. Please indicate whether the Account network has contributed to the listed improvements since 2015 by rating them on a scale from 1 to 4, where 1 represents “Not at all” and 4 represents “Significantly. Not at all 1 2 3 Significantly 4 Don’t know Improved whistleblower protection legislation on the state level (BiH) Contributed to drafting good whistleblower protection legislation in Republika Srpska Contributed to drafting good whistleblower protection legislation in Federation of BiH Improved whistleblower protection mechanism and free legal aid provisions to whistleblowers Contributed to drafting improved public procurement legislation. Improved public procurement procedures (e.g., through by-laws or instructions) in government/public institutions Increased CSOs involvement regarding improved procedures to suppress and punish corruption within healthcare institutions Contributed to the increase of anti-corruption education content in BiH schools Improved the actions taken by government(s)/public sector institutions to be more transparent in their employment processes Took action in reducing corruption at the municipal level (e.g., developing and implementing anti-corruption action plans) Took action in reducing corruption at the cantonal level (e.g., developing and implementing anti-corruption action plans) 119 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV On a scale from 1 to 4, where 1 represents “Totally disagree” and 4 “S Tottaly agree”, to what extent do you agree with the following statements: Completely disagree 1 2 3 Completely agree 4 Don’t know Over the past three years, media has reported more frequently on corruption Investigative journalism in BiH has improved in the past three years Overall media reporting on corruption is unbiased There are few media outlets that are unbiased Zurnal.info produces good quality investigative reporting Intervju.ba chooses very good interviewers and topics for the interview Please state at least 3 media sources that you trust or are fairly confident they provide accurate, unbiased and well-researched investigative reports on corruption. These media sources can be in any format (e.g. print, online, video, multi-media etc.)? • First: ______________________________________________________________ • Second: ______________________________________________________________ • Third: ______________________________________________________________ 120 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV If you are a member of the ACCOUNT media pool, please answer if and how often you have participated in the following activities of ACCOUNT (on average): On average Once a week Once month Once every quarter Once a year Did not participate Published articles on ACCOUNT CSOs’ activities (e.g., roundtable discussions, street actions, information asked by CSOs to share with the public, and events) Participated in coordination meetings with ACCOUNT CSOs to assist with the advocacy efforts Held meetings/or corresponded with individual CSOs to receive information on of the five areas (Whistleblower Protection, Transparency in Public Procurement, Transparency in Employment/Recruitment in Public Sector, Healthcare, or Education) Published investigative Reports on whistleblower protection issues (in any media format) Published investigative reports on public procurement corruption/anti-corruption measures (in any media format) Published investigative reports on corruption/anti￾corruption measures in public employment (in any media format) Published investigative reports on corruption/anti￾corruption measures in healthcare institutions (in any media format) Published investigative reports relevant to anti￾corruption education (in any media format) Conducted and published an interview with individuals that were interested in speaking about corruption and fighting corruption in BiH public sector institutions. 121 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Prepared a specialized analysis on a particular topic related to corruption in public institutions You publish content that instructs citizens how to fight corruption (e.g. Educational content to recognize, and to whom report corruption) On a scale from 1 – 4, where 1 represents “Not at all” and 4 “Significantly” , could you please state if public awareness of corruption has increased and to what extent: Not at all 1 2 3 Significantly 4 Don’t know Whistleblower Protection Issues Corruption in Public Procurement Corruption in Public Employment Corruption in Healthcare Institutions Corruption in Education Do you think citizens are more frequently reporting corruption today than three years ago? d) Yes e) No f) Don’t know 122 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ACCOUNT provides legal aid assistance to whistleblowers and victims of corruption. If you are familiar with these services, do you agree with the following statements: Strongly disagree 1 2 3 Strongly Agree 4 Don’t know The ACCOUNT Legal Aid Team is professional in their work The ACCOUNT Legal Aid Team has provided assistance to important cases and stopped further corruption in public institutions The ACCOUNT Legal Aid Team has provided adequate protection to victims of corruption and whistleblowers ACCOUNT Legal Aid Team is sustainable after donor support to ACCOUNT Network ends What would be your recommendations for future potential assistance to civil society sector’s engagement in anti-corruption in BiH? (If possible, please name specific strategies or instruments you think should be supported/developed) ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ ______ Pleas mark your gender? a) Male b) Female c) Other Thank you for participating. 123 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV DATA COLLECTION INSTRUMENT FOR ACCOUNT BENEFICIARY SCHOOLS/TEACHERS Dear Teachers/Professors, We would kindly ask you to take about 10 minutes of your time fill out this Survey, which has 16 questions and is estimated to take you around 7 minutes to complete. We are reaching out to you to learn about assistance from Civil Society Organizations that have collaborated with your school in the past three years on implementation of Anti-corruption Curriculum Content. Your input is very important as it will inform USAID’s external Performance Evaluation of Anti-Corruption Civic Organizations’ Unified Network Follow-on activity - ACCOUNT which will provide recommendations to USAID/BiH for possible future assistance in the education sector. We greatly appreciate you taking the time to answer this Mini-Survey! Your answers are confidential and you will not be identified by name or organization in any report. The Survey asks for your name only for the purpose of our own verification and better understanding of the context of your responses. IF YOU HAVE ANY QUESTIONS OR REQUIRE ANY CLARIFICATIONS ON THE SURVEY OR SPECIFIC QUESTIONS, PLEASE CONTACT USAID’S MONITORING AND EVALUATION PROJECT (MEASURE-BiH), WHICH IS CONDUCTING THIS EVALUATION AT ecosic@measurebih.com. Please tell us your name/name your institution. ________________________________________________ In which school do you teach? a) Elementary b) Secondary In which municipality/City is your school located? Please write: _____________________________________________ Have you heard of the ACCOUNT Network? a) Yes b) No Have you cooperated with one of the following organizations in the past three years? 1. Yes, 2. No 124 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV a) Centre for Development of Youth Activism (CROA) – Sarajevo b) Youth Center Trebinje c) AoC Justica - Srebrenik d) Helsinki Committee for Human Rights – Bijeljina e) Helsinki citizen parliament – Banja Luka f) Center for development civil society (CRCD) – Doboj g) Tolerance against diversity (Topper) – Doboj h) KAM – Zenica i) Civil organization Inicijator – Sarajevo j) Prizma – Brcko k) AoC Resource center – Brčko l) Centre for Women’s Rights Zenica m) Perpetum mobile – Banja Luka n) Foundation for Development and Democracy – Sarajevo How did you first get in contact with that organization? a) I/my school/my university received an information form the relevant Ministry of Education to cooperate b) I/my school/university was directly contacted by the organization c) I/my school/university found out about them and their program and reached out to them seek cooperation Could you please say which of the following that cooperation included and how would you rate it on a scale from 1 to 4, with one being Very high quality to 4 being Very low quality: Very high quality 1 High quality 2 Low quality 3 Very low quality 4 Do not know 5 Training for teaching personnel on how to teach anti-corruption curriculum by the organization that we cooperated with, was: Anti-corruption curriculum/lectures were delivered to students by the organization that we cooperated with, was: Event on the importance of anti-corruption education was: In general expertise and assistance of the organization with which we cooperated was: 125 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Is there anything else that you would like to add with regards to the cooperation that you had with the Organization that you worked with – please elaborate __________________________________________________________________________ ___________ Did you received training and teaching materials on how to teach about preventing corruption? a) Yes b) No To what extent was that useful to you? Very useful Somewhat Useful Not useful Usefulness of the training for teaching personnel Usefulness of the teaching materials/manuals Could you please answer how often you do one of the following? Once every two weeks Once a month Once quarterly Once a year Never Teach the full manual on anti-corruption to your students Use the examples or sections form that training to integrated it into other classes you teach Share the training materials with other teaching personnel Have form your school participated in the class where anti-corruption lessons were thought, either by you or the Organization civil society? a) Yes b) No 126 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Do you agree with the following statements? Yes No I don’t know The lessons were very interesting for students and they enjoyed the format The students have gained a very good understanding on identifying corruption and what they can do to prevent it Most students were participated in extracurricular activities that aim to promote prevention of corruption (e.g. prepared video clips, street performances, wrote short stories). Most of students from who attended the training/lectures participated in development of anti-corruption action plans. In general I think that students who participated in the classes/lessons would report corruption in the future In general I think that all education institutions should teach anti￾corruption curriculum With respect to preventative activities related to corruption in the education sector do you agree with the following statements? Yes No Don’t know In the past three years there is generally a higher volume of anti￾corruption activities in the education sector than there were three years ago? Awareness on corruption in the education sector has increased Corruption in the education sector itself has decreased as a result of higher awareness and implementation of anti-corruption measures What would be your recommendations for future potential assistance relevant to the education sector role in prevention of corruption (If possible, please name specific strategies or instruments you think should be supported/developed) 127 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV What is your gender? a) Male b) Female c) Other Thank you for your participation. 128 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX VIII: COMMENTS FROM THE IMPLEMENTING PARTNER AND USAID/BIH MISSION ON THE ACCOUNT EVALUATION GENERAL COMMENTS ON ACCOUNT PERFORMANCE EVALUATION Respected, IP CRMA would like to give some more detailed comments regarding the findings related to Evaluation question 3: to what extent has ACCOUNT increased quality and quantity of Investigative journalism targeting public corruption in BiH? ET: Finding 10: Majority of respondents are familiar with Zurnal.info, and to a lesser extent with Interview.ba and Media Pool. Citizens’ are mainly consuming traditional media sources over internet. Deeper knowledge about their work is mainly reserved to beneficiary CSOs and Media. Most agree that competing with public broadcasters still remains challenging in a context where citizens’ heavily rely on traditional media sources for information. IP CRMA: We would like to highlight that within limited funds we were not able to compete with Public broadcasters, as well as our goal wasn’t to compete with them. We are aware that citizens rely on traditional media sources but we should also have in mind the following facts: we have been offering televisions to broadcasting TV Affair for free in order to increase audience reach. Most of stories have been published on TV (BN TV, Al Jazeera Balkans). We can’t expect from PBS to broadcast our reports because there are under strong political influence. Also, with all respect for citizen’s habits, we need to stress that numerous analysis show that online media outlets are more valuable sources of information than traditional media. E.g. Analysis “Media Integrity Matters – Reclaiming public service values in media and journalism” (conducted by Peace Institute, Ljubljana) under chapter “The bright side of journalistic practice” stated: “The examples of good and resourceful journalism and contributions to pluralism can be found especially among online media outlets. Various sources suggest that good pieces of journalism and/or critical and engaged articles can be found in the online magazines Žurnal...” Also, in order to increase public reach, we printed the Affair in the form of a newspaper and shared to citizens. From the evaluation report, INFOHOUSE is glad that the evaluation shows that overall ACCOUNT performance is perceived as a positive CSO endeavor in the field of anti-corruption in Bosnia and Herzegovina. While we appreciate the efforts of the evaluation team (ET) to provide this comprehensive evaluation, below are a few remarks about this report from INFOHOUSE (IH) side, which relate to the following evaluators' comments when it comes to Page 19: 3. Evaluation Methods and Limitations (page 19), in particular to Limitation 5: Limitation 5: Reporting style focused on detailing activity-level vs output/outcome (page 23). 129 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ET: This reporting structure produced large amount of overlapping across the different sections of reporting within components but also across components, making it difficult to follow interventions contribution to the overall objectives of the intervention. IH: In advocating activities for strengthening development and implementation of complex anti-corruption legislations, policies and procedures at all levels of government with CSOs’ inputs, reporting on achievements can only be possible if implemented and reported as cross-sector interventions. Given that, the reporting format that ACCOUNT used is fully in line with the Project Description and the reporting style includes cross-sector reporting rather than overlapping. Given that, we find the repeated language about overlapping spelled out wrong in the evaluation, which can cause misunderstanding of technical aspect of the program. For example, introduction of anti-corruption Rulebooks in public health institutions in Zenica-Doboj Canton, could not be possible without the project relations and collaboration with the Anti-corruption Team of Zenica￾Doboj Canton on development and implementation of Anti-corruption Strategy/Plan. As double-effort activity, it is reported as linked activity in different sections of the ACCOUNT’s report and includes no overlapping. This is the issue of methodological style/approach rather than reporting style, and the way it is described in the evaluation by the ET is misinterpreted. Some specific comments about this is also included in the attached draft evaluation. Page 29: On the Evaluation Question 2, in particular Finding 4: Achievements per sector are difficult to assess as the program description did not define clear and measurable results for each sector and Annual Reports document progress along detailed activities instead of goals and objectives. ET: The Results Framework structure and terminology is not consistent across Activity documentation and reporting periods. For example, the Y3 report and the Y4 annual work plan introduce new language for 3 activities in C1 “intermediate results” and “key results” (activities 1.1, 1.3 and 1.5) that did not exist before. IH: We made an attempt to explain this to the ET as this issues was raised during the field meeting. It seemed that this comment by the ET included the evaluation of the reporting style/format rather than the content. As the major advocacy activities in the selected sectors are coming to conclusions we wanted to emphasize the key results for easier understanding of the overall process and its key results rather than listing them by quarters. We are sorry that the evaluators did not observe our explanation and include in the evaluation. Page 30: Finding 5: ACCOUNT facilitated the implementation of numerous activities and engaged a large number of stakeholders. The Program design established a very complex structure reflecting itself across program components, implementation responsibilities and actors involved, making the interconnections difficult to follow. Furthermore, in-depth work was not possible in all sectors, due to diverse foci and on the expansion of coverage across governance levels and institutions. ET: Insufficient synergies between Components… Despite the several cross-sector activities (especially in organizing joint events for HEALTH and EDU), IH: Synergies as a term was mentioned in no program documents as a goal to major achievements in the project sectors and this in no way affected the project dynamic or lessen the efforts. On the contrary, there were successful examples such as it is the case with the healthcare and whistleblowing sectors which proved strong inter-component synergies while adopting the whistleblowing mechanism in the anti-corruption Rulebooks. We find this qualification not being well documented in the evaluation. 130 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV Pages 32-32, 34: Finding 6: Progress has been made in all sectors, as confirmed by KII’s, online survey and Activity reports. However, results are not comparable in certain instances due to overlapping activities across sectors. Page 32: ET: Number of policy proposals: In EDU sector, there were no policy proposals. (Page 32) IH: This is wrongly spelled, as no policies proposals are targeted in the education sector. Page 33: ET: Some achievements cannot be segregated by sector due to the cross-sector nature of activities. For example, Activity reports indicate a high overlap between activities in the sectors of HEALTH, EMPL, EDU and local/ cantonal level activities. (Page 33) IH: As above, no overlaps were involved in the project but rather cross-sector relations which resulted in joint actions/results. Page 15: ET: In terms of geographic coverage, effects in WBPL and PPL are country-wide as ACCOUNT influenced the state and entity level laws. Activities in the sectors of HEALTH, EMPL and EDU and the local level activities are rather dispersed in various parts of the country due to a broad roll out of activities (Figure 15). IH: Health and education institutions are cantonal-based/budgeted institutions and were dispersed according to the demands/commitments from the relevant anti-corruption bodies and government institutions at the cantonal level. Pages 56, 57, 59: III Background ET: Neither of the Entities had adopted a Whistleblower Protection Law (WBPL) at the time of ACCOUNT initiation. IH: Where in the evaluation process the ET found this out? The RS Law on Protection of Whistleblowers was adopted in July 2018, with direct support from ACCOUNT. While accepted in the first reading, still pending is the adoption of the FBIH Law on Whistleblowing Protection. IH: Additionally, some specific data/numbers in the III Background in the evaluation (pages 56-57, 59) are spelled wrong and they need to be changed. Corrected numbers are included in the comments next to each section in the attached evaluation. Page 99: Figure 8: Comparison of available Zurnal.info and Interview.ba Audience Report Data IH: The project goal targeted no comparison of news report data provided by Zurnal.Info and Interview.ba as totally incomparable for the following reasons: 1. Different budgets. From the overall approved budget in the amount of USD 324,000 for these two media outlets: 92, 5 % of the overall budgeted amount was allocated to Žurnal while 7, 5 % was allocated to Intreview.ba. This difference was significantly increased through allocation of additional 131 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV funds to each of these two media outlets, so that 96, 5% of the overall additional funds was allocated to Žurnal and 3, 5% was allocated to INFOHOUSE. 2. Different time of formation/baselines of each of these media outlets 3. Different media profiles 4. Different size of media staff involved in news production Page 84: 5. ANNEX IV: COMPLETE LIST OF KEY INFORMATS Corrections of names and positions should be changed. Pls. see the right spelling in the text. 132 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV ANNEX IX: THE EVALUATION TEAM RESPONSES TO COMMENTS FROM THE IMPLEMENTING PARTNER AND USAID/BIH MISSION February 7, 2019 To: Elma Bukvic Jusic Development Assistance Specialist / MEASURE-BiH COR USAID/BiH Subject: THE EVALUATION TEAM’S RESPONSES TO COMMENTS FROM IPs AND USAID ON THE PERFORMANCE EVALUATION OF USAID/BIH’S ANTI-CORRUPTION CIVIC ORGANIZATIONS’ UNIFIED NETWORK FOLLOW-ON ACTIVITY (ACCOUNT) REPORT Dear Ms. Bukvic-Jusic, Included below are the evaluation team’s responses to the IP’s and USAID’s request for clarification on the Performance Evaluation of USAID/BIH’s ANTI-CORRUPTION CIVIC ORGANIZATIONS’ UNIFIED NETWORK FOLLOW-ON ACTIVITY (ACCOUNT) report. THE EVALUATION TEAM’S RESPONSES TO THE IPs’ AND USAID’s REQUEST FOR CLARIFICATION IP’s Comment #1: Executive Summary- MAIN FINDINGS AND CONCLUSIONS: It is not clarified why is not possible to measure achievement for other sectors...We report equally for all sectors (for sectorial working group) USAID’s Comment #1: Executive Summary- MAIN FINDINGS AND CONCLUSIONS: Please rephrase this to say that USAID’s Program Description defined results for two sectors but not for health, education and employment these results were not pre-defined. The Evaluation Team Response #1: The evaluation team reworded text on page 10 to address comment #1 (please see reworded text below). “Achievements are registered in all sectors. However, the ACCOUNT Program Description (PD) defined anticipated results for two sectors against which achievement can be measured (public procurement and whistleblower protection), while for the remaining sectors (health, education and employment) the results were not predefined.” 133 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV IP’s Comment #2: Executive Summary- MAIN FINDINGS AND CONCLUSIONS: We cannot agree with the above statement, considering that the journalists of Žurnal are investigating and documenting all their allegations. USAID’s Comment #2 Executive Summary- MAIN FINDINGS AND CONCLUSIONS: Please re-phrase this to address CRMA’s comment. The fact is that USAID’s focus was firstly on quantity, however, investigative reports were done professionally to The Evaluation Team Response #2: Being assessed as good, very good and excellent suggest that reports were done in a professional manner. The evaluation team’s statement was based on an assessment made by two external and qualified investigative journalists who analyzed 30 randomly selected Zurnal.info and the Media Pool members articles. Each article was evaluated against nine (9) categories (please see Annex…..) for quality, including: 1) Fairness; 2) Accuracy; 3) Clarity; 4) Public Interest; 5) Respect; 6) Ethical Conduct; 7) Rights; 8) Advocacy; 9) Investigative Approach. For detailed a methodology and description of each of the categories including the results, please see Annex IV on Quality of Investigative Reports Evaluation. Furthermore, the limited sample size of for the evaluation of the investigative reporting has been identified as one of the constraints of this evaluation. Also, the text in the report was re-worded in response to comment #2: “Based on the randomly selected sample of ACCOUNT’s reports, the quality of ACCOUNT’s investigative reporting, in particular that of Zurnal.info, has been assessed as good on average, with one in every four reports being assessed as very good or excellent. The reporting was focused on quantity, advocacy, and drawing attention to corruption.” IP’s Comment #3: Executive Summary- MAIN FINDINGS AND CONCLUSIONS: Cooperation with CSOs was not a focus of work, because before ACCOUNT there were no media with which they could cooperate. The main goal was the establishment of a network of media that are ready to report about corruption USAID’s Comment #2 Executive Summary- MAIN FINDINGS AND CONCLUSIONS: I agree with CRMA’s comment. This was not the primary focus of ACCOUNT. Please delete/rephrase this comment. The Evaluation Team Response #3: The evaluation team deleted commented sentence. 134 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV USAID’s Comment #4: Why is this highlighted? (page 19) The Evaluation Team Response #4: Corrected in the final version submitted. IP’s Comment #5: Limitation 5: Model of reporting was approved by USAID. Each quarterly and annually report was approved also by USAID. Bearing in mind the complexity of program structure we believe that this way of reporting is clear and concise. IP IH: With the reform-oriented activities, the cross-section relations and interventions are meant to contribute to the overall project success, and as such we find the reporting style descriptive rather than limited. It certainly does not include overlaps but rather cross-sector relations. USAID’s Comment #5: Limitation 5: Please note that USAID has no template for IP reporting under the Assistance. Local partners are encouraged to report on achieved results using the most appropriate format. I do agree that the reporting could be improved, but as AOR I’ve received adequate information through Q and A Reports that enabled me to manage the activity. The Evaluation Team Response #5: Limitation #5 is deleted from the final version of the report. IP’s Comment #6: Limitation 6: We strongly believe that this way of conducting an evaluation was not adequate. Considering the amount of content produced by determining a random sample, only 16 texts out of a total of 1034 are not an absolutely reliable finding. A small sample and random sample could not reflect the real effects of Žurnal’s reporting. We believe that it is necessary to have in mind the moment of the publication of the text, the context, the sequence of events, follow – up reactions, etc. USAID’s Comment #6: Limitation 6: I respect the concerns that CRMA is raising, however, I would not change this Limitation 6. 135 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV The Evaluation Team Response #6: Limitation #6 is unchanged. IP’s Comment #7: Comment Ref. Chapter 4.1.; Evaluation Question 1; Finding 1: We have this information: CSOs: 137 Individuals: 22 Public Institutions: 8 Other: 2 USAID’s Comment # 7: Figure #7: Please double check these figures. It is strange to have n/a and the total of 169. The Evaluation Team’s Response #7: The breakdown by type of member of the ACCOUNT network was not found in project documents available for the desk review. The evaluation team inserted figures provided in comments by IPs in figure #7. Also, the evaluation team inserted the following note under figure #7: “Note: 2015 figures related to the Account network membership are provided by the IP. The documents made available to the evaluation team did not contain data on the breakdown of the ACCOUNT network membership,” IP’s Comment #8: Comment Ref. Chapter 4.1.; Evaluation Question 1; Finding 2 (#1): Achievement is not on target because we reported actuals for Y1 according to the in that moment valid definition of indicator (draft version of M&E from November 2016), while final version of indicator’ definition (final version of M&E Plan) was approved on March 2017. So, we tracked this indicator for Y1 based on one definition, and for Y2 an Y3 based on another definition. Anyway, we believe that we will reach cumulative four-year target. The Evaluation Team’s Response #8: The numbers are verified, and the text was re-worded to recognize the over-achievement in the Year 1: 136 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV “CSOs submitted 59 public policies, of which 33 were accepted and 26 are pending (ref. indicator: Number of public policy proposals submitted to executive governments and parliaments with citizens’ input). This achievement is below the target of 65 submitted policies. However, ACCOUNT is over-performing with respect to adopted policies. Over the last three years 33 policies have been accepted, exceeding the target of 27. To reach the cumulative four-year target, CSOs should submit additional 46 policy proposals, with additional 14 to be adopted.).” IP’s Comment #9: Comment Ref. Chapter 4.1.; Evaluation Question 1; Finding 2 (#2): The same case as with Indicator public policies. We have high over-performance in Year 1, because we tracked actuals based on in that moment valid definition. The Evaluation Team’s Response #9: The numbers are verified, and the text was re-worded to recognize the over-achievement in the Year 1: “The Number of policy recommendations provided by Sector Groups (SGs) and Monitoring Teams (MTs) also exceeded targets (76 recommendations have been provided compared to a target of 60 to date).. This was driven in part by the large number of recommendations in WBP and PP in the Activity’s first year. The number of policy recommendations in other sectors has either been at or below target over the past 3 years” IP’s Comment # 10: Comment Ref. Chapter 4.1.; Evaluation Question 1; Finding 2 (#3): Below? Target for Y3 is 5, while our Actual is 7. USAID’s Comment #10: Finding #2: please double check these figures before publishing the final report The Evaluation Team’s Response #10: The numbers are verified and the text was re-worded to recognize the over-achievement in the Year 1: “In the first three years of the Activity, 13 monitoring reports on sector-based corruption were prepared. This falls below the target of 15 reports (annual target is 5 reports with 1 report per sector). More than targeted number of reports were issued in Year 2 and Year 3, as a result of over performance in the PPL and HEALTH sectors.” 137 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV IP’s Comment # 11: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 5 (#1): We can’t’ completely agree with this statement. Each call for advocacy grant program was based on findings of Monitoring teams, as well as each call for monitoring and reporting grant program was based on SG inputs. For the example, SG advocated for adoption of RS Whistleblower Law, and after the Law had been adopted, MT through monitoring checked how this Law is implemented. E.g. advocacy in education sector was focused on introduction of anti-corruption classes in schools, while MT monitored implementation of this anti-corruption measures. So, we believe that deeper insight in monitoring reports and its comparation with advocacy activities can provide clear proof of synergy between components. USAID’s Comment #11: Finding #5: This is a bit too blunt statement. The Evaluation Team’s Response #11: The evaluation team deleted commented sentence. IP’s Comment # 12: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 5 (#2): I feel sorry for the person who had to do this chart. Yes we are aware of this complex structure, but it was the structure proposed by two IPs, and approved by USAID at the beginning. The Evaluation Team’s Response #12: This comment does not require any changes in the report. IP’s Comment # 13: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 6 (#1): Health is done at the cantonal level USAID’s Comment #13: Finding #6: Please make sure the most accurate information is used for the final report 138 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV The Evaluation Team’s Response #13: A footnote with an explanation was added to the report. IP’s Comment # 14: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 6; Figure #13 (#2): Council of Ministers’ Work Group adopted ACCOUNT recommendations on 11 articles. ACCOUNT’s recommendations are officially included in Draft law on changes of PP Law, which is submitted to Parliamentary procedure. (Source: Annual report for Y3)) USAID’s Comment #14: Finding #6: please double check the data. The Evaluation Team’s Response #14: Data was checked and changes were inserted accordingly. IP’s Comment # 14a: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 6, Figure #13 (#3): Wrong statement (original text: Developed template rulebooks for employment. Signed MoU‘s with 2 municipalities and 8 cantons, where the Rulebook was rolled out (2 municipalities adopted, others pending)). USAID’s Comment #14a: Finding #6: Again, please double check the data for final report The Evaluation Team’s Response #15: Corrected on the basis of the last IPs’ report on policy proposals. New text: Developed template rulebooks for employment for 2 municipalities (accepted) and 3 cantons (pending). Data were checked and the text was changed accordingly 139 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV IP’s Comment # 15: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 6 (#4): This is wrongly spelled, as no policies proposals are targeted in the education sector. The Evaluation Team’s Response #16: The sentence was deleted as it was not relevant for the finding. IP’s Comment # 16: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 6 (#5): This score is based on CSO’s and Media perception, but we should have in mind that CSOs and Media wasn’t directly involved in PPL sector’ activities. All advocacy activities in PPL sector were conducted by IP CRMA because CSOs PPL applications received on 2nd and 3rd round of the Small Grant program did not meet selection criteria and therefore no CSO was awarded. USAID’s Comment #15: Keep the text. CRMA’ info is just FYI The Evaluation Team’s Response #17: This remark does not require any changes in the report. USAID’s Comment #16: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 6 (#6): I understand that it is hard to find a pattern in a complex system such as AC development in BiH. ACCOUNT is working in five areas, there are numerous levels of government open to different AC interventions. ACCOUNT was responding to the raising opportunities, meaning requests of certain government or interests of local NGOs. The problem is not if one NGO works on two issues, the problem is that we don’t have more AC NGOs. As IH explains above, these are not overlaps, but perhaps the open opportunities. The Evaluation Team’s Response #18: The critical sentence (second sentence in the paragraph) was re-worded and it states: “For example, Activity reports indicate that there were many activities implemented in parallel in the sectors of HEALTH, EMPL, EDU and local / cantonal level activities.” 140 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV IP’s Comment # 17: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 6 (#7): Health institutions and education institutions are cantonal-based/budgeted institutions. The Evaluation Team’s Response #19: The sentence was amended to indicate that the activities refer to cantonal institutions. IP’s Comment # 18: Comment Ref. Chapter 4.1.; Evaluation Question 2; Finding 6 (#8): Activities of the sectoral employment group are directed at the local level. No activities have been implemented covering the entire BiH area, just on cantons and municipalities. Although activities were not implemented at the state level, the effect applies to all citizens in BiH. The Evaluation Team’s Response #20: No changes were made in the report since the relevant details are provided in the map legend. IP’s Comment # 19; Finding #9: There is baseline for this indicator – 0. We also need to add that independent survey was conducted by USAID expert Valery Peri in which she conclude that beside Žurnal and CIN there is no media in BIH who investigate and report about corruption. The Evaluation Team’s Response #21: Baseline data for investigative reports for corruption do not exist for pre-ACCOUNT period in a form which can be confirmed and used by the evaluation team. However, the critical sentence (second sentence in the paragraph) was re-worded and it states: “The evaluation team could not access any data on activity and productions levels of individual media outlets pre-ACCOUNT with respect to investigative reporting that could confirm any of the perceptions expressed during the KIIs and Panel Discussion.” 141 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV IP’s Comment # 20; Finding #10: We would like to highlight a few things: Within limited funds we were not able to compete with PBS. We have been offering televisions to broadcasting TV Affair for free in order to increase audience reach. Most of stories have been published on TV (BN TV, Al Jazeera Balkans). Also, in order to increase public reach, we printed the Affair in the form of a newspaper and shared to citizens. Please, see more detailed comment on addition document. The Evaluation Team’s Response #21: The critical sentence (second sentence in the paragraph) was re-worded and it states: “Although the Activity made a significant effort to increase outreach to the general population, most respondents agree that competing with public broadcasters still remains challenging in a context where citizens heavily rely on traditional media sources for information.” IP’s Comment #21; Finding #10: This is totally understandable because most of published content on Žurnal includes government institutions and government representatives, so this is not surprise that they don’t want to comment their self. USAID’s Comment # 17; Finding #10 I agree The Evaluation Team Response #22: The evaluation team agrees that this could be a reasonable explanation. The new sentence is added: “This may be because most of published content includes government institutions and their representatives.” IP’s Comment # 21; Finding #10: In case of financial support we strongly believe that we will be able to encourage citizens to consume online platforms. Combination of online and print version was very effective (to print free newspaper with corruption related content on monthly basis). The Evaluation Team Response #23: This comment does not require any changes in the report. 142 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV IP’s Comment # 22; Finding #11, Figure 19: Previously (Figure 5. ACCOUNT Investigative reports sampling) was mentioned number 17? The Evaluation Team Response #24: Mistake made in figure #5 is corrected. IP’s Comment # 23: Finding #11: The volume was defined by indicators and program description. The Evaluation Team Response #25: This comment does not require any changes in the Report. IP’s Comment # 24: Finding #13, Figure #20: Unclear. We don’t understand? The Evaluation Team Response #26: All 0 values in the table also include values below 1 because figures were automatically rounded by excel –zero is changed into 1%. MEASURE COR Comment #1: Please have conclusions after each evaluation question. The Evaluation Team Response #27: Addressed in the final report MEASURE COR Comment #2: Conclusion #5: This is a finding, not a conclusion The Evaluation Team Response #28: Addressed in the final report 143 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV The Evaluation Team Response of IP’s Comments on the Evaluation Statement of Work (ANNEX 1: EVALUATION STATEMENT OF WORK) Please note that the Evaluation Statement of Work (SoW) is not a living document and it was designed almost a month before the evaluation team commenced with evaluation work and data collection. Also, please note that all figures stated in the SoW are based on the first two annual reports (at that time, these were the only reports available to the evaluation team). ACCOUNT’s comments on results stated in the SoW are updated results from FY 3. IP’s Comment #1; Background page #59: Where in the evaluation process this was found out? The Evaluation Team Response #1: The background section of the SoW was written based on information from the first two annual reports. IP’s Comment #2: Background page #60: This number is wrongly spelled. From the Y3 report: ACCOUNT noted successful introduction of anti-corruption pilot classes in 75 elementary and secondary schools in Herzegovina-Neretva Canton, Sarajevo Canton, Middle Bosnia Canton in the Federation of B&H, Bijeljina and Trebinje regions in the Republika Srpska entity and Brčko District The Evaluation Team Response #2: The background section of the SoW was written based on information from the first two annual reports. IP’s Comment #3; Background page #62: Data from the Y3 Report: The latest ACCOUNT monitored development and evaluation of plans of integrity for 1091 public institutions, out of which 659 were successfully completed and submitted to the RS Ministry of Justice The Evaluation Team Response #3: The background section of the SoW was written based on information from the first two annual reports. 144 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV IP’s Comment #4; Background page #62: In the first two years, ACCOUNT published 865 investigative reports (Y1 – 321, Y2 – 544). The Evaluation Team Response #4: The evaluation team cannot confirm or discard this statement. ACCOUNT Y1/Y2 Annual and Quarterly Reports do not show consolidated numbers of the published investigative reports. Based on this fact we have formulated a flexible conclusion in our finding: “…based on the ACCOUNT M&E Plan, they have produced and published over 765 investigative reports (in the first two years)”. IP’s Comment #5; Background page #63: 46, as above The Evaluation Team Response #5: The evaluation team found the next statement in Y1/Q4 Annual Report: “In the Year 1, fifteen new (15) cases of corruption were filed with the Legal Team, out of which two (2) cases with positive outcomes. Five (5) cases were rejected as having no elements of corruption. Two (2) cases were closed, and those who reported corruption were advised on further steps. The Legal Team also worked on monitoring of court proceedings of six (6) cases.” In Y2/Q3, ACCOUNT confirms the following facts: “Court proceedings for the 2 cases are in progress. The Legal Team continued to work on the 22 active cases (this number includes all cases that were received from the project start)”. Accordingly, it is not clear for how many cases in total they have provided assistance, since the assistance for some cases in Y1 was been continued in Y2. We cannot confirm the assistance for 46 cases in total, for Y1 and Y2. ANNEX II: SUB-AWARDS AND ROLES OF CSOs IP’s Comment #6: Figure Awards by type and per Activity page #67 In addition to the listed awards, two more CSOs received a Small grants awards – Foundation Cure and Association of Citizens Grahovo were awarded in Y3. So, we suggest to add these two CSO’s as C1-Sector member organizations/Whistleblowers/Y3 The Evaluation Team Response #6: The evaluation team has found, based on a list of all grantees received from ACCOUNT, that these two organizations were awarded grants for the support to women candidates in the elections (woman election rights). We never evaluated that modification and therefore these two organizations were not included in this evaluation report. IP’s COMMENTS ON ANNEX IV - Complete List of Key Informants The Evaluation Team Response #7: Addressed. 145 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV IP’s COMMENTS ON ANNEX VI – Account Google Analytics for Investigative Reporting IP’s Comment #7: Figure 8: Comparison of available Zurnal.info and Interview.ba Audience Report Data The project goal targeted no comparison of news report data provided by Zurnal.Info and Interview.ba as totally incomparable for the following reasons: 5. Different budgets. From the overall approved budget in the amount of USD 324,000 for these two media outlets: 92, 5 % of the overall budgeted amount was allocated to Žurnal while 7, 5 % was allocated to Intreview.ba. This difference was significantly increased through allocation of additional funds to each of these two media outlets, so that 96, 5% of the overall additional funds was allocated to Žurnal and 3, 5% was allocated to INFOHOUSE. 6. Different time of formation/baselines of each of these media outlets 7. Different media profiles 8. Different size of media staff involved in news production The Evaluation Team Response #8: Figure #8 is deleted from the final version of the report. 146 | PERFORMANCE EVALUATION OF ACCOUNT FOLLOW-ON ACTIVITY USAID.GOV MONITORING AND EVALUATION SUPPORT ACTIVITY (MEASURE-BiH) Fra Anđela Zvizdovića 1 UNITIC Tower B, Floor 21 71000 Sarajevo Bosnia and Herzegovina PHONE: + (387) 033 941 676 contacts@measurebih.com www.measurbeih.com