Strengthening Civil Society’s Legitimacy, Competence, and Professionalism and Increasing Its Influence on Public Policies in Bosnia and Herzegovina: Performance Evaluation of USAID/BiH Civil Society Sustainability Project (CSSP) SEPTEMBER 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by IMPAQ International under USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH) STRENGTHENING CIVIL SOCIETY’S LEGITIMACY, COMPETENCE, AND PROFESSIONALISM AND INCREASING ITS INFLUENCE ON PUBLIC POLICIES IN BOSNIA AND HERZEGOVINA: PERFORMANCE EVALUATION OF USAID/BIH CIVIL SOCIETY SUSTAINABILITY PROJECT (CSSP) September 2016 NAIDA CARSIMAMOVIC VUKOTIC KRISTIE EVENSON EMINA COSIC PULJIC ANESA HADZIC DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS ACKNOWLEDGMENTS ................................................................................................ III ACRONYMS..................................................................................................................... IV EXECUTIVE SUMMARY .................................................................................................. 1 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS ............................ 7 1.1 EVALUATION PURPOSE .................................................................................................................... 7 1.2 EVALUATION QUESTIONS .............................................................................................................. 8 2. CSSP BACKGROUND ............................................................................................... 9 3. EVALUATION METHODS AND LIMITATIONS ................................................. 18 3.1 METHOD DESCRIPTION ................................................................................................................ 18 DATA COLLECTION ........................................................................................................................................................................................... 18 3.2 EVALUATION LIMITATIONS ......................................................................................................... 20 4. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ............................. 22 4.1 FINDINGS ............................................................................................................................................ 22 EVALUATION QUESTION 1 ........................................................................................................................................................................... 22 EVALUATION QUESTION 2 ........................................................................................................................................................................... 29 EVALUATION QUESTION 3 ........................................................................................................................................................................... 37 4.2 CONCLUSIONS ................................................................................................................................. 54 4.3 RECOMMENDATIONS .................................................................................................................... 58 ANNEXES ........................................................................................................................ 62 ANNEX I EVALUATION STATEMENT OF WORK ......................................................................................... 63 ANNEX II EVALUATION MATRIX ....................................................................................................................... 75 ANNEX III DOCUMENTS REVIEWED ................................................................................................................. 76 ANNEX IV DATA COLLECTION INSTRUMENT – KEY INFORMANT SEMI-STRUCTURED INTERVIEW PROTOCOLS ...................................................................................................................................... 77 ANNEX V DATA COLLECTION INSTRUMENT – ONLINE SURVEY OF CSSP SECTOR NETWORK MEMBERS/PARTICIPANTS .............................................................................................................. 88 ANNEX VI LIST OF KEY INFORMANT INTERVIEWEES ............................................................................... 97 ANNEX VII CSSP OCA AREAS AND INDICATORS ....................................................................................... 99 ANNEX VIII PURPOSE OF GRANT FOR CAPACITY BUILDING AND SUSTAINABILITY .............. 100 ANNEX IX CATEGORIZATION OF NETWORKS’ CAMPAIGNS BY POLICY LEVEL ...................... 104 ANNEX X DISCLOSURE OF ANY CONFLICTS OF INTEREST ............................................................... 110 ANNEX XI MEASURE-BIH RESPONSES TO COMMENTS ON DRAFT CSSP PERFORMANCE EVALUATION REPORT ........................................................................................................................................ 114 ii FIGURES Figure 1. Illustrative Hierarchy of Expected Results of CSSP .............................................................. 10 Figure 2. CSO Sector Leaders/Co-Leaders ............................................................................................. 11 Figure 3. CSSP’s Tasks and Results Envisaged by Program Design, and M&E Indicators Reported by CSSP: Activity Goal .................................................................................................................................. 13 Figure 4. CSSP’s Tasks and Results Envisaged by Program Design and M&E Indicators Reported by CSSP: Objective I...................................................................................................................................... 14 Figure 5. CSSP’s Tasks and Results Envisaged by Program Design and M&E Indicators Reported by CSSP: Objective 2 .................................................................................................................................... 15 Figure 6. CSSP’s Tasks and Results Envisaged by Program Design and M&E Indicators Reported by CSSP: Objective 3 .................................................................................................................................... 16 Figure 7. Overview of Grant Purposes for CSO Sector Leaders/Co-Leaders ................................ 17 Figure 8. Number of Sector Network Member/Participants and Survey Responses ..................... 20 Figure 9. Composition of Sector Network Members/Participants .................................................... 23 Figure 10. Sector Network Members/Participants’ General Satisfaction with the Work of Networks......................................................................................................................................................... 24 Figure 11. Networks’ Methods for Identifying Sector Priorities ......................................................... 26 Figure 12. Sector Network Members/Participants’ Involvement ........................................................ 27 Figure 13. CSSP CSOs’ OCA 2014 and 2016 Scores ............................................................................ 31 Figure 14. Sector Network Members/Participants’ Feedback on Adequacy of Priorities ............. 33 Figure 15. Alternative Sources of Funding Planned within Sustainability Strategies ....................... 35 Figure 16. Categorization of CSSP Sector Priorities for Advocacy Campaigns by Level of Policy Influence ........................................................................................................................................................... 38 Figure 17. Policy Influence Levels (as adapted from Lindquist/Carden) ............................................ 41 Figure 18. Sector Network Members/Participants’ Feedback on Priorities for CSSP’s Work on Improving Policy Environment for CSOs ................................................................................................. 48 Figure 19. Sector Network Members/Participants’ Feedback on Familiarity and Progress of CSSP’s Work on Improving Policy Environment for CSOs ................................................................. 49 Figure 20. Feedback from Sector Network Members/Participants on CCI’s Monitoring of Parliaments and Governments .................................................................................................................... 53 iii ACKNOWLEDGMENTS The authors thank a variety of individuals who have contributed to the implementation of the CSSP performance evaluation and the analysis for the preparation of this report. At USAID/Bosnia and Herzegovina (USAID/BiH), Elma Bukvic Jusic, our Contracting Officer’s Representative (COR), provided valuable guidance throughout evaluation. Selma Sijercic, CSSP Agreement Officer’s Representative (AOR) contributed greatly to this evaluation by helping us understand the CSSP design and its management. We also thank numerous CSSP implementation team members for their assistance and prompt response to our information requests. Our colleagues at IMPAQ have provided valuable guidance and also made significant contributions to the content and structure of this report. Ye Zhang provided unparalleled technical leadership. Aida Hadzic provided excellent research assistance. Finally, we thank USAID/BiH, CSSP implementers, and CSSP sub-grantees that provided comments to the draft evaluation report. All the above organizations and individuals have contributed tremendously to this report and to the MEASURE-BiH activity more generally, but any remaining errors are the fault of the authors alone. iv ACRONYMS ACED Agency for Cooperation, Education and Development ADI Association for Democratic Initiative AOR Agreement Officer Representative ARD Association for Rural Development BiH Bosnia and Herzegovina CARD Center for Agriculture and Rural Development CAPP I Citizen Advocacy Partnership Program I CAPP II Citizen Advocacy Partnership Program II CBGI Capacity Building of Government Institutions, EU-financed project CCI Centers for Civil Initiatives CID Center for Informative Decontamination of Youth CIN Center for Investigative Journalism CoM Council of Ministers of BiH COR Contracting Officer’s Representative CPCD Center for Promotion of Civil Society CPI Public Interest Advocacy Center CRMA Centre for Development of media and analysis CRP Center for Development and Support CSO Civil Society Organization CSSP Civil Society Sustainability Project DEI Directorate for European Integration DRG Democracy, Human Rights, or Governance EC European Commission EE&RES Energy Efficiency and Renewable Energy Measures ET Evaluation Team EU European Union FBiH Federation of Bosnia and Herzegovina FSU Foundation for Social Inclusion IBHI Initiative for Better and Humane Inclusion ICVA Initiative and Civil Action IT Information Technology v KII Key Informant Interview LiNK Association for Entrepreneurship and Business Mostar MDP Centre for Management, Development and Planning Initiatives MEASURE-BiH USAID/BiH Monitoring and Evaluation (M&E) Support Activity M&E Monitoring and Evaluation NBR Independent office for Development NGO Non-Governmental Organization OCA Organizational Capacity Assessment OKC Youth Communication Center PfH Partnership for Health PR Public Relations PREDA Agency for Economic Development Prijedor-PD RS Republika Srpska SNA Social Network Analysis SOW Statement of Work TACSO Technical Assistance for Civil Society Organizations TI Transparency International U.S. United States USAID U.S. Agency for International Development USAID/BiH U.S. Agency for International Development in Bosnia and Herzegovina USAID/W U.S. Agency for International Development in Washington USG U.S. Government VAT Value-added tax 1 EXECUTIVE SUMMARY The Civil Society Sustainability Project (CSSP) is a $9.15 million activity financed by the United States Agency for International Development (USAID) to strengthen and maintain the capacity of Bosnia and Herzegovina (BIH) civil society to influence the creation and implementation of public policies of interest to citizens. USAID designed CSSP interventions to address five main problems: i) lack of government engagement with civil society, ii) inability of Civil Society Organizations (CSOs) to influence public discourse, iii) lack of CSO professionalism, iv) weak sustainability prospects of the civil society sector, and v) lack of reliable and unbiased information. The CSSP intervention began on September 1, 2013 and is expected to last until August 30, 2018. USAID/BiH has commissioned IMPAQ International (IMPAQ) under USAID/BiH Monitoring and Evaluation Support Activity (MEASURE-BiH) to conduct a performance evaluation to examine the CSSP intervention. This evaluation covers implementation up to April 2016 and was conducted in May and June 2016. Evaluation Purpose and Evaluation Questions Findings, conclusions, and recommendations of the CSSO performance evaluation will contribute to the knowledge and learning of several stakeholder groups: • USAID/BiH will be able to reassess the role of CSSP in strengthening and maintaining the capacity of BIH civil society to influence the creation and implementation of public policies of interest to citizens. The evaluation will help USAID/BiH staff achieve a better understanding of activity implementation, lessons learned, and best practices, which can inform adjustments of current interventions if needed, as well as future funding decisions and program designs in this area. • CSSP Implementing partners will learn about their strengths and areas for improvement. • Other USG stakeholders (including USAID/W and the U.S. Embassy) will better understand the USAID-funded civil society interventions in BiH. • Other stakeholders, including the BiH governing institutions, CSOs, and other international development donors and partners, may also benefit from USAID’s contribution to public knowledge of the most recent development efforts in strengthening BiH civil society. This evaluation answers the following four research questions, informed by CSSP’s expected results, structure, and logic model: 1. To what extent have interventions under CSSP increased CSO partnership and advocacy agenda consensus building with other stakeholders (government, media, and private sector)? 2. To what extent have interventions under CSSP strengthened the capacity and viability of selected CSOs in the 12 sectors? 3. To what extent have interventions under CSSP increased engagement of civil society and citizens in governance, and what was achieved within the 12 selected reform areas? 4. Which performance-related intervention factors could be adjusted to address intervention concerns and align CSSP activity with its articulated objective for 2018, and based on CSSP’s achievements and challenges, what are the lessons learned for future interventions in this area? The first three evaluation questions are addressed throughout the section on Findings and Conclusions. Based on these, the fourth evaluation question is addressed in the section on Recommendations. CSSP Background The USAID/CSSP activity aims to increase citizen participation in governance and hence increase accountability through achieving three objectives: 2 I. increased partnership and consensus among different stakeholders in support of selected advocacy agendas; II. strengthened capacity and viability of core civil society partners and of the broader civil society sector; and III. increased engagement of civil society and citizens in policy development and in government monitoring and oversight at the local, Entity, and State level. CSSP’s implementation is two-pronged: 1. Tasks led by CSSP implementers Centers for Civic Initiatives (CCI) and Center for Promotion of Civil Society (CPCD), which lead the campaigns for strengthening an enabling CSO environment in BiH, work on strengthening the selected CSOs’ capacities, and conduct comprehensive monitoring of the performance of BiH authorities. 2. Tasks implemented by the 12 sector networks1 led by 19 CSO Sector leaders/co-leaders through three grants for each of the 12 sectors, administered and managed by CCI. The first grant, in the amount of $50,000, was awarded for establishing sector networks to increase engagement of CSO sector leaders/co-leaders with other key stakeholders (other CSOs, citizens, government, private sector, academia, and media). The second grant, in the amount of $100,000, was awarded for strengthening internal capacities and organizational structures of the 19 CSO sector leaders/co￾Leaders to ensure sustainability and financial viability when CSSP ends. The third grant, in the amount of $250,000, is for public advocacy campaigns and monitoring. Evaluation Methods and Limitations Evaluation team employed a mixed methods in conducting this performance evaluation. Although performance evaluation cannot assess extent to which results of intervention can be attributable to an intervention, it can provide valuable information on how an intervention is being implemented, how it is perceived and valued, and to what extent the expected results are occurring. The evaluation team drew on analysis of a wide array of quantitative and qualitative data to inform and answer evaluation questions and sub-questions, including: I. Desk research of CSSP activity documentation and other secondary sources II. Semi-structured Key Informant Interviews (KIIs)2 III. Online survey of the sector network members/participants The data from above-listed sources are triangulated to address the same evaluation questions from multiple perspectives. We encountered a number of limitations during this evaluation—such as response and selection biases and bias associated with the small sample of sector network members/participants and government institutions interviewed—and developed mitigation strategies to overcome many of them. Findings Evaluation Question 1: Stakeholder partnership includes a broad set of actors across twelve sectors and CSSP networking efforts have fostered interests in continued cooperation among the current sector members/participants. However, while there is evidence of broad network formation efforts, the extent to which CSSP networks 1 The 12 sectors are: 1) anti-corruption; 2) employment and labor market, 3) economic policy, 4) education, 5) health care, 6) human rights of marginalized groups, 7) women's rights, 8) agriculture and rural development, 9) culture, 10) public finances, 11) environment protection and energy efficiency, and 12) the justice sector. 2 Information we received from the key informant interviews is paraphrased in text boxes throughout the report. 3 represent new levels of partnership is unclear, as are the breadth of stakeholder membership and depth of their involvement. Levels of broader stakeholder participation in networks’ thematic development and campaign implementation vary considerably, with more evidence of participation in issue identification than in the campaign design and implementation. Strategic inter-sector thematic partnerships have not been developed, partly as a result of implementers’ primary focus on administrative and operational guidance, as opposed to strategic guidance. Defined thematic areas vary in breadth and are not conducive to building up CSO sector expertise, as in some cases they result in narrow and overlapping foci. CSSP’s fragmented sub-grant schemes discourage comprehensive network planning. And donor decisions have led to some mixed signals and/or duplications among the CSOs, sectors, and/or funded initiatives. Evaluation Question 2: CSSP’s assistance in supporting sustainability of the CSO sub-grantees and application of Organizational Capacity Assessment (OCA) tool is perceived by all CSSP CSOs as innovative in the BiH context and useful. Improvement measures identified on the basis of the Organizational Capacity Assessment (OCA) process and subsequently included in the Sustainability Strategies and Communications Strategies of the CSSP CSOs are being implemented and most CSOs note that there have been organizational improvements resulting from CSSP’s assistance. However, most of the key tasks articulated within the applications for CSSP’s capacity building and sustainability grant are common across sub-grantees and focus on internal organizational procedures and operational capacities. M&E capacities of CSSP CSOs are sub￾optimal. Sub-grantees appreciate CSSP’s focus on sustainability and the freedom and flexibility in identification of priorities, in tasks that can be financed by sub-grants, and in ability for adjustments during implementation. They view this freedom and flexibility as contributing to building their leadership capacities. However, insufficient focus is given to building CSO sub-grantees’ sector-specific expertise in most cases, and there is some lack of understanding of how this segment of assistance to networks should be connected to the other two segments (establishing networks and advocacy campaigns and monitoring). Collaboration among the CSSP CSOs (in both content related to sectors and internal operational procedures) has not been used as a CSO capacity-building mechanism in CSSP. All the CSSP CSOs identified some targeted alternative ways of funding (other than foreign donations) within their Sustainability Strategies prepared under CSSP. Based on KIIs, around half the CSSP CSOs secured alternative funding sources in the last two years, or are about to do so. But assistance provided in identifying alternative funding is perceived as not being tailored enough to CSO type. Several CSOs perceive that CSSP implementers’ preferred option is registering an enterprise and several other aspects of sub-grant management are perceived as counter￾intuitive and clumsy, particularly in terms of the quantitative target for decreased dependence on foreign donors. CPCD’s Resource Centre (RC) has functioned primarily in parallel to CSSP, and neither CSSP reporting or KIIs suggest much synergy between the work of the RC and CSSP. Evaluation Question 3: The 40 chosen advocacy sector campaigns vary greatly among different networks, in definitions and scope of targeted policies, their political context, and their maturity/ripeness. CSSP reports describe all networks as having impacted government policies through CSSP, however, these claims cannot be confirmed because CSSP did not design campaigns with a clear way of tracking how their inputs result in influence, as illustrated by uneven CSSP reporting. The evaluation team’s analysis of campaigns’ policy influence shows that varying types of campaigns, levels of campaign specificity, and policy issue maturity levels result 4 in a mix of policy influence during the first two and a half years of CSSP. Of 15 campaigns described as achieving some level of decision regime influence, eight describe inputs into already adopted implementable legislation/strategic documents; the remaining are at earlier stages of the policy making process. Several aspects of up-to-date CSSP implementation emerge as success factors for maximum influence on decision regimes, defined as providing relevant and credible policy inputs to government and governments’ subsequent acceptance of those inputs: I. Evidence-based research/analytical products and concrete actionable technical recommendations, II. Meaningful consultation processes and a common approach to practical solutions among practitioners, III. Being able to bring together a combination of citizens and experts around the same priority, IV. Being able to seize political context and policy maturity, V. Media and public outreach and innovation in campaign visibility, VI. Use of CSSP infrastructure for reaching wider constituencies, VII. Campaign strategy and vision While the CSSP design emphasized that the policy influence should be focused on areas relevant for accelerating the process of European Union (EU) integration, CSSP implementation focuses little on policy changes specifically needed to accelerate EU integrations at this stage. Only a few sectors report on some connections with BiH’s immediate EU integrations requirements. The CSSP management process is perceived as overemphasizing public campaigning. CSSP management’s primary focus on administrative and operational guidance resulted in insufficient advice for campaign design and implementation for the sub-grantees that might need it. Design of the sub-grant for policy campaigns did not emphasize strategy development or relevant tools to manage how to track the effectiveness of a campaign’s strategy or approach. CSSP reporting seems to overstate influence and achievements of the campaigns. Only a few CSSP CSOs systematically monitor policy implementation and policy change processes in their sector. Activities led by CSSP implementers in creating an affirmative work environment for CSOs are perceived as generally well-targeted, and are mostly a continuation of efforts previously invested by the CPCD and other CSOs over the last decade. Significant preparatory work has been undertaken by CSSP implementers in all five campaigns advocating for CSO policy environment improvement, with some success already recorded in adoption of new policies in two out of five initiatives - submitted policy proposals have been partially taken into account in recently adopted legislation/methodology relevant for transparent system of public funding and a more enabling taxation framework for philanthropy in Republika Srpska (RS). However, awareness of and participation in efforts related to the CSO policy environment by other CSOs/networks is low and they perceive progress in these efforts as weak. Key officials from relevant government institutions are not sufficiently involved in CSSP’s work related to CSO policy environment, and coordination with other donor interventions in the CSO sector is generally perceived as insufficient. Additional issues perceived as priority but not addressed by CSSP include setting-up a mechanism for co-financing of EU funds for civil society, introducing legislation on social entrepreneurship, and campaigning for more transparent and coordinated donor funding for the civil society. CCI’s reports on monitoring of the work of governments and parliaments are perceived as useful. CCI’s recent efforts on focusing more on quality rather than quantity is welcomed by all stakeholders; however, all agree that there is space for further improvement in both: (1) analyzing the quality of authorities’ work and actual reasons for progress or lack of progress and (2) a presentation that is more reader-friendly for the public and more dialogue-enticing for the authorities. 5 Recommendations As a result of our evaluation (and to address the fourth evaluation question), we offer the lessons and suggestions for adjustment of CSSP in the remaining two years of implementation, as well as for any potential future interventions in the civil society sector. The recommended adjustments for the remaining CSSP programming time are mindful of general budgetary and management resources present (to the extent known by the evaluators) and primarily focus on ways to shift, accentuate, and utilize resources available in the most effective ways possible to address CSSP objectives. These include: 1. Continue to encourage further expansion and engagement with a diverse set of network members in particular to government and media representatives. 2. Engage network members more in campaign efforts and substantive campaign implementation, 3. Further facilitate collaboration across CSSP sectors and CSOs, to take advantage of peer-learning potential for capacity building in terms of the internal operational capacities of organizations. 4. Provide meaningful coordination and facilitate collaboration across CSSP networks and CSOs to take advantage of peer-learning potential for thematic expertise capacity building. 5. Coordinate closely among USAID same-sector initiatives, and among USAID CSO initiatives. 6. Consider closer coordination with other donors (particularly EU) at both donor and initiative level. 7. In future donor interventions, if network approach is used, select sectors in a more systematic way and align more with sectors related to EU-funding structure. 8. Minimize fragmentation of sub-grants in future donor interventions in CSO sector. 9. Continue providing assistance to sub-grantees to identify and implement measures targeted at improving CSO organizational sustainability. However, provide more hands-on guidance to focus on identifying needs and providing assistance to strengthen CSOs’ sector-specific thematic expertise, including through technical assistance and peer-learning. 10. Improve the M&E capacity building of all CSSP CSOs within further building of organizations’ operational capacities, including capacity for CSOs’ own project-based M&E practices, as well as capacity to monitor and evaluate public policies based on evidence-based research. 11. Continue to provide assistance related to identification and operationalization of potential alternative funding sources of CSOs, but in a more custom-made approach—to take into account different CSO types and visions (advocacy CSO, watchdog CSO, service provider to citizens from vulnerable groups, representing interests of associated private sector representatives, service provider to government, or think tank). Clarify that the quantitate target for decreased dependence on foreign donor funding (30% of core budget being funded from sources other than foreign donors) is not mandatory for continued implementation of CSSP. Consider possible additional analyses of achievement measures by this indicator by analyzing different assumptions made by CSOs about the level of their core budget. 12. Consider how to more fully use CPCD’s Resource Center to serve as coordination mechanism for the CSSP activity. Link Resource Center activities and efforts with CSSP objectives more clearly. 13. Improve CSSP reporting to more realistically capture the actual influence, especially from the perspective of different phases of each policy’s maturity/ripeness. Consider tracking campaigns according to policy influence types to better understand actual influence and manage expectations. 14. Share success factors of campaigning among networks. 15. Consider providing technical assistance (from other network partners or outside) to networks, to refine the design of campaign strategies (where wanted) and introduce/improve evidence-based policy proposals. 16. Include EU-integration dimension in network campaigns where possible, and coordinate with the DEI. 17. Modulate the emphasis on public protest campaigning where not appropriate. 18. Refine campaign management tools to assist partners to track the development and implementation of campaigns. 6 19. In future donor interventions, consider grant structure around more articulated strategic design including theory of change hypothesis, logical framework, or other tools. 20. Consider specific skill building in monitoring implementation of existing policies and encourage the CSOs to more systematically and actively use existing and planned mechanisms for civic participation in governance. 21. Continue work on current campaigns for creating an affirmative work environment for CSOs, prioritizing further improvement in increasing the transparency of the current system of public funding of CSOs. Activities on the establishment of public foundations should be abolished. Also, invigorate network participation in strategizing and advocating for CSO policy environment campaigns. Consider including additional campaigns on setting-up a mechanism for co-financing of EU funds for CSOs and social entrepreneurship legislation. Work more closely with the relevant authorities at operational level in all campaigns and continue increasing cooperation with other donor interventions in these areas. 22. Continue the work on monitoring sessions of the parliaments and governments, but with increased analyses of the quality of work of these authorities. Improve presentation and communication of findings of this work so that it is in a more reader-friendly format for the citizens and more dialogue￾enticing for decision-makers. 23. From a broader perspective, consider the following: - Manage expectations and breadth of a single intervention, consider distinguishing design by different objectives and different types of CSOs (service providers to citizens, service providers to government, general think tanks, advocacy CSOs, watchdog CSOs). - Promote different types of civic engagement to focus on educating citizens and encouraging volunteering. 7 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE Guided by the U.S. Agency for International Development (USAID) Evaluation Policy, USAID/BiH commissioned IMPAQ International through USAID/BiH Monitoring and Evaluation (M&E) Support Activity (MEASURE-BiH) to design and conduct performance evaluation of the USAID/BiH’s Civil Society Sustainability Project (CSSSP). The evaluation adopts a rigorous design and methodological approach to address the evaluation questions and to make links between findings, conclusions, and recommendations related to the extent of CSSP achievements in strengthening civil society to influence the democratic governance reform process in BiH. USAID/BiH will use this evaluation to examine the results of the first two and a half years of CSSP activity and the effects of the program on overall civil society development in the country. The specific purpose of this CSSP performance evaluation is three-fold: 1. to assess CSSP’s progress toward stated objectives, 2. to identify obstacles and opportunities to activity implementation, and 3. to provide recommendations for the adjustment of the CSSP intervention in its remaining implementation period. A key component of the evaluation design is inclusion of a diverse set of stakeholders in the evaluation process. This includes CSSP implementers, CSO partners, the broader CSO community, media, government officials, business actors engaged with the CSSP interventions, and other external policy actors. The findings, conclusions, and recommendations of this CSSO performance evaluation will contribute to the knowledge and learning of several stakeholder groups 1. USAID/BiH will be able to reassess the role of CSSP in strengthening and maintaining the capacity of BiH civil society to influence the creation and implementation of public policies that are of interest to citizens. The evaluation will help USAID/BiH staff achieve a better understanding of activity implementation, lessons learned, and best practices, which can inform adjustments of current interventions if needed, as well as future funding decisions and program designs in this area. 2. CSSP Implementing Partners will learn about their strengths and areas for improvement. 3. Other U.S. Government (USG) stakeholders (including USAID/W and U.S. Embassy) will better understand the USAID-funded civil society interventions in BiH. 4. Other stakeholders, including the BiH governing institutions, CSOs, and other international development donors and partners, may also benefit from USAID’s contribution to public knowledge on the most recent development efforts in strengthening BiH civil society. The CSSP intervention began on September 1, 2013 and is expected to last until August 30, 2018. This evaluation covers the implementation period between August 2013 and April 2016. The MEASURE-BiH team conducted this evaluation in May and June of 2016, with the field work taking place between May 23rd and June 13th. The Evaluation Statement of Work (SOW) is given in Annex 1. 8 1.2 EVALUATION QUESTIONS The evaluation answers the following research questions, informed by CSSP’s expected results, structure, and logic model: 1. To what extent have interventions under CSSP increased CSO partnership and advocacy agenda consensus building with other stakeholders (government, media, and private sector)? Sub-questions: Which types of assistance have been most effective in achieving these partnerships with each of the type of stakeholder? Which have been less effective? 2. To what extent have interventions under CSSP strengthened the capacity and viability of selected CSOs in the 12 sectors? Sub-questions: How was the assistance (in particular grant mechanisms) to the CSOs in the 12 sectors designed and implemented? Which types of interventions have been most and least effective in meeting the needs of the CSOs? 3. To what extent have interventions under CSSP increased engagement of civil society and citizens in governance, and what was achieved within the 12 selected reform areas? Sub-questions: Which types of policy influence are being achieved and what factors are contributing to this? What were the success factors in providing relevant and credible policy inputs to government and government’s acceptance of those inputs? 4. Which performance-related intervention factors could be adjusted to address intervention concerns and align CSSP activity with its articulated objective for 2018, and based on CSSP’s achievements and challenges, what are the lessons learned for future interventions in this area? The first three evaluation questions are addressed throughout the section on Findings and Conclusions, while based on these three, the fourth evaluation question is addressed in the section on Recommendations. The questions are defined to broadly follow the three objectives of CSSP defined in the CSSP Program Description, as listed below, within USAID/BiH’s Agreement with the CSSP Implementers of August 2013: 1. Objective 1: Increased partnership and consensus among different stakeholders in support of selected advocacy agendas 2. Objective 2: Strengthened capacity and viability of core civil society partners and of the broader civil society sector 3. Objective 3: Increased engagement of civil society and citizens in policy development, and in government monitoring and oversight at the local, Entity, and State level. Our performance evaluation examines the process of CSSP implementation, documenting the extent to which various CSSP components were implemented as planned and the reason for any deviations from the plans. The central part of the evaluation is an examination of the sector network approach. The evaluation also identifies key facilitators of and barriers to CSSP’s successful implementation. Finally, it, to the extent possible, provides evidence on which specific interventions implemented within CSSP are most likely to deliver results, and how these lessons from the first half of the CSSP intervention can be used to inform adjustments in the remainder of the implementation period and any future donor CSO support programs. 9 2. CSSP BACKGROUND CSSP is a $9.15 million activity financed by USAID to strengthen and maintain the capacity of BIH civil society to influence the creation and implementation of public policies of interest to citizens. USAID/BiH designed this activity to address the following needs: 1. Strengthen Civil Society Organizations’ (CSOs’) legitimacy in order to be true representatives of stakeholders rather than acting independently and without any real connection to the citizens; 2. Strengthen competence and professionalism of CSO experts in their respective sectors in order to earn the respect of the stakeholders and authorities with their own quality of work; and 3. Increase the CSOs’ impact on public sector policies as well as the impact on the CSOs’ working conditions, so they can act as partners with the relevant authorities in the policy processes, especially in oncoming negotiations with European Union (EU). CSSP is a unique intervention in civil society in BiH in several respects: 1. Large scope of intervention in terms of number of beneficiary CSOs (21 are grantees/sub￾grantees) funded continuously throughout the intervention. 2. Operationalization of the USAID FORWARD Initiative in terms of promoting sustainable development through increased investment directly to partner governments and local organizations by: a. CSSP implementation of a mechanism that includes local implementing partners, the Centers for Civil Initiatives (CCIs), and the Center for Promotion of Civil Society (CPCD) managing sub-grants, and b. Promotion of a bottom-up approach in identification of sector priorities through networks, to build a participatory approach and strengthen CSOs’ legitimacy in representing broad public interest. 3. Focusing on sustainability of CSOs, including decreasing dependence on foreign donor funding. The CSSP design was informed by lessons learned from USAID’s precursor civil society interventions— the Citizen Advocacy Partnership Program I (CAPP I) and Citizen Advocacy Partnership Program II (CAPP II), also implemented by CCI. CAPP I (value of $3.82 million) was implemented between August 2005 and August 2008, with the objectives to strengthen public advocacy, perform watch-dog functions, and provide sub-grants to local CSOs to advocate for specific policy changes.3 CAPP II (value of $9.95 million) was implemented between August 2008 and August 2013, with the objective to further development of participatory democracy by: i) supporting the Constitutional reform by advocating EU values and public opinion; ii) strengthening government accountability through campaigns for the adoption of anticorruption policies, active employment measures, and better public finance management; and iii) enhancing active civic participation in decision-making processes and improving the public image of non-governmental institutions (NGOs).4 3 USAID’s Evaluation of Civil Society Programs in Bosnia and Herzegovina from 2008 (available at http://www.ecnl.org/dindocuments/204_Bosnia%20Civil%20Society%20Report%2007-04-08%20%20FINAL.pdf) provides more details. 4 USAID’s Midterm Evaluation of Civil Advocacy Partnership Project (CAP II) from 2011 (available at http://pdf.usaid.gov/pdf_docs/Pdact374.pdf) provides more details. 10 Five main problems were to be addressed by CSSP: 1. Lack of government engagement with civil society 2. Inability of CSOs to influence public discourse 3. Lack of professionalism of CSOs 4. Weak sustainability prospects of the civil society sector 5. Lack of reliable and unbiased information5 The USAID/CSSP activity aims to increase citizen participation in governance—and hence to increase accountability based on a development hypothesis that CSOs’ increased capacity, and supported interaction with a wider set of stakeholders and types of engagement, will provide them with the necessary tools and ability to work with a wider set of actors to address key BiH reform concerns. USAID/BiH defined the following specific development hypothesis for CSSP: If leading CSOs strengthen their capacities and organizational structures to ensure sustainability and financial viability and effectively partner with a wide array of stakeholder groups, then we can expect civil society to be more effective in influencing and overseeing development and implementation of government policy. Furthermore, the hypothesis is that strong and effective civil society that advocates for issues relevant to broader constituencies will lead towards increase civic engagement in decision-making processes. This evaluation takes the development hypothesis as the basis from which to better understand how CSSP interventions contribute to CSOs’ better performance in targeted policy advocacy areas and hence improve CSSP’s ability to strengthen civil society to effect policy change and contribute to increased citizen participation in BiH governance. The illustrative hierarchy of the expected CSSP results articulated by USAID/BiH within the Draft SOW for this evaluation is shown in Figure 1. The CSSP design, as per the CSSP Program Description, is complex and attempted to comprehensively address each of the three objectives through a two-pronged approach: i) through direct tasks of the CSSP 5 See Annex I for more details. Figure 1. Illustrative Hierarchy of Expected Results of CSSP 11 Implementers CCI and CPCD, and ii) through 12 sector networks led by 196 CSO sector leaders/co￾leaders. The 12 sectors are: 1) anti-corruption; 2) employment and labor market; 3) economic policy; 4) education; 5) health care; 6) human rights of marginalized groups; 7) women's rights; 8) agriculture and rural development; 9) culture; 10) public finances; 11) environment protection and energy efficiency; and 12) the justice sector. Figure 2 lists sector leaders/co-leaders for each network. Objective 1 - increased partnership and consensus between different stakeholders in support of selected advocacy agendas - is designed to be achieved by the following six tasks: 1.1 Raising awareness for developing partnerships and networks through stakeholder management training for CSO sector leaders/co-leaders and training on building and strengthening networks for CSO sector leaders/co-leaders 1.2 Awarding small grants to CSO sector leaders/co-leaders to develop their sector networks and to create a list of priorities 1.3 Strengthening CSO relations with media, through media training about CSO’s sector advocacy issues and 'semiannual meeting media/CSOs' and periodic meetings with editors 6 The original number of CSO sector leader/co-leaders was 20; however, public finance sector co-leader Center for Informative Decontamination of Youth (CID) from Banja Luka dropped out of CSSP. In addition, in the justice sector, co-leader Foundation for Local Democracy from Sarajevo was replaced by the Helsinki Committee for Human Rights from Bijeljina. SECTOR CSO SECTOR LEADER CSO SECTOR CO￾LEADER HEALTH PfH (Sarajevo) ICVA (Sarajevo) ANTI-CORRUPTION Transparency International (Banja CIN (Sarajevo) ECONOMIC DEVELOPMENT NBR (Modrica) LINK (Mostar) EDUCATION OKC (Banja Luka) HUMAN RIGHTS - MARGINALIZED GROUPS IBHI (Sarajevo) FSU (Sarajevo) EMPLOYMENT AND LABOR MARKET KULT (Sarajevo) ENVIRONMENT PROTECTION AND ENERGY EFFICIENCY CRP (Tuzla) JUSTICE Citizens' Forum (Tuzla) Helsinki Committee for Human Righths (Bijeljina) AGRICULTURE AND RURAL DEVELOPMENT ARD (Banja Luka, formerly part of PUBLIC FINANCE CPI (Sarajevo) CULTURE AKCIJA (Sarajevo) MEDIA CENTAR (Sarajevo) WOMEN RIGHTS ZENE ZENAMA (Sarajevo) ZENE TRNOVA (East Sarajevo) Figure 2. CSO Sector Leaders/Co-Leaders 12 1.4 Strengthening CSO and private sector partnerships, through annual fairs, targeted consultations, and research to assess perspectives and key obstacles to the readiness of the civic and business sectors for cooperation, including Corporate Social Responsibility 1.5 Strengthening relationships between CSOs and citizens through 'open doors' events 1.6 Strengthening relationships between CSOs and governments, through Regional Dialogue Forums (five forums in both BiH entities discussing the ways and areas to strengthen partnership between CSOs and government), building joint agenda for EU reforms between CSOs and government (creative workshops - expert group - external consultations) adoption and signing of the joint agenda and its implementation, and annual sessions on progress (mapping and promoting good practices/results) Objective 2 - capacity and viability of core civil society partners and of the broader civil society sector - is designed to be achieved by the following two tasks: 2.1 Creating an affirmative work environment, through assessment of the environment affecting CSO sustainability and campaigns to improve CSO working environment (including i) advocating for transparent public financing, ii) creating a CSO Code of Conduct, iii) advocating a change in tax laws to support philanthropy, and iv) development of social entrepreneurship in BiH), strengthening CPCD's Resource Center for CSOs, and public foundations to support CSOs. 2.2 Strengthening capacities of CSO sector leaders/co-leaders, through Organizational Needs Assessment and Organizational Capacity Assessment (OCA), strengthening internal capacities through training on development of Sustainability Strategy and Communications Strategy, and Grants to CSO sector leaders/co-leaders for capacity building to create and implement Sustainability Strategies and Communications Strategies. Objective 3 - increased engagement of civil society and citizens in policy development and government monitoring and oversight at the Local, Entity, and State level - is designed to be achieved by the following three tasks: 3.1 Grants to CSO sector leaders/co-leaders for advocacy campaigns/monitoring/political analysis for resolving the problems identified by sector networks 3.2 Monitoring government performance (including CCIs’ monitoring of work of governments and parliaments and CSO sector leaders/co-leaders monitoring of their sectors) with a special focus towards real problems of citizens and progress in EU reforms 3.3 Citizen engagement (informing the citizens on programming public funds) Figures 3-6 provide detailed overviews of the expected results of these tasks (envisaged by the program design), as well as performance indicators reported by the CSSP for the first two years of the activity for each of the three CSSP objectives and the overall goal. Note that CSSP is currently in the process of revising and refining its Monitoring and Evaluation Plan with USAID/BiH to: group, adjust, and/or add indicators by additional level of results as per USAID/BiH Activity logical framework guidance; improve the names of several existing indicators to more clearly reflect what is measured; ensure that people-level indicators are disaggregated by sex; and to define/revise life-of￾activity targets. 13 1. Percentage of citizens participating in decision making process (37.15% in Yr 1, 36.13% in Yr 2) 2. Number of CSOs receiving USG assistance engaged in advocacy interventions (21 in both Y1 and Y2) 3. Average percentage of surveyed informed citizens who express positively about CSOs ( 28.03% in Yr 1, 28.35 % in Yr 2) 4. Percent change in perception of business, parliamentary and government representatives about CSOs (50.99% in Yr 1, 53.31 in Yr 2) ADDITIONAL CROSS-CUTTING INDICATORS 5. Number of governments responds/actions on CSSP (CCI and grantees outputs) -policy recommendations related with women’s participation, gender balanced language etc. (0 in Yr 1, 0 in Yr 2) 6. Number of CSSP outputs which underline the need and promote gender equity – including reports, analysis, and recommendations (0 in Yr 1, 698 in Yr 2) CSSP GOAL: TO STRENGTHEN AND SUSTAIN THE CAPACITY OF CIVIL SOCIETY TO INFLUENCE AND OVERSEE DEVELOPMENT AND IMPLEMENTATION OF GOVERNMENT POLICY-MAKING PROCESSES MAIN FOCUS AREAS • Strengthening Civil Society Organizations’ (CSO) legitimacy in order to be true representatives of stakeholders rather than acting independently and without any real connection to the citizens; • Strengthening competence and professionalism of CSO experts in their respective sectors in order to earn the respect of the stakeholders and authorities with their own quality of work • Increasing the CSOs’ impact on public sector policies as well as the impact on the CSOs’ working conditions, so that CSOs can act as partners in the policy processes by the authorities, especially in oncoming negotiations with European Union (EU). EXPECTED HIGH-LEVEL CSSP RESULTS ENVISAGED IN CSSP PROGRAM DESCRIPTION • Significantly higher sustainability of 10-12 selected CSOs, which will result in functional partnerships with stakeholders (selected CSOs have partner networks), an improved framework for the work of CSOs (3 public foundations for support of CSOs have been established), and developed sustainability and communication strategies together with diversified funding sources of CSOs (selected CSOs also have local funding sources); • Improved professionalism and accountability of 10-12 CSOs (adopted and applied the Code of Conduct of CSOs + targeted strategies developed), which will be imposed as competent partners to BiH authorities in specific sectors having legitimacy to represent the interests of the citizens (at least 30 policies changed by sector), especially in the context of ongoing EU reforms; and • Greater transparency and dynamics in the decision-making process in BiH (CSOs and stakeholder participation in the budgeting process, for the real problems of citizens), the openness of government institutions (public calls announced by authorities for participation in public debates/policy-making), with the existence of reliable and unbiased information (list of priority problems of citizens and at least 12 policy proposals). CSSP'S REPORTED M&E INDICATORS AND INDICATOR VALUES FOR FIRST TWO YEARS Figure 3. CSSP’s Tasks and Results Envisaged by Program Design, and M&E Indicators Reported by CSSP: Activity Goal 14 TASKS ENVISAGED BY CSSP PROGRAM DESCRIPTION 1.1 Raising awareness for developing partnerships and networks through stakeholder management training for CSO Sector Leaders/Co-Leaders and training on building and strengthening networks for CSO Sector Leaders/Co-Leaders 1.2 Small grants to CSO Sector Leaders/Co-Leaders to develop their sector networks and to create a list of priorities 1.3 Strengthening CSO relations with media through training for media about CSO’s sector advocacy issues and 'semiannual meeting media/CSOs' and periodic meetings with editors 1.4 Strengthening CSO and private sector partnerships through annual fairs, targeted consultations, and research to assess perspectives and key obstacles of the readiness of the civic and business sectors for cooperation, including Corporate Social Responsibility 1.5 Strengthening relationships between CSOs and citizens through 'open doors' events 1.6 Strengthening relationships between CSOs and governments through Regional Dialogue Forums (total of 5 forums in both BiH entities discussing the ways and areas to strengthen partnership between CSO and government), building joint agenda for EU reforms CSO- government (creative workshops - expert group - external consultations), adoption and signing of the joint agenda and its implementation, and annual sessions on progress (mapping and promoting good practices/results) • 10-12 Sector issue-based networks developed and work on specific policy issues; • 10-12 Sector issue-based networks functional and self-sustaining; • Joint agendas created and signed between CSO and BiH local governments regarding the cooperation on the needs of citizens and contents of EU reforms; • Citizens more informed about CSO activities on selected policy issues. 7. Number of stakeholders (other than CSOs) engaged in advocating for sectoral issues (0 in Yr 1, 238 in Yr 2) 8. Number of sectoral priority issues created (11 in Yr 1 and 12 in Yr 2) 9. Number of issue-based advocacy campaigns run by established networks with participation of all stakeholders (0 in Yr 1 and 35 in Yr 2) 10. Number of media articles related with CSSP and CSOs engaged (487 in Yr 1 and 1776 in Yr 2) EXPECTED RESULTS ENVISAGED IN CSSP PROGRAM DESCRIPTION CSSP'S REPORTED M&E INDICATORS AND INDICATOR VALUES FOR FIRST TWO YEARS OBJECTIVE 1: INCREASED PARTNERSHIP AND CONSENSUS BETWEEN DIFFERENT STAKEHOLDERS IN SUPPORT OF SELECTED ADVOCACY AGENDAS Figure 4. CSSP’s Tasks and Results Envisaged by Program Design and M&E Indicators Reported by CSSP: Objective I 15 11. Percentage of international funding within the engaged CSOs (91% in Yr 1, 87.12% in Yr 2) 12. Number of created sustainability and communication strategies of all selected CSOs (0 in Yr 1 and 19 in Yr 2) 13. Local organizational capacity assessment score (2.94 in Yr 1) 14. Index of perceived role of CSSP grantees (CSOs) among sample of non CSSP CSO (21.97 in Yr 1, 21..82 in Yr 2) 15. Number of policy proposals created I processes initiated for allowing local individual and corporate philanthropy (0 in Yr 1, 8 in Yr 2) CSSP'S REPORTED M&E INDICATORS AND INDICATOR VALUES FOR FIRST TWO YEARS OBJECTIVE 2: STRENGTHENED CAPACITY AND VIABILITY OF CORE CIVIL SOCIETY PARTNERS AND OF THE BROADER CIVIL SOCIETY SECTOR TASKS ENVISAGED BY CSSP PROGRAM DESCRIPTION 2.1 Creating an affirmative work environment through assessment of the environment affecting CSO sustainability, campaigns to improve CSO working environment (including i) advocating for transparent public financing, ii) creating a CSO Code of Conduct, iii) advocating a change in tax laws to support philanthropy, and iv) development of social entrepreneurship in BIH), strengthening CPCD's Resource Center for CSOs, and public foundations for supporting CSOs. 2.2 Strengthening capacities of CSO Sector Leaders/Co-Leaders through Organizational Needs Assessment through Organizational Capacity Assessment (OCA), strengthening Internal Capacities through training on development of Sustainability Strategy and Communications Strategy, and Grants to CSO Sector Leaders/Co-Leaders for capacity building to create and implement Sustainability Strategies and Communications Strategies EXPECTED RESULTS ENVISAGED IN CSSP PROGRAM DESCRIPTION • Created and implemented strategies sustainability and communication within the core group of 10-12 selected local partners where all of them reduced for at least 30% of international funding (trough membership fees, private sector, local financing, individual giving etc.), • Proposals for establishing 3 public foundations for supporting CSOs work are created and introduced into formal procedure, • Passed and implemented Code of conduct for CSOs, • At least initiated formal process for passing the Law on social entrepreneurship in BIH as one of the key pillars for CSO sustainability, • Proposal of Code for distributing the public money to CSOs is created and introduced into formal procedure (BiH Parliament, entities’ parliaments), • Created proposals and initiated process of passing the legislations, which allow and support local individual and corporate philanthropy, • Improved organizational capacities of CSOs - measured by OCA tool, • The core CSO group supports civil society as a whole and is perceived as representing a broader array of CSOs, especially those focused on tangible, non￾policy issues; • Increased capacity of a select core CSO group to use new media and information and communications technology (ICT) to enhance advocacy efforts and membership development; • Significant media visibility and influence on public discourse of CSO and USAID/CSSP in advocating for targeted improvements of legislations and increased performance responsibility within CSOs Figure 5. CSSP’s Tasks and Results Envisaged by Program Design and M&E Indicators Reported by CSSP: Objective 2 16 16. Number of public policies introduced, adopted, repealed, changed, or implemented with citizens input (0 in Yr 1, 11 in Yr 2) 17. Number of network members (organizations and individuals) participating in advocacy initiatives (0 in Yr 1, 411 in Yr 2) 18. Number of cases of cooperation between CSOs and think-thanks, media etc. (0 in Yr 1, 59 in Yr 2) 19. Number of CSOs which visibly participate in policy processes (0 in Yr 1, 24 in Yr 2) 20. Number of media articles about the monitoring reports during the project (458 in Yr 1, 533 in Yr 2) OBJECTIVE 3: INCREASED ENGAGEMENT OF CIVIL SOCIETY AND CITIZENS IN POLICY DEVELOPMENT AND GOVERNMENT MONITORING AND OVERSIGHT AT THE LOCAL, ENTITY, AND STATE LEVEL TASKS ENVISAGED BY CSSP PROGRAM DESCRIPTION 3.1 Grants to CSO Sector Leaders/Co-Leaders for advocacy campaigns/monitoring/ political analysis for resolving the problems identified by sector networks 3.2 Monitoring government performance (including CCI's monitoring of work of governments and parliaments and CSO Sector Leaders/Co-Leaders monitoring of their sectors) with a special focus towards real problems of citizens and progress in EU reforms 3.3 Citizen engagement (informing the citizens on programming public funds) EXPECTED RESULTS ENVISAGED IN CSSP PROGRAM DESCRIPTION • CSOs are providing inputs to BIH authorities and actively participate in planning and adoption of measurements which leads towards transparent planning system, • BIH authorities operated more transparently and accountable as an respond to the public demands (articulated through USAID/CSSP and networks) for progress in EU reforms • At least 1 mil BIH citizens informed about key priorities of citizens and EU integration demands as well as about lack of authorities’ actions in this regards (trough monitoring reports) • At least 20 CSOs have been recognized by the public and authorities as key counterparts within the reforms processes in the country. • Strategic partnerships among think-thanks, advocacy groups, watchdog organizations, local organizations and the media that are advocating public policies or taking part in monitoring governmental authorities CSSP'S REPORTED M&E INDICATORS AND INDICATOR VALUES FOR FIRST TWO YEARS Figure 6. CSSP’s Tasks and Results Envisaged by Program Design and M&E Indicators Reported by CSSP: Objective 3 17 CSSP’s implementation is carried out as follows: 1. Tasks led by CSSP Implementers CCI and CPCD, which lead the campaigns for strengthening an enabling CSO environment in BiH, work on strengthening the selected CSOs’ capacities, and conduct comprehensive monitoring of the performance of BiH authorities. 2. Tasks implemented by the 12 sector networks led by 19 CSO sector leaders/co-leaders through three grants for each of the 12 sectors, and administered and managed by CCI. The first grant, in the amount of $50,000, was awarded for establishing sector networks to increase engagement of CSO sector leaders/co-leaders with other key stakeholders (other CSOs, citizens, government, private sector, academia, and media). The second grant, in the amount of $100,000, was awarded for strengthening internal capacities and organizational structures of the 19 CSO sector leaders/co-leaders to ensure sustainability and financial viability when CSSP ends. The third grant, in the amount of $250,000, is for public advocacy campaigns and monitoring—including CSSP’s assistance to these groups to engage in policy oversight and development related to political, social, and economic reforms essential for EU integration, with a broader goal of increasing government accountability. Figure 7 gives an overview of the purposes for which grants were awarded to the CSO sector leaders/co-leaders, based on information submitted to the evaluation team by CCI. USAID/BiH added one more segment to CSSP, as a response to the 2014 floods, which included formation of network BRANA (Dam in English) led by CCI. With the aim of contributing to transparency and accountability in spending public funds, BRANA monitored the flood recovery process and collected and verified data on allocation of flood-recovery funds in 78 municipalities across BiH, from April 2015 to February 2016. The 23 CSOs participating in BRANA (including eight CSSP CSOs) collected and presented data to the public through an interactive online map. In total, data on more than $220 million of funds spent on flood recovery by donors and BiH authorities were collected. Moreover, through BRANA Hotline and organized public discussions, citizens were able to report abuses of funds and/or provide positive examples from their communities. BRANA network is not a subject of this evaluation. • To strengthen the organizational and individual staff capacity in order to increase competences in key areas. • To secure visibility of the campaigns and support (mobilization) of the public to solve the problem that is being advocated. • To contribute to solving 3-4 of the problems identified by sectoral network • To conduct advocacy activities focused primarily on policy changes to address identified sectoral priorities, through campaigns that are based on relevant analyses and research. • To monitor whether and how the identified sectoral problems have been resolved and to carry out activities to inform the public. • To develop Sustainability Strategy of the CSO Sector Leader and Co-Leader, including an implementation plan focusing on four areas 1) Institutional sustainability, 2) Programmatic sustainability 3) Financial viability and 3) External environment. • To develop Communications Strategy with the implementation plan. GRANT 1 (CONTRIBUTING TO CSSP'S OBJECTIVE 1) - ESTABLISHING SECTOR NETWORKS ($50,000): GRANT 2 (CONTRIBUTING TO CSSP'S OBJECTIVE 2) - CAPACITY BUILDING AND SUSTAINABILITY ($100,000): • To secure that by the end of CSSP, 30% of funding for the core activities of the CSO(s) comes from alternative sources (excluding international donors). GRANT 3 (CONTRIBUTING TO CSSP's OBJECTIVE 3) - ADVOCACY CAMPAIGNS AND MONITORING ($250,000): • To implement the measures identified within the Sustainability Strategy and Communications Strategy. • To establish sector networks that bring together representatives of governments, NGOs, academia, media, citizens. • To identify list of problems and priorities in the sector. • To ensure functioning of the network through various mechanisms of communication and periodic meetings of the network. • To periodically examine the relevance of selcted sectoral problems or expand the list of sectoral problems. • To look for support from other sources if possible and to undertake initiatives to work on ther problems from the sectoral list that are not covered by the campaigns financed within CSSP. Figure 7. Overview of Grant Purposes for CSO Sector Leaders/Co-Leaders 18 3. EVALUATION METHODS AND LIMITATIONS 3.1 METHOD DESCRIPTION Evaluation team employed a mixed methods in conducting this performance evaluation. Although performance evaluation cannot assess extent to which results of intervention can be attributable to an intervention, it can provide valuable information on how an intervention is being implemented, how it is perceived and valued, and to what extent the expected results are occurring. The evaluation team drew on analysis of a wide array of quantitative and qualitative data to inform and answer evaluation questions and sub-questions, including: I. Desk research of CSSP activity documentation and other secondary sources II. Semi-structured Key Informant Interviews (KIIs) III. Online survey of the sector network members/participants The data from above-listed sources are triangulated to address the same evaluation questions from multiple perspectives. Comparing and contrasting data help us to gain a more complete understanding of the issue and provide more confidence in the findings. The evaluation matrix is given in Annex II. The desk study included a review of all available activity documentation—including the program design defined within the CSSP implementation agreement, CSSP Annual Reports, CSSP Quarterly Reports, and additional documents submitted to the evaluation team by the CSPP Implementers (including Implementers’ own review of CSSP achievements and overview of purposes for which grants were awarded to each of the CSO sector leaders/co-leaders). The team conducted semi-structured KIIs with relevant direct and indirect CSSP stakeholders (49 institutions/organizations in all). The semi-structured interviews provided inputs into evaluation findings, conclusions, and recommendations. We consolidated the information we obtained and analyzed it by a thematic analysis, using interview transcription coding with NVivo software to identify information categories related to the evaluation research questions. We analyzed the information we obtained from the interviews using a process in which we consolidated multiple responses related to a similar theme by different categories of respondents, and then analyzed them for common themes, exploring the qualitative data inductively and then transforming them into categorical data for further quantitative analysis. This content analysis to uncover themes or categories enabled us to develop a detailed description of the CSSP implementation process and CSSP’s contributions. We developed the online survey with 19 questions to gather additional feedback from the sector network members and participants, given that we were limited in time and resources in conducting KIIs with such a large group of stakeholders. The survey was designed to: i) provide inputs to analyses of CSSP sector networks using the elements of the Social Network Analysis (SNA) approach7 and ii) collect general feedback from CSSP sector network members on other non-network related CSSP activities. Thus, the survey results informed all evaluation questions in this Report. Annex V shows survey questionnaire. DATA COLLECTION As noted, the data for the performance evaluation come from a desk review, semi-structured interviews, and an online survey of the sector network members/participants. 7 See Annex V for more details. 19 Desk review We conducted a desk review of CSSP documentation provided by CSSP implementation team and USAID/BiH. The documents that our team reviewed included:  The CSSP implementation agreement providing expectations and goals of CSSP intervention  Annual Work Plans, including anticipated activities, stakeholders, and resources on annual basis  Annual Reports, containing summaries of activities, resources spent during the year, and results achieved when measured against CSSP’s M&E indicators  Quarterly Reports, providing details on activities undertaken and subsequent results at quarterly basis  Documentation related to the OCA tool, providing details on the OCA design and process  CSSP surveys for M&E indicators, including some data sets and summary descriptive statistics  CSSP Implementers’ own reviews of CSSP achievements, including a summary per objective.  Overview of purposes for which all 36 grants were awarded to each of the CSO sector leaders/co-leader, including a short narrative on planned tasks Other materials we reviewed include evaluation reports of the previous USAID/BiH interventions in civil society, additional reports/documents given to the evaluation team by the KII interviewees, and reports of other civil society interventions in BiH. Annex III provides a detailed list of the documents we reviewed during this evaluation. Key informant semi-structured interviews We conducted extensive KIIs (49 in total) with different CSSP stakeholders: 1) USAID/BiH (2 interviews) 2) CSSP Implementers (referred to as Prime Implementer and Prime Partner in CSSP Program Description) CCI and CPCD (3 interviews) 3) CSSP CSO sector leaders/co-leaders (19) 4) Sector network members/participants (14 interviews) 5) Technical assistance providers (2 interviews) 6) General government officials relevant for CSO legislation, policy environment, and general views of the program (2 interviews, however 4 of the interviewed sector network members from government/public institution were from the institutions relevant for this aspect of CSSP’s work so they also provided information) 7) Other (non-USAID) donors providing assistance to CSOs (2 interviews) 8) CSOs that are not CSSP beneficiaries (4, 1 of which used to be a CSSP CSP sector co-leader) 9) Other donor CSO intervention (1 interview) To select the key informants for the semi-structured interviews for sector network members, we used random sampling. We collected data on network members published on network websites (290 in all) and selected two members from each network (excluding anti-corruption, as the network for this sector is functioning under a non-CSSP USAID/BiH intervention). Out of a total of 22 selected potential interviewees, we were able to hold interviews with 13, plus one additional interview with a member of the BiH Parliament, which cooperated with the CSSP’s anti-corruption sector. We developed separate interview protocols for use with each type of key informant based on elaboration of evaluation research questions, and used them to conduct the semi-structured interviews. The main interview protocols are presented in Annex IV. Annex VI provides detailed list of the key informant interviews. Online Survey of CSSP Sector Network Members/Participants The 19-question online survey was sent out electronically to 824 members/participants of the 12- sector networks, based on the contact lists for each network CCI sent to the evaluation team on our 20 request. In total, 297 responses were received, yielding a 36 percent response rate. Figure 8 shows for each sector the number of members publicly listed on websites, number of members/participants whose contact information was sent to evaluation team by Implementers, and number of survey responses received. 3.2 EVALUATION LIMITATIONS We encountered six major limitations during our evaluation. This section describes them and the ways we mitigate them in our analysis. Limitation 1: Response bias. CSSP implementers provided most of the documentation for desk review was provided. Furthermore, due to the benefits to them resulting from USAID/BiH intervention, the interviewees might have overstated positive effects and underplayed its negative aspects. Mitigation: We mitigated the potential response bias to the extent possible by drawing on multiple sources of information for each evaluation question, carefully designing interview guides (with specific examples required), guaranteeing the interviewees’ confidentiality, and ensuring broad coverage not only of the CSSP stakeholders in the KIIs/online survey but also of the stakeholders that are not CSSP beneficiaries. SECTOR Number of network members listed on websites Number of network members/participants sent to evaluation team for survey Number of network members/participants that responded to survey CULTURE 22 84 29 WOMEN RIGHTS 45 286 40 AGRICULTURE AND RURAL DEVELOPMENT 17 39 40 PUBLIC FINANCE - 20 26 ENVIRONMENT PROTECTION AND ENERGY EFFICIENCY 5 95 50 JUSTICE 20 32 17 HUMAN RIGHTS - MARGINALIZED GROUPS 73 76 64 EMPLOYMENT AND LABOR MARKET 22 36 38 ECONOMIC DEVELOPMENT POLICIES 13 27 20 EDUCATION 40 61 47 HEALTH 33 44 25 ANTI-CORRUPTION - 24 29 TOTAL 290 824 425 (note that total number of survey respondents is 297, but many selected multiple networks) Figure 8. Number of Sector Network Member/Participants and Survey Responses 21 Limitation 2: Selection bias. CSSP contacts with sector network members are based on information provided by the CSSP implementers and CSO sector leaders/co-leaders, which runs the risk that the lists may contain only members/counterparts that have positive CSSP experiences. Moreover, given the large number of CSSP direct beneficiaries and members of CSSP-supported networks, and the numerous components CSSP has built into its proposed outcomes, it was not possible to give comprehensive attention to all the many components under investigation. The evaluation team has design the evaluation to focus on key CSSP activities and a diverse set of actors throughout the country, with the knowledge that it likely will not be able to capture all of the diversity and extent of CSSP efforts and effects. Mitigation: We mitigated selection bias to the extent possible by ensuring broad coverage of the CSSP stakeholders’ KIIs/online survey, by random sampling from a broad list of sector network members for KIIs, and by holding interviews with stakeholders outside the contact list provided by the CSSP Implementers and CSO sector leaders/co-leaders. Limitation 3: Recall bias Given the numerous USAID/BiH interventions in civil society sector (including several consecutive interventions implemented by CSSP Implementers), as well as numerous interventions by other donors, the response KII interviewees and online survey respondents to the CSSP evaluation team are likely to have been somewhat influenced by their experiences with non-CSSP interventions. Mitigation: Within the semi-structured interviews, the evaluation team provided information on specific CSSP activities, and always asked respondents to back up their feedback with CSSP-specific examples. Limitation 4: Small sample bias associated with interviewed sector network members/participants and government institutions We were able to visit only a relatively small sample of sector network members/participants for the KII interviews (around 5 percent of sector network members listed on public websites, which is only 2 percent of the total number of sector network members/participants sent to evaluation team by the CSSP Implementers and CSO sector leaders/co-leaders. We attempted to secure a higher coverage of randomly selected sample of sector network members/participants (around 8 percent of sector network members listed on public websites) for site visits, but we had to rely on their availability. Mitigation: We triangulated information from the KIIs with the information from the online survey, to broaden our coverage of sector network members/participants. Limitation 5: Social desirability bias When asking respondents about any program that carried benefits with it, evaluators must always guard against the possibility that respondents give respondents they think the evaluators want to hear, rather than providing honest responses. Mitigation: To encourage honest responses, we informed all key informants that their responses would be kept confidential, and that they would not be identified in the evaluation report in terms of the specific feedback they provided. Limitation 6: Relatively low response rate to the online survey Out of 824 received contacts for the CSSO sector network members/participants, emails with the survey link were undeliverable to around 60 of them, due to email address errors. In addition, the evaluation team received around 20 emails from survey recipients who indicated that they should not be considered a CSSP sector network member/participant, since they did not participate in CSSP. In total, 297 responses were received, yielding a 36 percent overall response rate. However, out of the 297 respondents who started out filling the survey, around 50 dropped at the very beginning of it, while as many as one-third had dropped by question 4, at the point where specific questions about 22 the work of the network were being asked (e.g., questions on frequency and ways of communication among members). Around 60 percent of respondents completed the entire survey. Thus, survey rate when taking into account only complete survey responses relative to those who received survey drops to 20 percent. This may indicate that the lists of sector network members/participants we received in fact included people who did not identify themselves as a part of the network in question. 4. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 4.1 FINDINGS EVALUATION QUESTION 1 To what extent have interventions under CSSP increased CSO partnership and advocacy agenda consensus building with other stakeholders (government, media, and private sector)? Sub-questions: Which types of assistance have been most effective in achieving these partnerships with each type of stakeholders? Which have been less effective? Finding 1: Stakeholder partnership includes a broad set of actors. CSSP network formation8 was designed to bring together a wide variety of actors beyond CSOs, to foster a bottom-up stakeholder approach that would encourage partnerships and cooperation in addressing sector-specific issues. Whether large or small, informal or formal, new or old, the CSO sector leaders/co-leaders were tasked with spreading their sights beyond CSO actors and traditional constituency actors,9 to include a wider set of different types of actors and stakeholders, including government, media, and the private sector. Evidence from the KIIs and survey results suggests that the CSSP network process encouraged partners to cast a wide net in network formation. Based on the lists of the network members/participants received by the evaluation team for sending out the online survey, each of the networks has included government/public institution actors (whether as ‘friends of the network’10 or more formally), roughly two-thirds have included private sector actors, and almost half have included media actors (as shown in Figure 9). 8 Partnership design and formation was left to the sector leaders/co-leaders with the priority on building up a functioning manner of cooperation and partnership, rather than adhering to any particular network structure and level of formality. However, for the purpose of reporting, the term ‘network’ will be used. 9 Sector specificities mean different CSO and constituency focuses, however the broader outreach to citizens and interest groups is mainly achieved through organizations/associations that act as intermediaries between the specific interest groups and CSSP networks. For example, the economic development and employment and labor market networks gather associations of employers, guilds, and chambers of commerce that themselves represent employers and the business sector. Health care gathers diverse smaller and geographically scattered NGOs with specific interests—such as organizations of war veterans; diabetics, HIV, and PTSP sufferers; cancer patients, and disabled persons. Some networks are more expert oriented, like the environment protection and energy efficiency network, which gathers primarily individual experts on the environment and energy from public and private sectors. Other networks that draw a portion of their constituencies from service provision—such as women’s rights, which provides psychosocial counseling. 10 CSO Sector leaders describe the practice of engaging with public institution members informally rather than getting formal approval of representatives through their various ministries and public institutions. 23 Diversity of membership is also present among those actively engaged in network work. The online survey response rates provide a glimpse of active and engaged network members: 55 percent of 290 total survey respondents are members of an NGO that gathers citizens (e.g., citizen association); 7 percent are members of an NGO that gathers private sector representatives; 5 percent are representatives of a private sector company; 18 percent are representatives of government/parliament/public sector institution; 5 percent are media; and 15 percent are individuals. This broad membership was connected through different formal mechanisms of communication established within CSSP. This include face to face meetings and websites present for all CSSP networks, in addition to social media accounts, web platforms for internal network communications, and/or publication of network newsletters/bulletins for most of the networks. Finding 2: CSSP networking efforts have fostered interests in continued cooperation among the current sector members/participants. A majority of CSSP stakeholders are interested in continuing network-related cooperation and partnership. Survey results (Figure 10) show that in each network, at least 80 percent of respondents express medium or high general satisfaction with the work of their network. This level of satisfaction may imply that CSSP contributed to a broader set of stakeholders that are able and willing to continue to work together. Figure 9. Composition of Sector Network Members/Participants SECTOR NGO gathering citizens Private sector (companies and NGOs gathering private sector) Government/ public institutions Media Individual International organizations and donors CULTURE X X X X X X WOMEN'S RIGHTS X X X X X AGRICULTURE AND RURAL DEVELOPMENT X X PUBLIC FINANCE X X X ENVIRONMENT PROTECTION AND ENERGY EFFICIENCY X X X X X X JUSTICE X X X X X HUMAN RIGHTS - MARGINALIZED GROUPS X X X X EMPLOYMENT AND LABOR MARKET X X X X ECONOMIC DEVELOPMENT POLICIES X X X EDUCATION X X X X X HEALTH X X ANTI-CURRUPTION X X X TOTAL 10 9 12 6 8 3 24 Also according to the online survey results, network members/participants see their membership as being beneficial in closer/increased cooperation with other network members (84 percent selected this benefit), improved sector-specific knowledge (83 percent), learning about effective network strategies in addressing policy issues (73 percent), better access to policy makers (67 percent), and media exposure of the network (50 percent). Furthermore, almost all (92 percent) of members/participants who responded to the survey indicated plans to continue cooperating with other network members. According to the most frequent reason cited for this within the online survey comments (40 percent) as well as in the KIIs (all interviewed CSOs), there is wide recognition of the strength in working together to deal with complex problems. Indicative success factors of further partnership probabilities include members’/participants’ perception of sector leader’s capabilities, network vibrancy, and success to date in achieving campaign results. Based on KII and survey results and comments, illustrative examples of such networks are in the sectors of culture, energy efficiency, women’s rights, and economic development and policies. However, the KIIs also point out the limits of such partnership efforts. Network partners expressed concerns about keeping formal network processes in place, given what they perceived as insufficient financial support for network functioning (both at the levels present under CSSP and beyond). Three sector leaders explicitly described current difficulties to cover their obligations to work with and support the activities of network members under CSSP. One sector member specifically noted its current participation in network is limited due to financial capacity; and even positive comments given within the online survey often framed further cooperation in relation to specific donor projects and funding. Moreover, all interviewed government/public institution representatives stressed the need for tangible results of cooperation with the CSOs, and within the networks, in terms of preparing concrete and actionable proposals that can feed into reform efforts (further examined in Q3). “We have good experience in working with the sector leader and co-leader and we trust them. We will continue to work with them beyond the timeline of this project“ - Sector network member/participant Figure 10. Sector Network Members/Participants’ General Satisfaction with the Work of Networks 25 Finding 3: While there is evidence of broad network formation efforts, the extent to which CSSP networks represent new levels of partnership is unclear, as are the breadth of stakeholder membership and depth of their involvement. Network breadth across stakeholders should also be considered in relation to inclusion of all key BiH CSOs active in the specific sector. Here the evidence is more mixed. Based on interviews with non￾CSSP CSOs, in at least a couple of cases sector leaders did not include key sector CSO competitors in their network efforts. However, there were also cases (e.g., women’s rights) where competing CSOs were invited to join the network’s activities. There are no comparable baseline data on pre-CSSP cooperation amongst these actors.11 Key informant interviews suggest that all sector leaders reached out to known stakeholders from previous cooperation or work. In some cases (three networks), CSO sector leaders specifically described extensive work devoted to reaching out to a significantly broader set of actors within CSSP in comparison to stakeholders they worked with prior to CSPP. For example, the network on economic development policies expanded to include government officials, the network on women’ rights broaden to include private sector, the justice network specifically targeted judicial professional associations, while the energy efficiency network reached out to include journalists as active network members. Yet KIIs network member/participant lists suggest that members/participants from the government/public institutions, business, or media are often those known to the sector leaders from before, and not necessarily the most relevant representatives of sectors’ main government institutions. Only three networks spoke more broadly of their outreach to the private sector, and only four spoke of specific approaches/strategies for involving a broader media sector in network activities as opposed to only providing media coverage of network events. In addition to unclear breadth of stakeholder membership in the networks, the depth of involvement among different stakeholders in the work of networks is unclear. To some extent, this is to be expected given thematic area specificities, but three commonalities stand out. First, only three out of eight interviewed government/public sector counterparts identified themselves as active members/participants of their network. All mentioned being invited by lead organizations to events or having participated in events; however, only one of the government counterparts reported engaging regularly with their network. Second, the appropriateness of network members/participants from government/public institutions varies greatly among the networks selected, based on the composition of network members/participants. Third, KIIs and survey findings both suggest that media involvement in the networks was primarily to cover network event rather than substantive work on the issues as network members. In survey responses, only one third of respondents said that the media always or frequently participate in the work of network and substantive development of sector priorities, while in KIIs, two-thirds spoke of media engagement in relation to coverage rather than substantive involvement. While increase in media awareness and coverage of the networks and the issues they cover (as demonstrated in media hit data in CSSP reports) is clearly a positive development, substantive engagement of media representatives in the actual work of networks is insufficient, with a couple of exceptions, most notably in the environment protection and energy efficiency network. Finding 4: Levels of broader stakeholder participation in networks’ thematic development and campaign implementation vary considerably, with more evidence of participation in issue identification than in the campaign design and implementation. CSSP networks have as almost as many profiles as issues areas, but several types can be highlighted: i) networks that focus on being an advisory/assistance body for their members/participants (such as human rights network), ii) networks that focus on being an facilitating body for sector 11 According to CSSP reports, six out of eleven CSSP networks (excluding anti-corruption network, as CSSSP is envisaged to use the ACCOUNT network financed through a different USAID/BiH intervention) were described as new: 1) women’s rights, 2) human rights, 3) justice, 4) employment and labor market, 5) education, and 6) culture. However, according to the KIIs, sector leaders in only three sectors identify their network as previously existing: economic development policies, agriculture and rural development, and environment protection and energy efficiency. This was subsequently confirmed by the CSSP implementers. 26 experts/professionals (such as justice network), and iii) networks that bring together broader constituencies in specific issue areas (such as health network). The multiple types of networks and their roles likely play some role in stakeholder participation; however overall there is more evidence of engagement on idea identification than further design and implementation. Sector leaders reached out universally in their efforts to include network members/participants into issue identification and prioritization processes, and survey data suggest that a majority of members identified themselves as involved in this process. In eight of the 12 networks, over 60 percent of members/participants who responded to this online survey question identified themselves as involved in identifying sector priorities, although around one-third of survey respondents skipped this question). All networks provide some level of description of how they arrived at decisions about the issues the particular network addresses. As shown in figure 11, this included a variety of methods —from using research and surveys (10) to holding a series of meetings (all 12) to holding focus groups (2). In a few cases (particularly the new[er] networks), the process of identifying sector problems was reported as extensive with multiple methods used (e.g., the culture sector conducted both KIIs and a survey). For established networks or issue areas, the process was less extensive—focusing on a review of previously identified problems (e.g., in the sector for economic development policies, a short survey was conducted based on priorities defined in earlier period). Based on the KIIs, all sector leaders/co-leaders acknowledged that they had worked on most of the identified sector issues in the past and had in most cases framed the collection of data on network members/participants’ priorities around earlier experience and findings. Although there were consequently almost no new thematic areas of focus to come out of this process, sector leaders/co￾leaders posited that the consultation process itself was useful for better understanding different Figure 11. Networks’ Methods for Identifying Sector Priorities SECTOR Meetings Survey/ research Focus groups Pre-existing list of priorities CULTURE X X WOMEN'S RIGHTS X X AGRICULTURE AND RURAL DEVELOPMENT X X PUBLIC FINANCE X X X ENVIRONMENT PROTECTION AND ENERGY EFFICIENCY X X X X JUSTICE X X HUMAN RIGHTS - MARGINALIZED GROUPS X X X EMPLOYMENT AND LABOR MARKET X X ECONOMIC DEVELOPMENT POLICIES X X X EDUCATION X HEALTH X ANTI-CURRUPTION X TOTAL 12 9 2 3 27 aspects of the issues, both from the broader stakeholders’ perspective and for gathering richer issue detail (e.g., in health and in anti-corruption networks). The actual level of network membership participation beyond issue identification is more varied. The online survey results (Figure 12) suggest that over half the members/participants were involved in different levels of strategizing, planning, and helping to implement campaigns (although it should be noted that around one-third of survey respondents skipped this question). However, the KII responses suggest that such engagement needs to be further contextualized. For two-thirds of networks, sector leaders described small working groups or leadership bodies of the network as doing most of the work. While all sector leaders mentioned use of a wider set of members participating at meetings and in petitions, input on actual campaign design, planning, and implementation is less clear. Most of the KIIs with network members/participants (8 out of 10) suggest that members’ interest in participating more actively may be higher than current utilization of members by the sector leaders/co-leaders. Online survey results also suggest there is space for further engagement of members (in each of the 12 networks, at least 30 percent of respondents said they believe that opportunity for engagement by all members is provided only sometimes, rarely, or never). Finding 5: Strategic inter-sector thematic partnerships have not been developed, partly as a result of implementers’ primary focus on administrative and operational guidance, as opposed to strategic guidance. Partnership efforts have not focused on cooperation between thematic sectors. The KIIs show little knowledge of efforts of other CSSP networks and CSOs. Only one sector leader stated that the Figure 12. Sector Network Members/Participants’ Involvement 28 network is well informed of other sectors’ work. Even when there is some level of awareness of others’ activities, there has been no specific cooperation among networks working on similar and complementary issues.12 Two particularly illustrative examples of the potential for cross-network collaboration are between the health and human rights networks and among the networks on economic development policies, employment and labor market, and education. The health sector is working on assisting all citizens to get equal access to health care. The human rights sector campaigns are focused on socially marginalized groups of citizens, many of which are also those most often affected by uneven health coverage. Similarly, networks on economic development policies, employment and labor market, and education all advocate for measures on increasing employment: the sector for economic development policies through improvement of regulatory environment for the private sector, the sector for employment and labor market through stimulation of the private sector to employ additional workers, and the education sector for providing practical trainings to students. CSSP implementers self-describe their management style with the networks as not trying to be involved in the substantive aspects in order to allow for ownership and creativity by the sub-grantees. However, such hands-off management style appears to have resulted in little or no coordination amongst networks. CSSP implementers tended to give sub-grantees a wide space to design and implement their network efforts. This allowed sub-grantees to develop unique network processes for engagement and information sharing based on their own judgment. But it also meant that CSSP missed opportunities to facilitate internetwork cooperation. The CSSP coordination meetings13 did not ensure sufficient space for meaningful strategic coordination or collaboration among networks. While envisioned to be a space for facilitating cooperation, annual meetings are described by sub-grantees as primarily a presentation forum or a ‘show and tell’ of what had been achieved rather than meaningful opportunities to consider and plan cooperation options or to share lessons and strategies amongst stakeholders. Individual comments of survey respondents when asked about CSSP’s aspects that need improvement also highlight the need for better inter-sector cooperation (this was one of the most frequently mentioned comments). Finding 6: Defined thematic areas vary in breadth and are not conducive to building up CSO sector expertise, as in some cases they result in narrow and overlapping foci. According to the KIIs, the CSSP network leader selection process was systematic and is perceived as such by sub-grantees. However, the definition of thematic sectors, combined with the specific sector issues identified as priorities by the selected networks, resulted in a set of sectors of varying width and with a number of overlaps. 12 One particularly striking example is in the issue area of youth job market readiness. A member of the network on economic development policies working on issue of encouraging practical training with business associations and the public institutions had not heard of the education network’s efforts to focus attention on more practical training in school curriculum. 13 CSSP Implementers note that five coordination meetings during the course of the grant and two multi-day trainings held at the beginning of the grants. “I do not know how it is possible that we are half-way through a four-year project and we had only two joint meetings with other sector leaders/co-leaders. The fact that some of us communicate via e-mail and Facebook is great, but that is not formal communication within the project. Meeting and collaborations should be more frequent and all CSSP CSOs should be able to nominate agenda topics. We should try to figure out what experiences are interesting to other networks and CSOs and what obstacles we can work on jointly. Communication is the key. “ - CSO sub-grantee “I was thrilled with the sector leader application process, as our impression is that the selection process was entirely based on application quality, as opposed to being decided in advance.“ - CSO sub-grantee 29 Defined thematic sectors bring together a mix of broad issues, such as economic policies, as well as quite specific reform processes, such as energy efficiency. This attracts both more generalist and more expertise-focused CSOs. CSOs’ issue selection within the sector reflects this mix—resulting in some cases in topic areas that constitute only a narrow part of traditionally understood sector issues. For example, a portion of the priorities in the human rights sector is focused on socially marginalized populations, whereas priorities in the education sector are all focused on the alignment of education with labor market needs. Finding 7: Donor decisions have led to some mixed signals and/or duplications among the CSOs, sectors, and/or funded initiatives. CSSP is one in a series of USAID and other donor efforts in BiH to promote the work of civil society and the thematic areas more specifically. Given the broad themes covered under CSSP, the relatively large number of CSO sub-grantees, and the stated intentions of building sector-specific expertise and legitimacy among CSO sector leaders, efforts in coordinating donor interventions appear to be particularly salient. Almost all KII stakeholders (including 15 CSO sub-grantees and all government/public institutions) believe that donor coordination is insufficient. Several themes were highlighted specifically in relation to USAID funding. Parallel funding of multiple CSO interventions (even in the same sector, for example in human rights) is perceived as: i) undermining the objective of building CSO sector expertise, ii) potentially duplicative, and iii) possibly feeding non-transparent practices within civil society. Furthermore, general lack of coordination among USAID initiatives in the same sector (e.g., agriculture) is noted as problematic. More broadly, coordination among different donors that support the CSO sector is sparse. Almost all the government/public KIIs noted their frustration with the lack of coordination among major donors for what are in practice very similar initiatives. Finding 8: Fragmented sub-grant schemes discourage comprehensive network planning. Three sub-grant phases with three different proposals and focus areas were designed to give the sub￾grantees time to focus attention on network formation before moving on to capacity development and campaign issues. Over half of the grantees (10) suggest that in practice this phasing was less helpful than called for in its intended design. Delays in grant disbursement at the beginning left only a short time interval between grants, which minimized the potential advantage of phasing the grants to ensure gradual network building. This fragmented sub-grant scheme and their separate management by CSSP also resulted in the perception that the three separate phases had separate budgets with little cross-use potential. The lowest grant amount (sub-grants of $50,000) was awarded for networks; and sector leaders’ feedback suggests that many have spent much of this on their efforts at network formation through major meetings, website development, and so on. Sector leaders of two networks explicitly noted that they do not have sufficient resources for bringing the network members/participants together in the next two years, while several more also expressed concerns over limited resources. These two and eight additional sectors specifically noted that they mitigate limited resources for network functioning by using activities under other projects to have network meetings. Based on such information from the KIIs, this suggests there is little synergy between the sub-grant and activities on establishing networks with those on advocacy campaigns and monitoring. The extent to which network consultation processes are built into the campaign activities and sub-grant is not clear, but there appears some confusion on how to use campaign grant funds for further engaging with network members. Only one network (justice) provided a description of integrating membership in the campaign design, whereby members are actually formally engaged as experts within the three campaign working groups. EVALUATION QUESTION 2 To what extent have interventions under CSSP strengthened the capacity and viability of selected CSOs in the 12 sectors? Sub questions: How was the assistance (particularly grant mechanisms) 30 to the CSOs in the 12 sectors designed and implemented? Which types of interventions have been most and least effective in meeting the needs of the CSOs? Finding 9: CSSP’s assistance in supporting sustainability of the CSO sub-grantees and application of Organizational Capacity Assessment (OCA) tool is perceived by all CSSP CSOs as innovative in BiH context and useful.14 OCA is USAID’s participatory self-assessment tool is designed to enable organizations to define a capacity-building improvement plan, based on needs identified through a set of criteria examined by staff in a multi-level and multi-department approach.15 Within the CSSP, OCA has been carried out twice so far, in 2014 (baseline) and 2016 (midterm). The initial OCA was to identify and design appropriate technical assistance and financial support. The midterm OCA was to identify the extent of implementation of CSOs’ capacity-building plans and its effectiveness. Annex VII shows the 10 areas and 48 indicators used in CSSP OCA. Given the environment in which CSOs operate, often juggling among different projects and chasing funding, all CSSP CSOs appreciated the process that allowed them to devote specific time to internal organizational analyses and strategic planning. Five of them specifically noted that the structured OCA process was the first time they applied a holistic approach to their organizational building. Almost all of the 21 CSSP CSOs in explicitly praised the CSSP’s OCA mentors, and testified to having improved and/or introduced internal organizational procedures as a result of the OCA process. Finding 10: Improvement measures identified on the basis of the Organizational Capacity Assessment (OCA) process and subsequently included in the Sustainability Strategies and Communications Strategies of the CSSP CSOs are being implemented and most CSOs note that there have been organizational improvements resulting from CSSP’s assistance. Although OCA scores were generally high even in the first OCA round (2.94 out of a maximum of 4 for the 21 CSSP CSOs on average), based on information given to the evaluation team by KIIs, CSOs generally showed a high level of self-criticism; and OCA mentors also confirmed the organizations’ self-critical stance. Figure 13 shows OCA 2014 and 2016 scores. The average score increased by 10 percent, from 2.9416 (out of 4) in 2014 to 3.26 in 2016. All but one CSO recorded an improvement (noting that IBHI, whose score remained constant, already had an initial high score). 17 14 It should be noted that the scope of this evaluation did not include evaluating the OCA tool and its application within the CSSP. Thus, the evaluation lays out the results of the OCA self-assessment as reported within the CSSP documentation. 15 More information on OCA is available at https://usaidlearninglab.org/library/organizational-capacity-assessment-tool 16 Note that ARD (former part of the Agency for Cooperation, Education and Development (ACED) named CARD) did not participate in OCA 2016, as it is no longer a part of ACED, which makes the OCA incomparable between 2014 and 2016 (ACED was assessed in 2014). 17 Data is from the CSSP September 2013-September 2014 Annual Report (November 2014) and Draft Midterm OCA Report (May 2016). “OCA is a very nice tool and I believe that it was useful for all CSOs that devoted time and efforts to self-assess realistically, regardless of the size and experience of the organization.” “Focus on sustainability was one of the main reasons we applied for CSSP in the first place, as we were aware that there were many organizational procedures and strategic directions that we partially had in our heads, but that needed to be holistically thought through and written down. The process was fun and useful.” “I have been in civil society sector in BiH since 1996, and CSSP is the first donor intervention that deals with sustainability. It also coincides with the period of time when CSOs are in dire need of focus on sustainability, as donors are gradually withdrawing. This is why CSSP CSOs take the work on OCA and sustainability seriously.” - CSSP CSOs 31 Six sub-grantees noted that the timing of the OCA process coincided with the organization’s recognition of the need for such a holistic and strategic approach; and these organizations seem to have advanced somewhat more in the implementation of identified measures than others, based on their 2014 and 2016 scores. In a couple of cases, the OCA process helped the organizations to focus their area of work on the sector in which they have comparative advantage (CRP for example). Finding 11: Most of the key tasks articulated within the applications for CSSP’s capacity building and sustainability grant are common across sub-grantees and focus on internal organizational procedures and operational capacities, however, within this mix, M&E capacities of CSSP CSOs are sub-optimal. According to the Draft Report of Midterm OCA from May 2016, the OCA components that improved the most were Strategic Management (17 percent improvement from 2014 to 2016), Media and PR (14 percent), and Governance and Structure (13 percent). This directly reflects the fact that all CSO sub-grantees included measures for improvement in these areas in their Sustainability Strategies, for which funding was awarded through this CSSP sub-grant.18 Annex VIII shows key tasks for which $100,000 grants were awarded to the 19 CSO sub-grantees within the objective of strengthening their capacity and sustainability. 18 It should be noted that the sub-grant descriptions available to the team imply generality in the presentation of funded tasks, which precludes having a more detailed understanding of how the grants were used. CSSP CSO OCA 2014 OCA 2016 AKCIJA (Sarajevo) 2.25 2.79 ARD (Banja Luka, formerly part of ACED) 2.49 Citizens' Forum (Tuzla) 2.54 2.90 CRP (Tuzla) 2.63 2.90 ZENE TRNOVA (East Sarajevo) 2.69 3.13 ZENE ZENAMA (Sarajevo) 2.75 3.32 LINK (Mostar) 2.76 3.34 CCI (Tuzla) 2.93 3.08 CPI (Sarajevo) 2.97 3.08 PhP (Sarajevo) 2.97 3.37 CPCD (Sarajevo) 2.98 3.33 CIN (Sarajevo) 3.03 3.23 NBR (Modrica) 3.04 3.36 OKC (Banja Luka) 3.10 3.23 FSU (Sarajevo) 3.12 3.37 ICVA (Sarajevo) 3.13 3.37 Helsinki Committee for Human Righths (Bijeljina) 3.16 3.51 TI (Sarajevo) 3.30 3.42 KULT (Sarajevo) 3.35 3.59 IBHI (Sarajevo) 3.53 3.53 Figure 13. CSSP CSOs’ OCA 2014 and 2016 Scores 32 CSSP provided some M&E training to the CSO sub-grantees; however, based on available CSSP M&E reports and feedback from some interviews, which imply possible inconsistencies in M&E reporting among CSOs, there is space for further improvement in overall M&E capacities of all CSSP CSOs, both from the perspective of internal CSO M&E practices for their project planning and implementation, as well as from the perspective of capacity to monitor and evaluate public policies. Around two- thirds of the CSO sub-grantees noted poor information and/or clumsy communication on overall CSSP expectations and reporting, with some perceived pressure to report on achieved results as attributable to CSSP mentioned by several CSOs to the evaluation team. Finding 12: Sub-grantees appreciate CSSP’s focus on sustainability and the freedom and flexibility in identification of priorities, in tasks that can be financed by sub-grants, and in ability for adjustments during implementation. They view this freedom and flexibility as contributing to building their leadership capacities. However, at the same time, the negative side of such a flexible approach is insufficient focus given to building CSO sub￾grantees’ sector-specific expertise in most cases and lack of understanding of how this segment of assistance to networks should be connected to the other two segments (establishing networks and advocacy campaigns and monitoring). All sub-grantees noted that they and their networks selected their sector priorities freely and independently, which contributed to building the sector leaders/co-leaders’ leadership capacities and improved the perceptions of their legitimacy as representatives of a broader set of stakeholders involved in their networks. With the focus within CSOs’ Sustainability Strategies being on strategic management, organizational structure, and visibility, not enough attention has been given to capacity building in sector-specific content expertise. Only about half (nine) of the CSO sub-grantees mentioned any tasks related to building these capacities within the purpose of their grant applications (see Annex VIII). Similarly, only a couple of the KIIs articulated specific tasks taken to improve sector-specific expertise to the evaluation team. Sector network members/participants’ survey responses generally show that high shares of network memberships believe that chosen sector priorities are adequate and that campaigns are actionable and well-focused (Figure 14). But there are some variations among networks (for example, 30 percent to 40 percent of network members/participants do not think campaigns are actionable and focused in sectors of women’s rights, employment and labor market, education, and public finance); and the overall share of those with the strongest support for the selected priorities is low (below 20 percent on average) across networks. “There is a great focus on reporting to be visible and to show that we changed policies, although in our initial CSSP application we never committed ourselves to change policies in the short-term. We all know that it is not easy to change a policy in a few months and that not every campaign will change laws and policies and result in new strategies. There are many things that play a role in order for such change to take place, especially in BiH. This perhaps results in some organizations gathering information from their activities under other projects or from other activities in their sector to report.” “Not enough attention is given to capacity building in the areas of monitoring, evaluation, and reporting. The quality of CSSP reports needs to improve.” - CSSP CSOs “We never felt like any specific advocacy priorities for our network were imposed or pressured in CSSP. This is very important, because we had a negative experience with some other projects. In CSSP, we are allowed to adjust to what is going on the ground, rather than doing something, which is not a priority anymore. As such, CSSP is a positive exception among donor projects.” “Originally, development of the civil society sector in BiH was based on socio-psychological assistance, humanitarian aid, and reconciliation. Only recently are the CSOs turning to providing expert services, advocacy, and other activities that require expertise and skills. Thus, the capacity building in these areas is essential and it is great that CSSP provides funding for it. However, CSOs have not focused enough on this so far.” - CSSP CSOs 33 This may imply that in some cases CSO sub￾grantees need further capacity built, in order to base their advocacy work on evidence-based analyses in their sector—an implication that is generally confirmed in the KII interviews with the government/public institutions. Based on the KIIs and reviewed CSSP documentation, the capacity building and sustainability component of CSSP assistance is understood as mostly unconnected to the components of establishing the networks and public advocacy campaigns and monitoring. More broadly, 10 CSSP CSOs expressed during the interviews that the implementation was unnecessarily fragmented and insufficiently inter￾connected among the three segments of assistance to sub-grantees, as well as between the “Analytical and sector-specific capacities are generally missing in CSOs. We and other government institutions on numerous occasions tried to include CSO representatives in some strategic working groups, for example; however, it is almost impossible to find CSOs that have capabilities to contribute and are interested in participating. They generally go after donor projects and avoid working with the government unless they are directly paid by the government. When they do get involved through donor projects, research is missing, and they generally attack everything prepared or done by the government by default, without evidence or credible arguments. Thus, any attempts by the government to include CSOs in decision-making process unfortunately usually ends up being counter-productive and thus provides additional excuses not to include them. - Government/public institution representative 0% 20% 40% 60% 80% 100% Women's rights Agriculture and rural development Ecnomic development policies Publlic finance Employment and labor market Justice Health Human rights Education Environment protection and energy efficiency Culture Anti-corruption Chosen priority advocacy campaigns reflect well real priorities in this sector Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 0% 20% 40% 60% 80% 100% Environment protection and energy… Employment and labor market Women's rights Education Publlic finance Human rights Anti-corruption Health Agriculture and rural development Justice Ecnomic development policies Culture Chosen priority advocacy campaigns are actionable and well￾focused Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Figure 14. Sector Network Members/Participants’ Feedback on Adequacy of Priorities 34 two CSSP implementers, whose role is perceived as mostly administrative as opposed to strategic and advisory, as well as insufficiently integrated between CCI-managed tasks and CPCD-managed tasks. The process that included an overall sector leader/co-leader application and three separate sub-grant mechanisms meant that intense efforts needed to be invested in the preparation and administration of the grants, which took away from focus on technical capacity and expertise building. At the same time the potential advantage of phasing the grants to ensure gradual building could not be achieved in any case, since the time interval between the grant approvals was in most cases only three months. Furthermore, USAID’s requirement of monthly reporting is perceived as over-burdensome and unnecessary by several CSO sub-grants, thus also taking away from technical capacity building. Finding 13: Collaboration among the CSSP CSOs (in both content related to sectors and internal operational procedures) has not been used as a CSO capacity-building mechanism in CSSP. Coordination among networks is lacking when it comes to strategic cross-sector planning and implementation. Almost all interviewed KIIs believe there is not enough cross-sector content coordination among the networks. They perceive network coordination meetings held by CSSP implementers as focused on reporting on past tasks, as opposed to collaborative strategic planning of future tasks. Moreover, they recognize there are joint content areas among networks that could strengthen the work of each network and the CSO sector leader/co-leaders. In addition to scarce content-specific cross￾sector coordination, there is no cooperation and peer-learning among the CSSP CSOs on common operational capacities needed by all CSOs. Finding 14: All the CSSP CSOs identified some targeted alternative ways of funding (other than foreign donations) within their Sustainability Strategies prepared under CSSP. As shown in Figure 15, all CSOs identified at least one possible type of alternative funding, with three alternative funding sources on average being identified. All organizations selected user services/contracted research, with the second most frequently selected type being provision of services to BiH authorities. Half of the organizations are considering registering, or have already registered, a company, which will be used for transferring profit into the CSO. “I think that we have not utilized enough opportunity to collaborate with each other. Partially it was due to lack of time and partially perhaps due to lack of coordination by CSSP implementers. I think that the mistake was made at the very beginning, as there was no collaboration and coordination during the stages when each network defined sector problems. We only later found out what other networks are working on. There is definitely space for improvement. If we could present to each other plans for future activities and concrete problems we face, we could surely define ways to work together for more effective results. - CSO sub-grantee “The three segments of assistance are good, but they should form one comprehensive activity. Implementation is fragmented with each segment being treated as independent from the others, especially in terms of the sustainability component. Thus, it ended up like three separate projects, including the reporting and different staff who manages it from the CSSP Implementers without sufficient coordination. At first, we were told that branding of the network is most important, and then the focus was switched to building visibility of the CSOs who lead the networks. This and several other switches of focus during implementation makes planning difficult for us.” - CSO sub-grantee 35 Finding 15: Based on KIIs, around half the CSSP CSOs secured alternative funding sources in the last two years, or are about to do so. Examples include securing service contracts with BiH authorities (five CSOs) and through registered commercial companies (four CSOs). Several organizations noted that the alternative funding sources would not have been secured had it not been for the Sustainability Strategy development process within CSSP. Finding 16: Assistance provided in identifying alternative funding is perceived as not being tailored enough to CSO type. Several CSOs perceive that CSSP implementers’ preferred option is registering an enterprise. The feedback on CSSP’s assistance in developing Sustainability Strategies and in designing and implementing measures to decrease dependence on foreign donor funding is mixed. The CSSP implementers explain that they did not favor any of the alternative funding options nor put forward any preferred options to the sub-grantees in documents and during meetings and that the sub-grantees themselves defined their forms of funding diversification. Nevertheless, the perception of around half “The result of the process of working on Sustainability Strategy is that we are already one step away from solving our sustainability problem. We are in the final process of negotiating a contract with the Government under which we will provide services for them on an annual basis.” - CSO sub-grantee “I do not think that sustainability of civil society is for all of us to register enterprises. The purpose of civil society would be lost in that case.” “Our mission and vision is primarily humanitarian. Doing business defies the purpose of civic activism entirely.” “As an advocacy CSO, registering a company is against our principles. - CSO sub-grantee CSO SECTOR LEADER/CO￾LEADER Membership fees Services (User services and contracted research) Funding from BiH authorities: Grants Funding from BiH authorities: Services Crowdfunding/ philantrophy Registering a company TOTAL CURRENTLY MEETING/CONFIDENT ABOUT MEETING THE 30% TARGET X X X X X 5 X X X X X 4 X X X X 4 X X X 2 X X X X 3 X X X 3 X X X X X 4 X 1 X X 2 X X 2 X X X 3 X X X 2 X X X X 3 X X X X 3 X X X X 3 X X 1 X 1 X X 2 X X X 3 X X X 2 X TOTAL 4 20 5 12 3 9 11 KULT (Sarajevo) AKCIJA (Sarajevo) ZENE ZENAMA (Sarajevo) Zene Trnova (East Sarajevo) ARD (Banja Luka, formerly part of ACED) CPI (Sarajevo) CRP (Tuzla) Citizens' Forum (Tuzla) Helsinki Committee for Human Righths (Bijeljina) FSU (Sarajevo) IBHI (Sarajevo) CIN (Sarajevo) CCI (Tuzla) CPCD (Sarajevo) NBR (Modrica) LINK (Mostar) OKC (Banja Luka) PhP (Sarajevo) ICVA (Sarajevo) TI (Sarajevo) Figure 15. Alternative Sources of Funding Planned within Sustainability Strategies 36 of sub-grantee CSOs (eight) expressed in KIIs is that they feel pressured to register an enterprise, regardless of their CSO type. CSSP implementers’ examples of registering commercial companies put forward as best case examples frustrated these sub-grantees. Resistance to the idea of registering an enterprise by some CSOs in part stems from undeveloped regulation that would distinguish social enterprise from traditional for-profit enterprise in BiH (reflected later in Finding 31 and Recommendations 21). Assistance provided for development on Sustainability Strategies is perceived by eight CSOs as not being equally useful and applicable for all CSOs that participate in CSSP, depending on their targeted purpose (advocacy CSO, watchdog CSO, service provider to citizens from vulnerable groups, representing interests of associated private sector representatives, service provider to government, or think tank) and ideological issues with the notion of civic activism moving to the private sector. Finding 17: Some aspects of sub-grant management are perceived as counter-intuitive and clumsy, particularly in terms of the quantitative target for decreased dependence on foreign donors. Most of the sub-grantees (ten) believe that the quantitate target for decreased dependence on foreign donor funding (30 percent of core budget being funded from sources other than foreign donors) should not be the same for all sub￾grantees given the specifics of different CSO types, in particular those who perform watchdog activities or provide services to socially vulnerable groups. CSO sub-grantees were informed about the financial sustainability target only a year into the process of CSSP implementation. Furthermore, although CSSP implementers explain that they never presented this target as obligatory, several CSOs (five) perceive this target as mandatory for continuation of the CSSP implementation and consequently find it counter-productive and frustrating. Moreover, each organization defined its core level of needed financing differently, rendering the indicator not comparable among CSOs. Criteria for tasks that can be supported by the CSSP capacity building and sustainability sub-grants is perceived and explicitly described as not being clear by some sub-grantees (six), who were originally under the impression that CSSP funding would also be applicable for startup capital for implementation of alternative sources of financing. Finding 18: CPCD’s Resource Centre (RC) has functioned primarily in parallel to CSSP and neither CSSP reporting or KIIs suggest much synergy between the work of the RC and CSSP. There are few clear connections between RC development efforts and the CSSP intervention. CPCD continued expansion of its RC program offerings, with the CSSP reports noting that the RC acquired 120 new members during the first two and a half years of CSSP implementation, indicating a higher number of organizations and individuals using RC information services. CSSP reports also indicates that the RC has delivered over 80 trainings and events in this period, including several rounds of basic trainings for organizations and individuals in the area of project management, use of social media, financial management, RC marketing/promotional events, and information on EU financial programs. However, it is unclear whether and how some of the reported RC trainings are connected to CSSP (e.g. Hour of Code engagement of schools and students and English conversation classes for students) and generally the connection between the RC and efforts and CSSP are not specified. Based on CSSP reports, the overlap of the CSSP activities and the RC efforts is present in CPCD’s use of the RC in their activities related to CSO policy environment and in related communication with other CSOs, as well as in use of the RS expert database to identify potential technical assistance providers to sub- “The 30% target cannot be applied to all organizations, as organizations come from a wide spectrum in terms of topics we work on. For some, such as those working in agriculture, it is only natural that they are able to reach this, but for others that work on increasing accountability of government, this is very hard. We also were informed about this indicator late, when it was already too late for us to give up the entire CSSP project, which was our only alternative other than to accept it. - CSO sub-grantee 37 grantees within the capacity building and sustainability sub-grant. There is little evidence that the RC is used for other CSSP needs and that CSSP CSOs consider the RC as part of the CSSP resources available to them. EVALUATION QUESTION 3 To what extent have interventions under CSSP increased engagement of civil society and citizens in governance and what was achieved within the 12 selected reform areas? Sub-questions: Which types of policy influence are being achieved and what are some of the factors contributing to this? What were the success factors in terms of providing relevant and credible policy inputs to government and government’s acceptance of those inputs? Finding 19: The 40 chosen advocacy sector campaigns vary greatly among different networks, in definitions and scope of targeted policies, their political context, and their maturity/ripeness. CSSP reports describe all networks as having impacted government policies through CSSP, however, these claims cannot be confirmed because CSSP did not design campaigns with a clear way of tracking how their inputs result in influence, as illustrated by uneven CSSP reporting. CSSP was started with the idea of influencing the reform process and improving governance. Hence, CSSP design implicitly aimed for the campaigns to result in some type of policy change. The extent to which CSSP interventions have affected policy change during the first two and a half years of the activity is unclear due to methodological and reporting concerns/limitations. Policy change is seldom attributable to a specific intervention. Policy processes by definition are complex, and designing such a system of verification includes articulated theories of change or process tracing approaches. In the case of CSSP, these approaches were not part of the initial CSSP M&E baseline effort. Rather, CSSP interventions were undertaken without clear ways to distinguish the actual inputs and influences CSSP activities might have in a reform area. 38 Expanding policy capacity Broadening policy horizons Affecting decision regimes Campaign not started 1 Unequal access to health care and discrimination of citizens, who have different access to health care services depending on their place of residence x 2 Exclusion of citizens from health care system by withholding information x 3 Citizens uniformed of their rights and obligations in the healthcare system x 4 Violence against women and domestic violence x 5 Discrimination of women in labor market x 6 Gender sensitive budgets x 7 Improving position of women in rural areas x 8 Lack of transparency in public procurement process and lack of anti-corruption mechanisms in the legislative framework (campaign aimed at improving the legislation on public procurement) x 9 Limited circle of persons and institutions that are subject of the legislation on conflict of interests and mild sanctions that do not incite adherence to the legislation (campaign aimed at improving the legislation on conflict of interest) x 10 Lack of control over asset declaration forms (campaign aimed at establishing mechanism for monitoring accuracy of elected officials’ asset declaration forms and responsibility for reporting false and incorrect information in the asset declaration forms) x 11 Request for urgent establishment of a single Creative Europe Program Desk x 12 Solving the position of the BiH National Museum x 13 Developing Culture Strategy in Sarajevo Canton, and initiating development of strategic documents at other levels x 14 Depolitization of managerial functions in the public institutions of culture as part of reform of public sector of culture in BiH x 15 Introducing minimum standard for social security in both Entities and activating social protection x 16 Improving extra-institutional support to marginalized groups x 17 Amending the legal framework for improvement of social entrepreneurship x 18 Amending the BiH Election Law in terms of electoral rights of national minorities x Culture Human rights Sector Priorities for Advocacy Campaigns LEVEL OF POLICY INFLUENCE Health Women's rights Anti-corruption Figure 16. Categorization of CSSP Sector Priorities for Advocacy Campaigns by Level of Policy Influence 39 Expanding policy capacity Broadening policy horizons Affecting decision regimes Campaign not started 22 Rational and efficient allocation of budget funds to combat unemployment at the Entity level x 23 Government institutions design and implement employment and economic incentives programs in line with the recommendations for efficient, transparent and effective performance x 24 Exempting the employers from paying social insurance contributions on employees’ salaries during the first year of employment, and postponing payment of VAT until payment is received for invoices based on which VAT was accrued x 25 Enactment of the Law on Agricultural Subsidies Rural Development in Republika Srpska x 26 Enactment of the changes to the Law on Agricultural Advisory Services in FBiH x 27 Enactment of a policy that regulates the area of organic food production x 28 Ensure constitutional and legal preconditions for independence of judicial institutions x 29 Improve process of appointing judges and prosecutors and strengthening the role of the HJPC BiH as an independent institution x 30 Ensure financial independence and autonomy of judiciary in BiH x 31 Amending the Law on Budgets in FBiH, the of RS and Brčko District to introduce Budget for Citizens x 32 Amendments to the Law on Audit of BiH Institutions related to measures and sanctions for implementation of audit findings x 33 Improving implementation of regulatory framework on rules for participation of interested public in the process of drafting BiH Federation legal documents and other regulations x 34 Acceleration of the business registration process in FBiH x 35 Lowering the number of parafiscal levies x 36 Improving concrete support to business to counter flood damage x 37 Designing policies for development of concrete models and instruments of support to business (e.g. business incubators, issuance of construction and usage permit, credit guarantee funds, etc.). x 38 Lack of demonstration classes in schools x 39 Inadequate syllabuses/curriculums x 40 Poor alignment of enrollment policy with trends on the labor market x 1 19 15 5 LEVEL OF POLICY INFLUENCE Sector Priorities for Advocacy Campaigns TOTAL Employment and labor market Agricultur e and rural Justice Public finance Economic development policies Education 40 CSSP reporting reflects these limitations. As shown in Figure 16, 40 sector campaigns differ by scope, with some campaigns targeting a specific legislation/strategic document (e.g., Law on Agricultural Subsidies and Rural Development), while others target more general policy area (e.g., rational and efficient allocation of budget funds for combatting unemployment). Furthermore, six campaigns have not been designed with the goal of affecting a legislation/strategic document, but are rather focused on raising citizen awareness and/or political will, or targeting improvement of general position in society (e.g., educating and motivating citizens to implement measures for energy efficiency and renewable energy sources). Finally, in some sectors, campaigns are overlapping, i.e. a more narrow campaign defines for a sector is essentially a subset of a more broadly defined campaign in the same sector (e.g. employment and labor market sector, education sector, and energy efficiency sector). Reporting of policy changes is uneven, with the reporting clarity and quality varying from campaign to campaign. CSSP reports suggest that all sectors achieved policy impact; however, evidence of causality is partial or unclear. KIIs confirm concerns about reporting accuracy. All sector leader/co-leaders explained that CSSP campaigns are building on or expanding their previous work on chosen sector issues, rather than being solely the result of CSSP. This refers to both continuation of previous work on a specific issue (e.g., economic development policies) and in some cases narrowing the focus of previous advocacy efforts (e.g., in education). Moreover, in many cases (13), sector leaders/co-leaders also described that they currently working on the same advocacy issue or different aspects of the same advocacy issue on multiple donor projects. Only one quarter of sector leaders/co-leaders were able to describe in more specific terms how they might have contributed to policy influence, but even those cited multiple factors for the policy influence achieved. Finding 20: The evaluation team’s analysis of campaigns’ policy influence shows that varying types of campaigns, levels of campaign specificity, and policy issue maturity levels result in a mix of policy influence during the first two and a half years of CSSP. To better understand and display sectors’ campaign variation, the evaluation team analyzed campaigns’ results described in CSSP reports19 against the Lindquist/Carden20 categorization used for tracking policy influence in the policy process. These topologies provide a descriptive way of understanding relative campaign achievements, as shown in Figure 17. 19 Progress to date is based on CSSP Annual Reports, CSSP Quarterly Reports, and CSSP write-ups on achievements, as well as KIIs and where possible a review of partner websites and related documents. These claims did not go through a verification process. 20 Based on the work on policy influence topologies developed by Evert Lindquist (2001) http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.497.5165&rep=rep1&type=pdf, later expanded by Fred Carden (2009) http://idl-bnc.idrc.ca/dspace/bitstream/10625/37706/1/IDL-37706.pdf and also used in the Politics and Ideas Online Training Course to Strengthen MEL on Policy Influence in Central and Eastern Europe, which was produced by Vanesa Weyrauch with collaboration of Kristie Evenson and Dena Lomofsky http://www.politicsandideas.org/?page_id=2303. 41 Policy influence coding suggests that a majority of campaigns (19 of 35 that started21) can be described as in the ‘broadening policy horizons’ phase. One campaign can be described as expanding policy capacity22 and 15 are described with the CSSP reporting as affecting decision regimes, as shown in Figure 17 above (Annex IX shows details on categorization of each policy campaign). Some attributes of broadening policy horizons and decision regimes are discussed in more detail below. In general, efforts in broadening policy horizons relate to gathering diverse actors in the society around the same issue, to give the same priority to that issue and assume the same language on the definition of the problem and results expected from policy interventions. The CSO sector leaders describe taking three different main approaches to identifying key policy actors they deem necessary for building up a joint approach. First, some sought out technical experts among operational practitioners from government/public institutions. For example, in the judicial sector, CSO sector leaders described a specific concentration on technical experts from within the judicial system by focusing on associations of professionals from the justice system (e.g., Association of Judges). Similarly, the energy efficiency sector leaders described how they identified the relevant stakeholders among local government administrative workers, with whom they could build a common approach under the issue and then use this to pressure higher levels for the adoption of the Energy Efficiency Law. Second, some sector leaders have focused on direct targeting of individual policy decision actors or specific institutions from either legislative or executive government. The anti-corruption campaign leaders focused on working with selected parliamentarians to secure their support for their public procurement initiative and proposed changes to the public procurement legislation. Other networks, such as employment and labor market and economic development policies, focused on specific ministries relevant to their sector. 21 All together 40 campaigns were described as planned, but five had not yet started by the end of second quarter of CSSP’s Year 3 implementation (January-March 2016). In the analysis done by the evaluation team, the campaigns are taken as defined within the CSSP reporting, although it is questionable whether there are actually 35 separate campaigns, as in certain cases the way campaigns are defined is more in name than in practice. For example, the education network has two campaigns listed (lack of demonstration classes and inadequate syllabus/curriculum) that are in practice interlinked, and campaign efforts for both are similar and jointly implemented, according to CSSP reporting. Moreover, both of these education network campaigns are essentially a subset of the third campaign defined in this sector (poor alignment of enrollment policy with trends on the labor market). 22One campaign is specifically coded as being in the expanding policy capacity phase. Expanding policy capacity often includes educating key actors around the issues; but in this case, the evaluation team highlighted the fact that sector leader was specifically building the technical capacity of officials to develop a Cantonal Cultural Strategy and planning to later initiate a similar process at other government levels. Expanding policy capacity Strengthening the ability of public authorities to recognize the need and benefits of receiving evidence-based policy proposals and to be able to utilize those proposals. Broadening policy horizons Getting relevant stakeholders to have a similar understanding of an issue, which might include putting the issue on the agenda, defining its emphasis, and connecting it to other issues. Building a shared understanding of an issue and its priorities builds consensus for taking eventual policy action. Affecting decision regimes Research findings or advocacy efforts modify the development of a certain legislation, governmental structure, or regulation or the way policy process decisions are designed and made. Figure 17. Policy Influence Levels (as adapted from Lindquist/Carden) 42 The network for economic development policies has focused on highlighting its priorities mainly with the ministries responsible for economic development and entrepreneurship at all levels. The third approach used by some networks is a broader multi-audience approach. For example, in the health care sector, the sector leader described efforts and activities that reach out to both the general population and decision makers within the government. Getting citizens aware of their rights for health care and institutions actors more sensitive to this is an example of how framing an issue for stakeholders is a phase for setting the base for further policy advocacy. Finding 21: Of 15 campaigns described as achieving some level of decision regime influence, eight describe inputs into already adopted implementable legislation/strategic documents; the remaining are at earlier stages of the policy making process. As shown in Figure 1723, 15 of 35 campaigns describe some type of policy influence on decision regimes. Nine of 12 sectors have influenced decision-making regimes in one or more campaigns (anti-corruption, culture, environment protection and energy efficiency, employment and labor market, economic development policies, health, women’s rights, human rights, and education). However, a closer look at the results, as shown in Annex IX, suggests that getting to the point of actual inputs into legislation takes time, partly due to the complex political context in BiH. Only eight campaigns categorized as affecting policy regimes describe actual inputs into already adopted implementable legislation/decisions. Of these eight, two results of two campaigns resulted in temporary, rather than permanent solutions (reopening of the BiH National Museum and financing of the health insurance for 32 children from vulnerable categories). The remaining six campaigns described some concrete and or formal change in the policymaking in early stages, such as policy proposals entering work plans and CSO representatives being part of working groups drafting strategic documents. The campaign examples, in line with political theory, also suggest that getting to the point of proposing specific inputs into policy design often first requires different mechanisms in order for the policy proposal to reach maturity. One such mechanism is advocating to ensure that the need for policy proposal/change is recognized and will be considered in the future, by inserting it into the relevant government institution’s work plan or strategic documents. For example, the sector leader for economic development policies describes the achievement of having one municipality include the network’s suggested model of business/agro incubation into its work program as a business support instrument. In human rights sector, the campaign to regulate extra-institutional support to the marginalized citizens resulted in inclusion of this priority in the Federation of BiH (FBiH) Government’s work plans. A second mechanism is getting a seat at the table, and thereby forming and being active members of working groups to develop specific policies. For example, the education network participated in the Working Group for Development of the RS Youth Policy and the Working Group for Development of the Education Strategy in the Republika Srpska (RS), whereby they advocated for those strategies to include a priority goal of linking education system with labor market needs. Similarly, members of the network for economic development policies participated in the creation of the Action Plan for allocation 23 It should be noted that the categorization of the campaigns by the policy influence levels was based on CSSP reports and did not go through a process of verification with the relevant government policy makers. Furthermore, campaigns were categorized as affecting decision regimes in all cases in which network’s recommendation (fully or partially) entered official legislation or strategic documentation of institutions (e.g. Work Plans, establishment of Working Groups), Finally, the campaigns were coded as affecting decision regimes in the cases where network’s recommendations that were adopted refer to only a subset of overall defined campaign priorities, either in scope of policy (e.g. regulation related to maternity benefits do not address comprehensive priority of discrimination of women in labor market in its entirety) or in terms of government level (e.g. a problem of para-fiscal levies being addressed in one municipality). 43 of funds for incentivizing economic development by the Federal Ministry for Development, Entrepreneurship, and Crafts. Furthermore, some campaigns focus on getting adopted legislation to be implemented as designed. For example, the anti-corruption sector’s campaign led to the Agency for Protection of Personal Information to issue an official statement saying that there are no obstacles to release the information on property cards of public officials, thus ensuring adequate application of the existing legislation. Finding 22: Several aspects of up-to-date CSSP implementation emerge as success factors for maximum influence on decision regimes, defined as providing relevant and credible policy inputs to government and governments’ subsequent acceptance of those inputs. As noted before, the diversity of scope of the thematic campaigns and their contextual specifics and different maturity levels mean that sectors and campaigns cannot be compared to one another. However, it is possible to examine different techniques and manners in which the sector leaders/co-leaders took on the issues. From this analysis, a number of patterns and hence factors can be delineated that appear to contribute to the levels of policy influence achieved. Evidence-based research/analytical products and concrete actionable technical recommendations as a base for engaging in the policy process Those campaigns described as having some policy influence at the decision regime level all have some analytical basis. Campaigns have varying levels of analytical product development, with some providing high-quality evidence-based analyses of the current situation and expected influence of proposed changes; for example, the network for economic development policies mapped para-fiscal levies. Others provided analyses based on monitoring implementation of current policies to show where and why changes are needed. Three sectors described such analyses based on monitoring the implementation of current policies - the health network is tracking implementation of citizens’ health care access, the agriculture network is monitoring implementation of organic farming practices, the anti-corruption network is monitoring implementation of public procurement legislation. A second point in relation to such products is their level of ‘actionability.’ Several sector leaders described their submitted policy recommendations as providing concrete solutions taking into account both legislative and budgetary aspects. Examples include culture, women’s rights, anti-corruption, and energy efficiency. Interviewed representatives of government/public institutions unanimously confirmed the utmost importance of proper analyses and actionable policy proposals from CSOs, noting that in most cases these are missing. Meaningful consultation processes and a common approach to practical solutions among practitioners A combination of meeting with the key stakeholders from inside public institutions and out and having such collaborative processes be a point of dialogue and purposeful exchange is an area that several cite as a key factor in success. Although this collaborative process between the CSOs and the relevant government institutions is usually not as visible as for example street actions, it is nevertheless essential in both collecting the relevant information and perspectives from the practitioners, and in building trust between the government and CSO sector and increasing the credibility of CSOs’ proposals from the government “A general observation of mine would be that we need more direct cooperation between government institutions and CSOs at the beginning and middle of the projects and campaigns, not only at the end. We need projects not presentations to which I will be invited, will attend it, be on the photo and maybe give a brief verbal comment or two. We need to work together and there is space and will for that, provided that quality and well￾researched inputs are prepared.” - Government/public institution representative 44 side. An illustrative example of the importance of these efforts is the culture network’s campaign for reopening of the National Museum. Less on the radar than the very public later actions to keep the museum open, the sector leader described eight months of research into the practical and legal specificities of the issue, consultation processes with museum officials, and then consultation with the Ministry of Civil Affairs prior to the public campaign. Another successful example is the environment protection and energy efficiency network, where the sector leader was directly contacted by the relevant government institution for advice or contracted for development of strategic documents, given their reputation based on the quality of their research and policy proposals. Notably in this example, the sector leader first provided detailed evidence-based analysis of the proposed legislation and presented it to the technical expert colleagues in Entity Government institutions. Technical quality of this analysis was appreciated, resulting in the sector leader being invited to work on action planning with the Government. Similarly, the anti￾corruption network works closely with the BiH Agency for the Prevention of Corruption and Coordination of the Fight against Corruption. While a majority of campaigns have described consultation with different government institutions,24 KII feedback from government officials suggests that still in many cases, the CSOs’ consultation process with the relevant government institutions takes the form of presentation rather than dialogue. Being able to bring together a combination of citizens and experts around the same priority The energy efficiency network provides a good example of how to bring together diverse groups for a common objective. By implementing educative campaigns on energy efficiency legislation targeted at both citizens and the municipal and cantonal government institutions, the sector leader secured the network’s legitimacy and credibility and approached the FBiH Parliament with a letter signed by 40 public institutions (municipalities, towns, ministries, etc.) requesting adoption of the Energy Efficiency Law that was halted in the adoption procedure at that moment. This case also provides an example on how campaigning towards the citizens can take forms other than the most frequently used street actions and protest. In this case, by educating the citizens, the sector leader simultaneously earned legitimacy from the citizens to represent their interest and gained their support in exerting pressure on policy makers. Other sectors have not used such mechanisms. Being able to seize political context and policy maturity In a complex political environment such as BiH, issues can go unaddressed or unresolved for years, often due to a combination of (lack of) political will and or political inertia. Knowing the political moment is thus essential for increasing the likelihood of advocacy success. It is both a talent and a skill, and many organizations ‘know it when they see it,’ but do not necessary anticipate or plan for it. For example, in the case of the network on employment, public pressure of business associations and media to adjust the value-added tax (VAT) payment schedule (so that VAT is paid after collection of debts based on which VAT was accrued) was in the making for some time and was being considered by the incoming government, so the sector leader campaigns coincided with a maturing of the political idea. Similarly, pre-election pressure appears to have aided the campaign on resolving issues on unequal benefits to pregnant women and mothers, when several NGO initiatives peaked in parallel with the support of the Women Caucus in the FBiH Parliament, the existing lobbying group, and a number of politicians who stated this as their priority in the pre-election campaign. Media and public outreach and innovation in campaign visibility 24Twenty nine out of 35 active campaigns undertook a combination of consultation and public campaigns. 45 Public outreach is a common component of all campaigns and the media exposure CSSP campaigns are receiving is notable. Yet, interviews with media professionals suggest that campaigns generally are still not achieving media attention to their fullest potential. Media is paying attention to campaigns (as described by media monitoring data from CSSP reports), but most of this is still in the form of presentation format (roundtables, press conferences etc.). All exposure is likely useful, but doing media and public outreach in an innovative way appears to increase the likelihood of policy success. The culture network’s National Museum campaign provides a good example of how using creative messaging, interesting events, social media, and celebrity power can build up interest and pressure – even creating citizen volunteer opportunities (to ‘guard the museum’) as a way to encourage citizen engagement. While not every issue is likely to stir such public attention or utilize the public in such ways, there is a middle ground for finding ways to make the issue area interesting to identified target audiences and ways to engage with them. There is space for peer-learning from media expert partners and innovative campaigns targeting public engagement. Use of CSSP infrastructure for reaching wider constituencies CSSP’s broad networks and infrastructure can be used to quickly engage with and gain support from a wide set of potentially active and engaged stakeholders. As noted earlier, few network leaders or members had detailed knowledge on campaigns of other networks, except for a few cases of campaigns that were pro-active in reaching out to broader CSSP network for support (examples include the women right’s campaign on maternity benefits and the culture sector’s campaign for the National Museum). Campaign strategy and vision All the previously mentioned success factors come back to how a campaign is designed and articulated. For example, ‘changing and improving the law on conflict of interest’ is arguably more focused and finite than a campaign to ‘improve position of women in rural areas.’ In such a comparison, progress in the more focused area (all other things being equal) might be faster than in the latter. Conceptualization specificity and campaign design strategy are closely linked to the above. Generally speaking, if a campaign is articulated in vague terms, at least the articulated (if not intuitive) strategy for addressing the issue is less conceptually ‘crisp’ than in other situations. Finding 23: While the CSSP design emphasized that the policy influence should be focused on areas relevant for accelerating the process of European Union (EU) integration, CSSP implementation focuses little on policy changes specifically needed to accelerate EU integrations at this stage. Only a few sectors report on some connections with BiH’s immediate EU integrations requirements. Despite the declared CSSP program design to focus on reform processes in line with EU integration, only a couple of CSSP campaigns relate to the most immediate EU integrations requirements, such as the work of energy efficiency network on National Energy Efficient Action Plan (an obligation under the Energy Community) and the work of human rights network on electoral rights of minorities (an obligation under European Court Decision on the Sejdi-Finci case). A few other campaigns can be more broadly connected to some longer-term EU integration requirements (such as the justice network’s campaigns that may help to prepare the grounds for implications of EU acquis related to the judicial sector and the agriculture and rural development network’s campaign to improve the legislation on agricultural advisory services, which is important to the implementation of EU rural development assistance). Finally, a few additional campaigns may be considered more broadly as contributing to achieving EU standards/best practices, however they are not explicit integration requirement. 46 In addition to the low share of campaigns that focus on adjustments explicitly needed for this stage of EU integration, an indication of insufficient attention (relative to CSSP design) given to EU-related reforms is the fact that there has been no cooperation with the BiH Directorate for European Integration (DEI) during CSSP implementation, although a DEI representative was included in the Local Advisory Group that selected sector leaders/co-leaders. Finding 24: The CSSP management process is perceived as overemphasizing public campaigning. The name of the sub-grant for policy influence (‘campaigns’) and even more so the actual implementation and CSSP management’s perceived emphasis on the need to make a public and interactive call for action in different reform areas. Certainly such actions and tactics are part of a larger advocacy effort, but they are not necessarily the major part of the effort, and should also be implemented at the specific phase of policy maturity (and after other forms of collaboration with the decision-makers have been exhausted). For some sectors and campaigns, depending on campaign nature and maturity, less publicly visible efforts are more appropriate. Policy advocacy efforts are not limited to outreach towards the citizens, but also on building partnership and legitimacy with the authorities. Moreover, in addition to different ways of expressing citizens’ dissatisfaction with the policies, the spectrum of civic engagement includes other types of engagement, such as educating citizens and encouraging volunteering. The CSSP implementers explain that they insisted on public visibility of the campaigns, without necessarily being focused on negative campaigns. Yet, based on the KIIs, almost all sector leaders perceive some pressure by the CSSP management and USAID/BiH to focus on public campaigning and street actions voicing public criticism of authorities. Several CSOs described this as very frustrating and potentially undercutting their own planning. They cautioned that advocacy techniques need to be carefully assessed and that such public campaigns with critical stances may in certain cases undermine progress on issues that may be close to resolution by stimulating too much negative attention. Furthermore, several sub￾grantees perceive the need for public exposure as mandatory and have consequently adjusted their campaigns to be in line with this request. Finding 25: CSSP management’s primary focus on administrative and operational guidance resulted in insufficient advice for campaign design and implementation for the sub-grantees that might need it. Many of the factors for success described above are related to the relative abilities of the sector leaders. If an organization has a vision, strategy, and ability to engage the right people in the right way with the right evidence-based and actionable materials, their chances of getting positive results are higher—even given the difficult political context in BiH. If an organization is not organically equipped with such capacities/features/talents, finding ways to assist development of these abilities is the next logical step. While the CSSP management’s regular involvement with the sub-grantees is evident (from both KIIs and the CSSP sub-grantee survey conducted and reported by the CSSP implementers in which less than 2% of “We are under the impression that CSSP management is instructed to put pressure on us to focus on street actions and citizen protests. So we tried to expand activities to do more of those, although that was not easy, as we did not plan to do so originally.” “I am afraid that campaigns targeting citizens’ protests end up being a lot of smoke and not enough fire. We can gather media coverage easily, but our goal is not a street action covered by media in which a few people say a few words. We strive to get involved in serious dialogues with policy makers based on analyses.” “Our primary role should be to convince policy makers that improvements are necessary and that we can jointly solve issues and propose good recommendations if we work together”. - CSO sub-grantee 47 the sub-grantees note that the implementers are rarely or not at all involved in the activities of the sub￾grantees), its emphasis so far has primarily been on administrative and operational guidance, rather than substantive strategic guidance that would include sector-specific advice and structured and consistent criteria for definition and implementation of targeted campaigns and tasks and for reporting on them (as evident from both CSSP reports and KIIs). Finding 26: Design of the sub-grant for policy campaigns did not emphasize strategy development or relevant tools to manage how to track the effectiveness of a campaign’s strategy or approach. CSSP reporting seems to overstate influence and achievements of the campaigns. Conceptualization of the campaign idea also undercuts a more comprehensive way of designing and assessing steps in a larger advocacy process. CSSP did not require, nor did most organizations design, a campaign proposal that would include basic elements of strategies and tactics. The KIIs suggest that some organizations have done this intuitively, even if they did not articulate it in their three sub-grant proposals.25 However, overall, there is a significant variety in the level of strategic planning from campaign to campaign, and in most cases there is no clear link between how specific actions can lead to the intended objective. As a result, both CSSP Implementers and the CSO sub-grantees have had less than meaningful management tools for tracking their campaign efforts26. Proposals were not required to have an articulated strategic approach in the form of a logical framework and/or development hypothesis; and they are not designed with M&E tools that can provide meaningful feedback on the relevancy of their approach and influence levels reached.27 Organizations have regular reporting requirements, but these are used more as compliance mechanisms to ensure output progress, rather than ways to assess the effectiveness of approach and efforts. CSSP reports seems to overstate CSSP’s influence in campaign achievements and include uneven descriptive accounts, unsubstantiated impact claims, and often incomplete information. Several KIIs perceive that pressure exists to present CSSP impacts. Finding 27: Only a few CSSP CSOs systematically monitor policy implementation and policy change processes in their sector. 25The culture sector leader, for example, articulated a strategic approach to their campaign. The anti-corruption sector leader and co-leader also described a planned campaign and how they divided their roles to target both public and government institutions. The environment protection and energy efficiency sector leader articulated how its campaign design was mutually reinforcing to lead different target groups (citizens, municipal officials and civil servants, and parliaments) towards the same objective. 26 It should be noted that in the process of providing comments to the first draft of this evaluation report, CCI explained that in the second half of 2015 the CSSP management identified the lack of campaign clarity in some cases and has subsequently introduced a campaign management tool (including campaign action plans and assistance in their development), which they believe would result in improved campaign planning later in the upcoming period (as the plans they worked with the eight sectors have started in 2016). However, this tool was not specifically mentioned in the KIIs conducted for this evaluation and in the CSSP reports provided to the evaluation team; nor was a sample or template of the tool provided to the evaluation team before or during the evaluation. 27The evaluation team’s difficulty in tracking actual progress in the respective sector campaigns reflects the lack of meaningful reporting formats. “It is requested of sub-grantees to show how much they influenced a policy in their sector, and due to this pressure, organizations overestimate their results and state that they can be attributed to any change that occurred...” - CSO sub-grantee 48 The anti-corruption and health sector networks are the only networks that explicitly articulated their focus on monitoring of existing policy implementation and policy change process in their sectors, with environment protection and energy efficiency sector planning to additionally focus on this as well. All interviewed representatives from government/public institutions note that the CSOs generally do not sufficiently and systematically use existing mechanisms for civic participation, such as submitting policy proposals during legislation adoption procedures, EU-fund programming, and e-consultations. Finding 28: Activities led by CSSP implementers in creating an affirmative work environment for CSOs are perceived as generally well-targeted, and are mostly a continuation of efforts previously invested by the CPCD and other CSOs over the last decade. Five campaigns have been implemented by CPCD and CCI under this part of the CSSP intervention: 1. More enabling taxation framework for individual and corporative philanthropy (implemented by CPCD) 2. Transparent system of public funding of CSOs (implemented by CPCD) 3. Establishment of functional cooperation mechanisms between governments and CSOs (implemented by CPCD) 4. Promotion and introduction of the Code of CSO Ethics and Good Governance (implemented by CPCD) 5. Establishment of public foundations (implemented by CCI) The KIIs confirmed the relevance of these initiatives, with most noting that the priority initiatives out of these five are initiatives for transparent system of public funding of CSOs and more enabling taxation framework for individual and corporative philanthropy, while the initiative on establishment of public foundations is seen as not realistic and even legally not implementable (since foundations cannot be established by the state). The vast majority of sector members/participants also confirmed that the five interventions are of high priority, as shown in Figure 18. Figure 18. Sector Network Members/Participants’ Feedback on Priorities for CSSP’s Work on Improving Policy Environment for CSOs 49 Several interviewed CSOs noted that work on these initiatives has been invested by numerous CSOs over the last decade. All interviewed government institutions welcomed the initiative on Code of CSO Ethics and Good Governance, noting that it could represent an important step in building trust and partnership between the civil society sector and government. Findings 29: Significant preparatory work has been undertaken by CSSP implementers in all five campaigns advocating for CSO policy environment improvement, with some success already recorded in adoption of new policies in two out of five initiatives (submitted policy proposals have been partially taken into account in recently adopted legislation/methodology relevant for transparent system of public funding and a more enabling taxation framework for philanthropy in RS). However, awareness of and participation in efforts related to the CSO policy environment by other CSOs/networks is low and they perceive progress in these efforts as weak. CPCD (the implementer of four of the five campaigns) held informative-consultative meetings with representatives from CSOs, government, and the business sector, and issued public calls to form an informal cross-sector coalition for civil dialogue, that would advocate campaigns for a better institutional, legal, and taxation framework for “I have to admit that I am not familiar with these initiatives, although they are certainly important for all CSOs. CSSP implementers need to talk more about these during our internal CSSP meetings, and not only sector themes. They mostly take on the role of coordinating the sub-grants, so we do not know enough about their own CSSP tasks and initiatives. I am sure we can find synergies among all of us to work together on these CSO policy environment initiatives.” - CSO sub-grantee “These initiatives are of vital importance to all of us, to all CSOs, and we expect the CSSP Implementers to strongly push for these initiatives, because they received funds for those, while we do not have the capacities. However, this is not their initiative only, as we have worked on these for years already. We should all work together, all CSSP CSOs and even wider, all CSOs.” - CSO sub-grantee Figure 19. Sector Network Members/Participants’ Feedback on Familiarity and Progress of CSSP’s Work on Improving Policy Environment for CSOs 50 development of civil society in BiH. In addition, press conferences were held for all of the four CPCS-led campaigns. However, most CSSP sub-grantees and network members/participants are poorly informed on CSSP’s work in this area. On average, only half of the sector network members/participants were aware that CSSP works on the five campaigns under this part of interventions, while only one-fifth of them believe that progress has been made since the start of CSSP in 2013 (Figure 19). Based on the KIIs, CSO sector leaders/co-leaders are also generally insufficiently informed about these interventions and noted weak up￾to-date progress in their adoption, with the exception of the initiative for revision of the Agreement on Cooperation between the Council of Ministers of BiH and the NGO sector. CPCD noted that the extensive preparatory work done to date (including several research reports, proposals for legislation amendments/proposals, and meeting sessions and events that included both CSO representatives and government representatives) should result in visible progress during the next period. But CPCD also acknowledged that there should be more coordination and cooperation with the sector leaders/co-leaders in these initiatives. In terms of establishment of functional cooperation mechanisms between the governments and CSOs, objectives are: i) to raise awareness about the role of CSOs’ participation in the dialogue with governments, ii) to establish new and/or support the work of existing government bodies for cooperation with the civil society at the State, Entity and Brcko District level, iii) to advocate for implementation of the 2007 Agreement on Cooperation between the CoM of BiH and the NGO sector (or its revision/adoption of a new agreement), and iv) to support establishment of an umbrella institution of CSOs that would be the CoM’s partner in development of public policies on the key civil society development issues in BiH. Under this campaign, the focus has been on the Agreement on Cooperation between the CoM of BiH and the NGO sector. A series of events were held, including a large conference in the Parliamentary Assembly of BiH, which was organized in cooperation with the Cabinet of Chairman of the CoM of BiH and attended by representatives of government institutions, CSOs, academic and donor communities, and international organizations. As a result of this event, a comprehensive initiative28 was agreed on, with the necessary steps to unlock the deadlock in the process of dialogue between the CoM of BiH and interested NGOs on the Agreement on Cooperation between the CoM of BiH and the NGO sector. The initiative is currently in the process of collecting online signatures; and it underlines the need for establishing bodies for cooperation with the CSO sector within the CoM of BiH and encourages the Entity and Brcko District governments to adopt similar agreements and establish bodies for cooperation with the CSOs. Moreover, the initiative also refers to other CPCD-led campaigns for improving CSO policy environment, including the Code of Ethics. Given that EU-funded CSO project CBGI (Capacity Building of Government Institutions) has worked with the Ministry of Justice of BiH on this initiative, CPCD has recently established a closer cooperation with the CBGI and the Ministry of Justice of BiH, with the plan that CPCD will undertake public consultations with the wider CSO community upon adoption of the Agreement (expected soon). In terms of the campaign for the introduction and promotion of the Code of Ethics and Standards of Good Governance for the CSOs, several roundtable discussions were held, during which the norms of transparency of the CSO sector were discussed—noting that transparency in reporting to donors needs to be accompanied with transparency in reporting to citizens on sources of financing, internal control systems, and results of activities. The conclusions of the roundtable discussions include plans to work on building the culture of transparency and accountability internally (targeted at CSO employees and volunteers) and externally, as well as to constantly exert pressure on government institutions to increase decision-making transparency and provide opportunities for citizen engagement in that process. The need 28 Available at http://www.cpcd.ba/files/Inicijativa_za_hitnu_uspostavu_institucionalnog_mehanizma_dijaloga_i_saradnje_Vijeca_ministara_BiH _sa_civilnim_drustvom1.pdf. 51 for establishing an online CSO registry was also underlined. CPCD explained that the work on drafting the Code is being finalized (during which special attention is given to financial transparency), and that public discussions will be held combined with other initiatives within this component. CPCD notes that progress has been made in both establishing guidelines for increased transparency in public funding of CSOs and in more enabling taxation framework for individual and corporative philanthropy in RS, while they are working with the authorities to improve these areas in FBiH as well. In terms of establishing guidelines for increased transparency in public funding of CSOs, CPCD aims to raise awareness and inform CSOs and the public about how public funds are awarded and spent, as well as to adopt a code or guidelines for transparent allocation and management of public funds awarded to CSOs. Based on CPCD’s analyses of CSOS’ current funding from public funds in BiH and on EU guidelines for funding CSO interventions, inputs were provided to the RS Ministry of Finance29 most of which were taken into account by the Ministry in preparation of the Grant Management Methodology for Programs and Projects Financed or Co-financed from the RS Budget. CPCD is currently working with FBiH authorities on preparation of a similar methodology for the FBiH level. In terms of a more enabling taxation framework for individual and corporative philanthropy, primary objectives were to amend legislation on personal income tax and corporate income tax to include incentivizing philanthropy. Analyses conducted by the CPCD and a proposal for adjustment of legislation was submitted and presented to relevant Finance Ministries. Some of the recommendations were taken into account in the new corporate income and personal income taxation legislation adopted in RS, which broaden the types of services for which donations can be exempt from taxation. CPCD plans to continue to work with the RS authorities to further improve legislation in this area (especially in terms of recognizing donations as personal deductions under personal income legislation) as well as intensify cooperation with the FBiH authorities on amending its taxation legislation. Finally, the campaign for the establishment of public foundations (led by CCI) included organization of several events, presentation of the Croatian model, and analysis of institutional support to civil society through public foundations in the region and EU, with recommendations for BiH (the analysis was submitted to BiH authorities). However, given the generally weak reception of this initiative and the recent problems with the Croatian model, which was used as a basis for this proposal, CCI will consider switching from this campaign and instead synchronizing their activities with the CPCD’s campaign on increasing transparency in the current system of public funding of CSOs. 29 Methodology adopted by the RS Ministry of Finance is available at http://www.cpcd.ba/files/Metodologija_upravljanja_grantovima_RS1.pdf . 52 Finding 30: Key officials from relevant government institutions are not sufficiently involved in CSSP’s work related to CSO policy environment, and coordination with other donor interventions in the CSO sector is generally perceived as insufficient. Based on feedback provided to the evaluation team by KIIs with government institutions and CSO sub￾grantees, close cooperation with the relevant stakeholders from key government institutions for these initiatives is in some cases missing (e.g., with the Budget Sector of the Federal Ministry of Finance). Sector members/participants’ online survey comments also include recommendations for closer cooperation with key government officials and more coordinated actions and focus on results rather than processes. Furthermore, coordination with other donor interventions in the CSO sector is seen as weak, although recent examples of CPCD and CBGI coordinating their activities on the Agreement on Cooperation between the CoM of BiH and the NGO sector are welcomed. Finding 31: Additional issues perceived as priority but not addressed by CSSP include setting-up a mechanism for co-financing of EU funds for civil society, introducing legislation on social entrepreneurship, and campaigning for more transparent and coordinated donor funding for the civil society. CSOs identify additional needs for CSO environment improvements beyond the initiatives CPCD currently prioritizes. When asked whether there are other campaigns that should be CSSP's focus in terms of campaigns for a better institutional, legal, and taxation framework for development of civil society, 18 respondents gave comments. Repeated recommendations include setting up a mechanism for co-financing of EU funds for civil society, regulations for social entrepreneurship, and more transparent and coordinated donor funding of the civil society. Similarly, around half the CSO sector leaders/co-leaders independently noted that co-financing for EU CSO funding should be a priority. A regulatory framework for social entrepreneurship and more transparent and coordinated donor funding of the civil society was also repeated as a priority in several KIIs. Finding 32: CCI’s reports on monitoring of the work of governments and parliaments are perceived as useful. CCI continuously monitors sessions of parliament and government sessions at the Cantonal, Entity and State levels. More than 250 sessions are attended on annual basis, while additional data sources for CCI’s monitoring include official minutes of sessions, as well as transcripts, official gazettes, and other official documents. Based on this monitoring effort, CCI identifies recommendations, which are presented to the public along with the monitoring findings, in highly visible media outreach (press conferences, press releases, CCI website, Virtual Parliament portal, and TV shows at the Initiative TV channel and other “The initiative related to increasing transparency in public funding of CSOs did not have any systematic progress, as far as we know and see on the ground. I was present at the meeting organized by CSSP for this initiative, and their presentation was good, however, what was missing is the participation of relevant representatives from the Ministries to explain this from formal and legislative stand-point. - CSO sub-grantee “We have to express our concerns with the lack of coordination among donors who fund CSO interventions. There are many duplications. And if donors do not coordinate enough with the relevant government institutions, funds end up being given for activities that are already done or under way.” - Government/public institution representative “There is no official donor coordination, only cases of ad￾hoc coordination at times. We need better information exchange and policy-level coordination among us.” - Representative of a donor working in CSO sector “Systematic solution of co-financing of EU funds for CSOs is an absolute priority”. - CSO sub-grantee 53 interested TV channels). The information is also published at the CCI’s website, the Virtual Parliament, and social networks. CCI monitoring is welcomed as the only holistic overview of the authorities’ work, and as having some influence on citizens and authorities as the most widely known CSO output in BiH. Several KIIs provided examples of how government representatives quote CCI’s reports. A couple of sector leaders/co-leaders noted that they also use CCI’s reports in their own advocacy. On the other hand, a few survey respondents noted that this monitoring is not effective in terms of public outreach, as the citizens are saturated by such information and see it as a simple confirmation of what was already known—while the governments are not deterred by this monitoring, as they do not see that the citizens use such information to reward/punish by their votes. High visibility of this CCI product is confirmed by our online survey, in which around 75 percent of respondents said they are familiar with the reports, with practically all of those believing that the product are useful (Figure 20). Finding 33: CCI’s recent efforts on focusing more on quality rather than quantity is welcomed by all stakeholders; however, all agree that there is space for further improvement in both: (1) analyzing the quality of authorities’ work and actual reasons for progress or lack of progress and (2) a presentation that is more reader-friendly for the public and more dialogue-enticing for the authorities. “CCI monitoring reports are very visible and while there are shortcomings from such quantitative monitoring and politicians complain about that, the fact is that everybody knows about this monitoring.” “We of course follow CCI’s monitoring reports for years now and in several cases we actually used them as background information for our own advocacy efforts.” - CSO sub-grantees Figure 20. Feedback from Sector Network Members/Participants on CCI’s monitoring of Parliaments and Governments 54 CCI has also invested efforts in providing more qualitative analyses of the work of governments and parliaments. In total, CCI reports that seven qualitative analyses were prepared: 1. Work planning as a mechanism to increase accountability, efficiency, and effectiveness of performance of government institutions in BiH 2. Publicity of the work of governments and parliaments in BiH: official web pages of the institutions as an efficient tool for dissemination of information 3. Floods in BiH – Natural disasters and/or institutional inefficiency 4. Public discussions in BiH – From form to essence 5. Enforcement of the Law on State Aid System. 6. Effects of Laws - The unknowns in BiH 7. Pre-accession funds – Utilization of financial instruments as mechanisms for progress of BiH CCI reports that their analyses contain over 400 recommendations, 100 of which have been adopted by the authorities (including those related to increasing transparency by providing information on websites and abolishment of post-mandate 12-month salary for elected and nominated officials at the State and FBiH levels). CCIs are perceived by most CSOs and by all interviewed government representatives as highly negative in how they present their monitoring reports, with populist statements that do not always provide all necessary explanation. This provides a dose of alibi to the authorities for not taking the CCI reporting seriously, and it also hinges on CCI’s desired role of doing not only watchdog activities, but also advocacy and service contract/grant work for the authorities. Only a few KII interviewees knew details on how CCI has improved in qualitative analyses of work of governments and parliaments; and all types of KIIs (including government representatives) as well as survey respondents believe that further improvement is needed to ensure thorough analyses of work quality of work by governments and parliaments and to more effectively present the findings and recommendations to both the decision￾makers (with concrete recommendations for improvement in specific areas and without attacking language) and the public (with more creative out-of￾box visualizations). Other suggestions for improvement include a focus on a specific sector within each report (as an annex) for in-depth quality analyses and context and cooperation with the sector networks, so that the quality of the authorities’ work in those areas can be analyzed in more depth. 4.2 CONCLUSIONS Evaluation Question 1: CSSP’s implicit objective with the networking component was to initiate a broader and richer type of consultation and consensus building process amongst a broader set of stakeholders. This in turn would result in a mindset shift in how CSOs can and do work with key actors within and outside the CSO sector. The first two and a half years of CSSP appear to have created a wide framework for CSOs to work with each other and others on reform efforts. Given the lack of baseline data, it is not possible to say whether this is a net increase in stakeholder partnership and advocacy agenda consensus building in comparison to “CCI monitoring reports need to focus more on quality as opposed to superficial counting, as actual reasons for the lack of adoption of planned legislation/reforms needs to be explained for the most important reforms. There is no need for sensationalism in presenting such reports, in-depth analyses would be more effective, as opposed to current situation which prevents dialogue. There is a space and need in BiH for a watchdog CSO for sure, but it needs to be evidence-based and realistic, otherwise it is counter-productive.” - Representative of a donor working in CSO sector 55 pre-CSSP efforts, but KIIs and survey results indicate that broad alliances have been either created or fortified through CSSP in all networks. Yet getting the right people engaged in useful ways appears to be still be challenging. While a majority of actors in most networks see the utility of further cooperation, diverse sets of people have only partially been utilized in a productive way; many government officials are generally still not sufficiently engaged in discussion; media appear to still be mostly covering campaign issues rather than part of the development of campaigns. These all suggest space for further use of different talents and views that could enhance campaign effectiveness. Variations in network vitality are inevitable given different themes and profiles of sector leadership, but CSSP sub-grant structuring and management processes have contributed to silo thinking in network planning of sector leaders. This has been further challenged by the mixed set of broad and narrow network issues as well as parallel funding of similar or corresponding initiatives appears to have diluted potential of the networking approach. Building on the base that has been established, CSSP management should provide further mechanisms and incentives for consultative thinking and acting. Evaluation Question 2: CSSP’s focus on capacity development and sustainability is beneficial to all CSSP CSOs, and they seized the opportunity to set aside time for internal institutional analyses and strategic discussions. OCA process is taken seriously by all CSSP CSOs and OCA findings have been operationalized in preparation of Sustainability Strategies and Visibility Strategies. The improvements in institutional capacities have already been recorded in all CSSP CSOs. This process allowed organization to focus on themselves, to be self￾critical, and to begin to address organizational practices that needed developing and revising. Having the space to do this with relatively flexible grants and uses of the funding at their own pace allowed ownership. However, this hands-off approach also appears to have resulted in gaps in critical skills learning by most of the organizations, as capacity development improvements are less comprehensive than CSSP design had implicitly intended. Most of the improvement measures identified and implemented within the sub￾grants for capacity building and sustainability focus on internal organization procedures and operational capacities. CSSP implementers’ sub-grant management is mostly administrative as opposed to strategic and advisory and criteria for which tasks can be financed through sub-grans for capacity building and sustainability is not clear to all sub-grantees. Little attention is devoted to building CSO sub-grantees’ sector-specific skill enrichment and there is lack of connections among the three segments of assistance to sub-grantees (establishing networks, advocacy campaigns and monitoring, and capacity building and sustainability). Furthermore, little attention is devoted to enhancing M&E skills all CSSP CSOs, thus there are needs for both improving CSSP M&E reporting, as well as providing broader skills to CSOs for internal M&E practices for other projects and for capacity to monitor and evaluating public policies. The focus on financial sustainability was particularly a wakeup call for many. Having the space to consider how to be more strategic about developing alternative funding strategies has been praised. All CSOs were successful in identifying some potential alternative ways of funding other than foreign donors and there is some evidence of improvement of financial viability of CSSP CSOs already taking place. The learning process for both implementers and sub-grantees in actually doing this has been more fraught. Insufficient custom-made assistance is provided in identifying alternative funding based on CSO type (advocacy CSO, watchdog CSO, service provider to citizens from vulnerable groups, representing interests of associated private sector representatives, service provider to government, or think tank). Perceived insistence by CSSP management of the preferred option of registering a commercial company frustrates a large number of CSOs, as well as a perceived arbitrarily-set uniform mandatory achievement of the quantitate target for decreased dependence on foreign donor funding (30% of core budget being funded from sources other than foreign donors). At the same time, practical usefulness of this indicator to CSSP and USAID management and external stakeholders is also questionable as definition of core budget is not set on the same assumption by all CSOs. Overall, for many, financial sustainability is very much laced with questions 56 of how to find a way that their organization can practically carry this out, and they do not see the CSSP mechanism currently providing such support. Coordination of work of CSSP CSOs is limited to sharing of information of implemented tasks and possible CSSP cross-sector and cross-CSO synergies were not explored to their full potential. CSSP has taken little advantage of peer-learning potential for capacity building among the CSSP CSOs, both in terms of contents related to sectors and in terms of internal operational capacities of organizations. The opportunity to build a coalition of 21 of the most developed CSOs in the country that are CSSP beneficiaries seems to not have been used so far for cross-CSO and cross-sector capacity building, with the exception of the BRANA effort, in which eight of the CSSP CSOs participated. Evaluation Question 3: Steps towards more engagement of citizens in policy making and monitoring government implementation were formulated through campaigns in twelve different areas of policy. We found differing levels of political feasibility, policy development, and CSO and citizen skills in the twelve areas, but regardless civil society efforts to affect change in the twelve different reform areas are making some headway. Most campaigns are still focused on getting the policy actors to agree on ideas and approaches even as a growing number have managed to step into decision-making circles and have some influence in these areas. Factors of success are difficult to gauge due first to the unclear role of the CSSP intervention in achieving such policy influence levels. Many of the issues have been on the agenda for years, and the work of countless actors likely has gone into many of the achievements described by CSSP. In general, there is a discrepancy between the actual level of CSSP networks’ policy influence and the level of influence implied in CSSP reports, with CSSP reporting likely overestimating policy influence. This is partially due to fact that policy change is a complex phenomenon that takes time and can hardly be attributable to a specific intervention. However, it is also due to varying definitions and scope of targeted policies, their political context, and maturity/ripeness and to hands-off sub-grant management and insufficient guidance for campaign design and implementation and overall strategic planning. Campaigns insufficiently focus on areas relevant for accelerating the process of EU integrations, thus deviating in the implementation relative to design of CSSP. At the same time, many of the factors related to manner of implementation appear to give a higher probability for achieving set objectives. Providing some evidence for a particular policy position, using this in an interactive way that brings key players into the discussion, being smart on political timing and possibilities (including how to use and align with EU standardization processes), and doing this in a way that uses the talents of those supporting the cause in innovative ways have certainly helped a number of campaigns to achieve some level of policy influence. Yet missing from many campaigns is a sense of strategic and tactical thinking that provides a backbone for such implementation best practices. Neither CSSP sub-grant structure nor management has helped very much. The sub-grant structure encouraged idea visioning without a lot of structure for clear-eyed assessments of what really can be accomplished during the CSSP campaigns. The CSSP management has primarily focused on operational and administrative guidance so far, as opposed to sector-specific advice and structured and consistent criteria for campaign definition, implementation and reporting. CSSP implementers acknowledge that in the first part of CSSP implementation, the focus was on providing support to sub-grantees in operational grant management (selection of sector partners, basic training sessions, sub-grant applications, establishment of grant management system, etc.), while their plan is to focus more on facilitating information sharing/cooperation among sub-grantees in line with the development of their advocacy campaigns. General CSSP management’s guidance related to policy advocacy means focuses on public outreach and visibility as opposed to building partnership and legitimacy with the authorities. In addition, within this 57 focus on public outreach, civic engagement has been primarily understood and encouraged in the shape of public campaigning in the form of street actions to voice public criticism of authorities with not enough focus given to educating citizens, encouraging volunteering or using the mechanisms at their disposal for civic participation. This in many cases hampers the collaboration with the government and building of the trust and credibility of the CSOs. Overall, campaigns are achieving what they can where they can, but there is potential for improvement. Five campaigns for creating an affirmative work environment for CSOs are being advocated by CPCD (four campaigns: more enabling taxation framework for individual and corporative philanthropy, transparent system of public funding of CSOs, establishment of functional cooperation mechanisms between the governments and CSOs, and promotion and introduction of the Code of CSO Ethics and Good Governance) and CCI (one campaign: establishment of public foundations). These campaigns are focusing on the issues relevant for CSOs, with the transparent system of public funding of CSOs seen as top priority. However, the initiative for establishment of public foundations is seen as unrealistic and premature. In the first two and a half years of CSSP implementation, foundations were laid in advocacy for all campaigns through meetings with representatives from CSOs, government, and business sector, gathering the inputs of a broader sent of CSOs, and preparation of important analyses and recommendations for specific policy proposals. However, so far, authorities have adopted only (partial) recommendations in recently adopted legislation/methodology relevant for transparent system of public funding and more enabling taxation framework for philanthropy at the level of RS. Overall, neither CSSP sub-grantees nor network members/participants have a clear sense of implementation of advocacy campaigns for a better CSO policy environment. Coordination by CSSP implementers so far mostly consisted of sharing invitations to events with other CSSP CSOs, rather than active partnership and advice-seeking. Additional efforts are needed to work more closely with the key officials from relevant government institutions in some of these campaigns. Although official donor coordination of the interventions working in CSP policy environment is missing, CPCD recently successfully agreed on coordination and division of efforts on Agreement on Cooperation between the CoM of BiH and the NGO sector with the parallel EU-funded intervention. An unaddressed priorities within this component of CSSP work is the set-up of a mechanism for co-financing of EU funds for civil society and introduction of legislation on social entrepreneurship. In continuation from predecessor USAID/BiH CSO intervention, CCI has continued to monitor sessions of the parliaments and governments at the Cantonal, Entity and State level in a comprehensive effort. This monitoring conducted by CCI is perceived as the most visible product of CSO sector in BiH. It is the only holistic overview of the authorities’ work and is frequently cited by the officials. The main criticism of CCI’s monitoring reports is that the reports focus on quantitative monitoring of adopted legislation, measures, and inputs by individual politicians without providing sufficient analyses of their quality and the reasons for lack of reforms. Thus, the recent work on CCI on providing some qualitative analyses of specific issues related to work of authorities is a welcome step, however, additional efforts are needed to improve both the quality and the presentation of these reports to provide better visualization of the findings for the public and a more dialogue-enticing recommendations for decision-makers. Broader Conclusions: From a broader perspective, the CSSP intervention is one that has many moving parts and ambitious objectives, with CSSP CSOs including multiple types of organizations with different visions (advocacy CSO, watchdog CSO, service provider to citizens from vulnerable groups, representing interests of associated private sector representatives, service provider to government, or think tank). CSSP design complexity already suggests from its inception that implementers themselves have an extraordinary ability to do many things at the same time from basic grant management, to multi-actor and group facilitation, to technical assistance, to their own advocacy efforts. CCI and CPCD have most experience in doing traditional sub- 58 grant management and their own advocacy, and this shows. Competence in sub-grant oversight is clearly present. Skill and comfort in serving the other roles CSSP requires are less present. Facilitation efforts appear to be rushed or on the side or at least not a clear focus of the CCI and CPCD management efforts. This is demonstrated in the lack of network information sharing, functional idea sharing, and insufficient level of meaningful consultation with CSSP leaders, co-leaders, and network members-participants on their own advocacy efforts. Moreover, USAID and other donor decisions have affected implementers’ management maneuverability. This includes USAID parallel funding of similar initiatives as well as fluctuating levels of directives in relation to the campaigns and results. 4.3 RECOMMENDATIONS As a result of our evaluation, we offer the following lessons and suggestions for adjustment of CSSP in the remaining two years of implementation, as well as for any potential future interventions in civil society sector. The recommended adjustments for the remaining CSSP programming time are mindful of general budgetary and management resources present (to the extent known by the evaluators) and primarily focus on ways to shift, accentuate, and utilize resources available in the most effective ways possible to address CSSP objectives. These include: 1. Continue to encourage further expansion and engagement with a diverse set of network members in particular to government and media representatives. Sector leaders should be encouraged to look again at who is relevant for their sector and reach out formally (if appropriate) to bring additional stakeholders into their networks, especially civil servants and other representatives of relevant government/public institutions at operational level. This should also include natural non￾CSSP CSO partners in a number of networks where such peers are missing. 2. Engage network members more in campaign efforts and substantive campaign implementation Survey and KII feedback suggests there is room for further utilization of network members in campaigns. While this requires careful consideration of best ways to manage these processes, consider how untraditional network members such as media, private sector, and government officials can assist in guiding and facilitating campaign efforts to be more innovative, facilitative and effective in reaching specific campaign objectives. A part of this is to consider how to get more regular and systematic feedback from stakeholders on campaign plans, tactics, and ways they see their inclusion. 3. Further facilitate collaboration across CSSP sectors and CSOs, to take advantage of peer-learning potential for capacity building in terms of the internal operational capacities of organizations. Possible examples would be media relations, communicating with the public in the most reader-friendly and effective format, logistics of running campaigns targeted at citizens, establishing partnership with the authorities, working with the private sector, financial management, as well as thematic areas relevant for all sectors, such as government budgeting, anti￾corruption, monitoring and evaluation etc. In other words, there is no peer-learning and cooperation facilitation that could both build capacities and strengthened the relationship among all CSSP CSO. Implementers need to consider how to facilitate cross learning through both in-person and virtual mechanisms. CCI’s facilitation should be more robust and focus on creating a space for strategic planning and discussion among. Facilitation should also include assisting information sharing amongst sectors with natural linkages either through virtual platforms or implementer-initiated facilitation. 4. Provide meaningful coordination and facilitate collaboration across CSSP networks and CSOs to take advantage of peer-learning potential for thematic expertise capacity building. Possible examples include collaboration among sectors working on employment, economic 59 policies, and education on employability; collaboration between justice and public finance sectors on financing of judiciary; collaboration between sectors for women’s rights and agriculture on women in rural areas; collaboration between health and human rights sectors on marginalized groups’ rights etc. 5. Coordinate closely among USAID same-sector initiatives, and among USAID CSO initiatives. USAID should consider reviewing intended objectives for its same-sector initiatives as well as for broader sector priorities and objectives. Any future funding decisions should be based on how the objectives line up. Any current support should endeavor to craft coordination mechanisms that will enhance the articulated objectives of the individual projects. 6. Consider closer coordination with other donors (particularly EU) at both donor and initiative level. Current initiative implementers should be encouraged to coordinate implementation efforts more closely. 7. In future donor interventions, if network approach is used, select sectors in a more systematic way and align more with sectors related to EU-funding structure. 8. Minimize fragmentation of sub-grants in future donor interventions in CSO sector. Consider sub-grant structure as a single sub-grant with different phases of deliverables. This would encourage the applicants already at the beginning to have a clearer vision for how to design and use the different components to achieve their objectives. It would also provide a clearer link between budget planning and multi-year costs and likely reduce administration costs. 9. Continue providing assistance to sub-grantees to identify and implement measures targeted at improving CSO organizational sustainability. However, provide more hands￾on guidance to focus on identifying needs and providing assistance to strengthen CSOs’ sector-specific thematic expertise, including through technical assistance and peer￾learning. Also communicate more clearly the connections among the three segments of assistance to sub-grantees (establishing networks, advocacy campaigns and monitoring, and capacity building and sustainability). 10. Improve the M&E capacity building of all CSSP CSOs within further building of organizations’ operational capacities, including capacity for CSOs’ own project-based M&E practices, as well as capacity to monitor and evaluate public policies based on evidence-based research. 11. Continue to provide assistance related to identification and operationalization of potential alternative funding sources of CSOs, but in a more custom-made approach— to take into account different CSO types and visions (advocacy CSO, watchdog CSO, service provider to citizens from vulnerable groups, representing interests of associated private sector representatives, service provider to government, or think tank). Clarify that the quantitate target for decreased dependence on foreign donor funding (30% of core budget being funded from sources other than foreign donors) is not mandatory for continued implementation of CSSP. Consider possible additional analyses of achievement measures by this indicator by analyzing different assumptions made by CSOs about the level of their core budget. 12. Consider how to more fully use CPCD’s Resource Center to serve as coordination mechanism for the CSSP activity. Link Resource Center activities and efforts with CSSP objectives more clearly. 13. Improve CSSP reporting to more realistically capture the actual influence, especially from the perspective of different phases of each policy’s maturity/ripeness. Consider tracking campaigns according to policy influence types to better understand actual influence and manage expectations. Adjust the reporting to be finer grained and realistic. Whether CSSP decides to use the policy influence topology used in this evaluation or other ways to track policy influences, a system should be considered which will assist both implementers and grantees to better track progress and link activities/outputs to intended policy influence outcomes. 60 14. Share success factors of campaigning among networks. As part of the functional cross learning, policy influence success factors highlighted in this report should be shared amongst CSSP sectors. This can be as basic as a good practices checklist and a summary of how specific campaigns exemplified different good practices. Efforts should be made to frame the practices around different types of campaigns (and not only a few specific successful examples that are not replicable in most cases) in order to give others a sense of how more technical and less ‘popular’ campaigns have managed these processes. This should particularly include an emphasis on media and outreach tactics. 15. Consider providing technical assistance (from other network partners or outside) to networks, to refine the design of campaign strategies (where wanted) and introduce/improve evidence-based policy proposals. Beyond best practices, some sectors might seek out additional assistance for strategizing specific campaigns or tactics within campaigns. Expertise from within CSSP from the different sectors should be emphasized, for example, CIN and Media Center could be consulted about different media outreach approaches, CRP about ways to educate the public on technical issues, and Akcija about innovative citizen engagement. CSSP should also make outside technical assistance available in areas of policy advocacy design, through mentoring and one-on-one on-the-job training of CSSO CSOs. External experts could be brought in to train CSOs on preparing actionable and evidence-based policy proposals. Flexibility in budget (re)allocations from other sub-grants to campaigns could provide funding for this. 16. Include EU-integration dimension in network campaigns where possible, and coordinate with the DEI. 17. Modulate the emphasis on public protest campaigning where not appropriate. CSSP should continue to encourage campaigns to be active and have clear tactics for their advocacy efforts Emphasis should be on effectiveness and the good practices noted in the success factors. Highly visible public campaigns should be framed as one of these approaches, not the preferred one. 18. Refine campaign management tools to assist partners to track development and implementation of campaigns. More recent emphasis on working with partners to develop more concrete and coherent campaign planning should be further developed and expanded to ensure that campaigns have a clear and targeted focus. 19. In future donor interventions, consider grant structure around more articulated strategic design including theory of change hypothesis, logical framework, or other tools. Whether the design is called a campaign, policy advocacy efforts, research advocacy or other, future efforts should be structured around a clearer design model. This would require organizations to articulate in more detail what specifically they want to achieve and how they think their interventions will lead to this. Such a framework will aid all sides to be clear in attention and to also better understand why an approach may or may not be effective. 20. Consider specific skill building in monitoring implementation of existing policies and encourage the CSOs to more systematically and actively use existing and planned mechanisms for civic participation in governance. At this stage, only a few sectors have undertaken systematic monitoring of different policy implementation as part of their campaigns. It is likely that more will consider this option or should consider this option during and beyond CSSP. Therefore, consider adding specific monitoring skill building options for the different sectors. Encourage the CSOs to more systematically and actively use existing and planned mechanisms for civic participation in governance, such as e-consultations, EU-fund programming, and submitting policy proposals during legislation adoption procedures. 21. Continue work on current campaigns for creating an affirmative work environment for CSOs, prioritizing further improvement in increasing the transparency of the current system of public funding of CSOs. Activities on the establishment of public foundations should be abolished. Also, invigorate network participation in strategizing and advocating for CSO policy environment campaigns. Consider including additional campaigns on setting-up a mechanism for 61 co-financing of EU funds for CSOs and social entrepreneurship legislation. Work more closely with the relevant authorities at operational level in all campaigns and continue increasing cooperation with other donor interventions in these areas. 22. Continue the work on monitoring sessions of the parliaments and governments, but with increased analyses of the quality of work of these authorities. Improve presentation and communication of findings of this work so that it is in a more reader-friendly format for the citizens and more dialogue-enticing for decision-makers. 23. From a broader perspective, consider the following: - Manage expectations and breadth of a single intervention, consider distinguishing design by different objectives and different types of CSOs (service providers to citizens, service providers to government, general think tanks, advocacy CSOs, watchdog CSOs). - Promote different types of civic engagement to focus on educating citizens and encouraging volunteering. 62 ANNEXES 63 ANNEX I EVALUATION STATEMENT OF WORK USAID/BiH Democracy Office STATEMENT OF WORK Performance Evaluation Civil Society Sustainability Project (CSSP) PURPOSE OF THE EVALUATION The purpose of this performance evaluation of the Civil Society Sustainability Project (CSSP) is three-fold: (1) assess CSSP’s progress toward stated objectives; (2) identify obstacles and opportunities to activity implementation; and (3) provide recommendations for activity adjustments. The Mission will use evaluation findings, conclusions, and recommendations to reassess the role of CSSP in strengthening and maintaining the capacity of BIH civil society to influence the creation and implementation of public policies that are of interest to citizens. Other USG stakeholders, including USAID/W and U.S. Embassy, will use evaluation findings, conclusions, and recommendations to better understand the USAID-funded civil society activities in BiH. The implementing partner will have an opportunity to learn about their strengths and areas for improvement. Other stakeholders, including the BiH governing institutions, civil society organizations (CSOs), the European Commission’s (EC) Delegation to BiH and other international development donors and partners, may also benefit from USAID’s contribution to the marketplace of public knowledge on the most recent development efforts in strengthening BiH civil society. PROGRAM INFORMATION Activity/Project Name Civil Society Sustainability Project (CSSP) Contractor Centers for Civic Initiatives (CCI) in partnership with Center for Civil Society Promotion (CPCD) Cooperative Agreement/Contract # AID-168-A-13-00006 Total Estimated Cost (TEC) $9,150,000 Life of Project/Activity September 1, 2013 - August 30, 2018 64 Active Geographic Regions Entire BiH Mission Development Objective (DO) IR 1.2 Increased citizen participation in governance BACKGROUND The main purpose of the USAID/Civic Society Sustainability Project (CSSP) is to strengthen and maintain the capacity of BIH civil society to influence the creation and implementation of public policies that are of interest to citizens. USAID/CSSP is focused on three areas: 1. Strengthening Civil Society Organizations’ (CSO) legitimacy in order to be true representatives of stakeholders rather than acting independently and without any real connection to the citizens; 2. Strengthening competence and professionalism of CSO experts in their respective sectors in order to earn the respect of the stakeholders and authorities with their own quality of work; and 3. Increasing the CSOs’ impact on public sector policies as well as the impact on the CSOs’ working conditions, so that CSOs can act as partners in the policy processes by the authorities, especially in oncoming negotiations with European Union (EU). One part of the activity is led by USAID’s partners CCI and CPCD. These two organizations lead the campaigns for strengthening an enabling CSO environment in BIH, strengthening the selected CSOs’ capacities, and conduct a comprehensive monitoring of the BiH government performance. Second and largest part of the activity assists selected CSOs to better engage and join forces with other key stakeholders in business, government and media through sector-based network development. They also receive assistance to strengthen their internal capacities and organizational structures in order to ensure sustainability and financial viability after the life of the project. Finally, the activity assists these groups to engage in policy oversight and development related to political, social and economic reforms essential for EU integration, with a broader goal of increasing government accountability. The activity supports 19 local CSOs to create 12 sector networks that include citizens and representatives from business, academia, media and government. The networks have identified priority issues for advocacy in each of the selected sectors and organized some effective campaigns. The 12 sectors are: 1) anti-corruption; 2) employment and labor markets; 3) economic policy; 4) education; 5) health care; 6) human rights of marginalized groups; 7) women's rights; 8) agriculture and rural development; 9) culture; 10) public finances; 11) environment protection and energy efficiency; and 12) the justice sector. Additional segment of USAID/CSSP was added in March 2015 as a response to 2014 floods in BiH. This additional component supports Network BRANA in collecting, analyzing, disseminating information regarding realization of the funds donated for flood recovery process. 65 PROBLEM STATEMENTS USAID CSSP addresses five major challenges (listed below) that hinder civil society effectiveness and its impact on policy development, oversight and implementation. By strengthening the capacity and viability of selected CSOs and enhancing partnerships between these organizations and other stakeholders, this activity creates momentum for sustained long￾term citizen engagement in the oversight of public policy decision-making and implementation. 1. Lack of government engagement with civil society The government does not view civil society as a major stakeholder, and it does not consider civil society as a legitimate representative of constituents, which, if mobilized, could hold them accountable. Unfortunately, this opinion is not completely unwarranted. Years after the war, many CSOs continue to rely solely or heavily on external donor funding, and are accountable primarily to those funders, not their beneficiaries. The government does not understand civil society’s role in advancing democratic governance. The EU accession process promotes inclusion of CSOs in the integration process and calls for their active participation in decision-making. But institutional mechanisms to promote collaboration and constructive dialogue between government and civil society are for the most part missing, or are improperly implemented or underutilized. 2. Inability of CSOs to influence public discourse Civil society is unable to influence public discourse on substantive issues. CSOs do not know how to engage effectively with media or establish a common agenda with BiH citizens, which then reduces public confidence in civil society’s ability to advance democratic reforms. For their part, citizens are often reluctant to engage in civic activities. This is due to various factors: A lack of understanding of civil society and its role, as well as their own role, in democratic governance; politicization of some CSOs; confusion about the political system and how to bring about change; and, as is often the case, the belief that their efforts will have no effect. 3. Lack of professionalism of CSOs To effectively engage with government and other key stakeholders over the long term, civil society must obtain new skills and operate more professionally. Even the strongest CSOs in BiH lack solid internal systems and succession planning to ensure the viability and transparency of their organizations. Often the vision and mission of organizations do not correspond with their competencies. To raise their level of professionalism, CSOs need to focus more on areas of comparative advantage. CSOs generally do not identify their own strongest competencies (conducting research or training, drafting policy papers, organizing citizen action, issuing grants, or media relations) and build on these. Instead, they offer all possible services called for in a donor solicitation to obtain funding. 66 4. Sustainability of the civil society sector Sustainability of the broader sector is at risk primarily because of its over-dependence on international funding. This affects public perceptions – of the government, the private sector, and citizens – of civil society. It has also resulted in distorted accountability as CSOs see themselves as accountable primarily to their donors. Furthermore, as noted above, external funding has shaped the civil society agenda, de-emphasizing civil society’s role in aggregating demand from citizens and, instead, placing priority on its ability to execute specified projects, manage funds and produce clear outputs. Overdependence on donors is particularly worrisome given that international donors are leaving BiH and the funding for civil society is diminishing. Local funding sources remain limited, due largely to the poor economic, legal and fiscal environment that is not supportive of individual and corporate philanthropy. Government funding for provision of social services is on the rise, but with considerable transparency and accountability issues. CSO grant recipients are often affiliated with the ruling government and the selection criteria for grants and contracts is not transparent. And, as is the case in most countries, CSOs working on democracy and governance are rarely able to obtain funding from government sources. The previously mentioned disconnect between citizens and CSOs threatens the sustainability of the sector, as citizens do not believe many CSOs are legitimate or represent their interests. 5. Lack of reliable and unbiased information The effectiveness of civil society, particularly in policy advocacy and oversight, depends on access to reliable and unbiased information, which is becoming more difficult to obtain in BiH because the media is increasingly polarized and politicized. The problem is compounded by poor coordination between CSOs and the media. Development Hypothesis USAID/CSSP address the lack of civic engagement in policy development, implementation and oversight, with a broader goal of increasing government accountability. Development Hypothesis: Two primary development hypotheses underlie the proposed strategic and operational approach for this new activity. They are interrelated, yet unique in their own right. This project hypothesizes that if leading CSOs: I. Strengthen their capacities and organizational structures to ensure sustainability and financial viability and II. Effectively partner with a wide array of stakeholder groups, Then we can expect civil society to be more effective in influencing and overseeing development and implementation of government policy. Furthermore, the hypothesis is that strong and effective civil society that advocates for issues relevant to broader constituencies will lead towards increase civic engagement in decision￾making processes. 67 Link To USAID Strategy And Assistance Objective USAID/CSSP directly contributes to the goal stated in USAID’s Country Development Cooperation Strategy for Bosnia and Herzegovina 2012-2016, which is: “BiH is a stable, prosperous, democratic, market-oriented, multi-ethnic country progressing towards Euro Atlantic integration,” and it supports Development Objective 1: A more participatory, inclusive democratic society, IR 1.2: Increased citizen participation in governance, and sub IR 1.2.1: Civil society effectively informs, represents and advocates on behalf of citizens. Activities Objective 1: Increased partnership and consensus between different stakeholders in support of selected advocacy agenda Activity 1.1 Raising awareness for developing partnerships and networks Activity 1.2 Small grants Activity 1.3 Strengthening CSO relations with media Activity 1.4 Strengthening CSO and private sector partnerships Activity 1.5 Strengthening relationships between CSOs and citizens Activity 1.6 Strengthening relationships between CSOs and governments Objective 2: Strengthened capacity and viability of core civil society partners and of the broader civil society sector Activity 2.1 Creating an affirmative work environment Activity 2.2 Strengthening capacities of Sector Leaders Objective 3: Increased engagement of civil society and citizens in development and government monitoring and oversight at local, entity and state level Activity 3.1 Grant support Activity 3.2 Monitoring government performance Activity 3.3 Citizen Engagement 68 The illustrative hierarchy of the project results is as follows: Purpose: CIVIL SOCIETY INFLUENCES GOVERNMENT POLICY Civil Society is representative and credible 1. Civil society forms consensus in support of advocacy agendas Civil Society effectively partners with multiple stakeholder groups to identify needed policy changes Citizens informed and engaged to support policy changes Media recognize importance of policy changes 2. CSO sustainability and financial viability is strengthened Capacity of core CSO partners strengthened to articulate needs of citizens and lead other groups Core CSO partners develop effective communication strategies Core CSO partners develop effective sustainability strategies 3. Civil society effectively engaged in government oversight Civil Society provides credible policy inputs to government Civil Society conducts monitoring of policy implementation Civil Society conducts advocacy campaigns for policy change 69 EVALUATION QUESTIONS The evaluation will be used by USAID/BiH to examine the results of the first two years of the activity and the effects of the program to the overall civil society development in the country. 1. To what extent have interventions under CSSP increased CSO partnership and advocacy agenda consensus building with other stakeholders (government, media, and private sector)? 2. To what extent have interventions under CSSP strengthened the capacity and viability of selected CSOs in the 12 sectors? 3. To what extent have interventions under CSSP increased engagement of civil society and citizens in governance and what was achieved within the 12 selected reform areas? 4. Which performance related intervention factors could be adjusted to address intervention concerns and align the CSSP activity with its articulated objective for 2018 and, based on CSSP’s achievements and challenges, what are the lessons learned for future interventions in this area? EVALUATION DESIGN AND METHODOLOGY EVALUATION QUESTIONS SUGGESTED PRIMARY DATA SOURCES RESEARCH DESIGN 1. To what extent have interventions under CSSP increased CSO partnership and advocacy agenda consensus building with other stakeholders (government, media, and private sector)? Sub-questions: Which types of assistance have been most effective in achieving these partnerships with each of the type of stakeholders? Which have been less effective? 1) CSSP reports 2) Semi-structured KII of CSSP implementers, partner CSOs, network members/participants, non￾partner CSOs 3) Online mini-survey of network members/participants Mixed method triangulation Summary findings will be disaggregated by sectors 2: To what extent have interventions under CSSP strengthened the capacity and financial viability of selected CSOs in the 12 sectors? Sub questions: How was the assistance (in particular grant mechanisms) to the CSOs in the 12 sectors designed and implemented? Which types of interventions have been most and least 1) CSSP reports 2) Semi-structured KII of CSSP implementers, partner CSOs, network members/participants, technical assistance providers, other government officials, and donors 3) Online mini-survey of network members/participants Mixed method triangulation Comparison of OCA and strategy related documents in relation to work plans in the 12 areas and in relation to OCA/USAID markers for capacity development Analysis of KII and mini-survey findings explaining efforts levels 70 effective in meeting the needs of the CSOs? reached and why intervention was or was not effective; compared with review of training materials/methods 3: To what extent have interventions under CSSP increased engagement of civil society and citizens in governance and what was achieved within the 12 selected reform areas? Sub-questions: Which types of policy influence are being achieved and what are some of the factors contributing to this? What were the success factors in terms of providing relevant and credible policy inputs to government and governments’ acceptance of those inputs? 1) CSSP reports 2) Semi-structured KII of CSSP implementers, partner CSOs, network members/participants, non￾partner CSOs, other government officials, and donors 3) Online mini-survey of network members/participants Mixed method triangulation Classification of achieved ‘impacts’ by 3 levels of policy influence for all 12 areas, examination of 3 ‘most successful’ areas comparison/analysis of factors affecting this performance, and verification/further data collection in KIIs and online survey 4: Which performance related intervention factors could be adjusted to address intervention concerns and align the CSSP activity with its articulated objective for 2018, and based on CSSP’s achievements and challenges, what are the lessons learned for future interventions in this area? 1) CSSP reports 2) Semi-structured KII of CSSP implementers, partner CSOs, network members/participants, technical assistance providers, non￾partner CSOs, other government officials, and donors 3) Online mini-survey of network members/participants Mixed method triangulation Examination of CSSP intervention efforts per 3 objectives and summarizing recommendations for adjustment of the remaining CSSP implementation period. Examination of remaining needs and lessons learned for future interventions beyond CSSP in strengthening civil society and citizen engagement 71 DELIVERABLES AND REPORTING REQUIREMENTS TENTATIVE DATES TASKS AND DELIVERABLES May 4 – 16 Desk review of CSSP materials and other secondary documents; drafting of Evaluation Plan May 16 Submission of draft Evaluation Plan to USAID May 16-23 Logistical preparation, scheduling for KIIs, preparing mini-survey May 23 – June 10 Data collection through KIIs (primary field work to be undertaken from May 23-June 3 by the whole Evaluation Team, with additional interviews taking place from June 6-10 depending on identified needs for additional data collection after the primary field work) and online mini-survey May 23-24 Evaluation Team working meeting to assign tasks; further desk review, final scheduling for KIIs; tentative meetings with CSSP implementers May 25 Initial meeting with USAID (in-briefing) June 3 Second meeting with USAID (mid-term briefing) with USAID June 6-10 Finalize data collection and wrap up KIIs; interview transcribing; data analysis June 13 – 20 Continued interview transcribing and data analysis; drafting evaluation findings June 21 or 22 Presentation of summary findings and recommendations to USAID (out-briefing) June 24 Submission of Draft Evaluation Report to USAID July 11 Start with the review of consolidated USAID comments on Draft Evaluation Report (expected to be received 10 working days after Draft Evaluation Report submission) and subsequent revision of the Report July 18 Submission of the revised Draft Evaluation Report to USAID (to be completed 5 working days after receiving USAID comments) July 25 Start with the review of consolidated USAID comments on revised Draft Evaluation Report (expected to be received 5 working days after revised Draft Evaluation Report submission) and subsequent revision of the Report August 8 Submission of Final Evaluation Report to USAID (to be completed 10 working days after receiving USAID comments) 72 EVALUATION TEAM COMPOSITION USAID/BiH anticipates that a four-person team would be adequate for conducting this mid-term performance evaluation: Evaluation Team Leader: This person must have strong team management skills, and sufficient experience in designing and/or conducting performance evaluations of international development activities. ET Leader(s) must have good knowledge of USAID Evaluation Policy and evaluation reporting requirements. Prior experience in monitoring and evaluation for Democracy, Human Rights, or Governance (DRG) programming required, experience in BiH or other relevant Balkans states strongly preferred. Knowledge of civil society development contexts is preferred. Excellent communication, both verbal and written, skills and experience managing performance evaluations of large USAID activities are desirable. Evaluation Team Members: The Contractor must assign up to three team members from BiH (or other relevant Balkans states as accepted by USAID/BiH) that collectively demonstrate strong understanding of data collection and analysis methodologies; substantial experience with international donor programs; deep knowledge of BiH civil society development; DRG programming more generally; good knowledge of USAID Evaluation Policy and evaluation reporting requirements. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy (www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf). USAID/BiH will approve of all personnel. FINAL REPORT FORMAT The evaluation final report should include an executive summary; introduction; background of the local context and the projects being evaluated; the main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable) as described here. The report should be formatted according to the evaluation report template. The executive summary should be 3–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • Evaluation SOW 73 • All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • Sources of information, properly identified and listed; and • Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest. The contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.1 • The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. • Evaluation reports shall address all evaluation questions included in the SOW. • The evaluation report should include the SOW as an annex. All modifications to the SOW— whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline—need to be agreed upon in writing by the AOR/COR. • The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 74 OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in an electronic file in easily readable format agreed upon with the Contracting Officer’s Representative (COR). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. 75 ANNEX II EVALUATION MATRIX EVALUATION QUESTIONS DATA SOURCES RESEARCH DESIGN 1. To what extent have interventions under CSSP increased CSO partnership and advocacy agenda consensus building with other stakeholders (government, media, and private sector)? Sub-questions: Which types of assistance have been most effective in achieving these partnerships with each of the type of stakeholders? Which have been less effective? 1) CSSP documentation 2) Semi-structured KII of CSSP implementers, partner CSOs, network members/participants, non-partner CSOs 3) Online survey of network members/participants Mixed method triangulation 2: To what extent have interventions under CSSP strengthened the capacity and financial viability of selected CSOs in the 12 sectors? Sub questions: How was the assistance (in particular grant mechanisms) to the CSOs in the 12 sectors designed and implemented? Which types of interventions have been most and least effective in meeting the needs of the CSOs? 1) CSSP documentation 2) Semi-structured KII of CSSP implementers, partner CSOs, network members/participants, technical assistance providers, other government officials, and donors 3) Online survey of network members/participants Mixed method triangulation 3: To what extent have interventions under CSSP increased engagement of civil society and citizens in governance and what was achieved within the 12 selected reform areas? Sub-questions: Which types of policy influence are being achieved and what are some of the factors contributing to this? What were the success factors in terms of providing relevant and credible policy inputs to government and governments’ acceptance of those inputs? 1) CSSP documentation 2) Semi-structured KII of CSSP implementers, partner CSOs, network members/participants, non-partner CSOs, other government officials, and donors 3) Online mini-survey of network members/participants Mixed method triangulation Classification of achieved ‘impacts’ by 3 levels of policy influence for all 12 areas, examination of success factors, and further data collection in KIIs and online survey 4: Which performance related intervention factors could be adjusted to address intervention concerns and align the CSSP activity with its articulated objective for 2018, and based on CSSP’s achievements and challenges, what are the lessons learned for future interventions in this area? 1) CSSP documentation 2) Semi-structured KII of CSSP implementers, partner CSOs, network members/participants, technical assistance providers, non-partner CSOs, other government officials, and donors 3) Online survey of network members/participants Mixed method triangulation 76 ANNEX III DOCUMENTS REVIEWED 1. Civic Advocacy Partnership Project II (August 2008 – August 2013), final report 2. Mid-term Evaluation of the Civic Advocacy Partnership Project (CAPP II) (August 23, 2011) 3. Evaluation of Civil Society Programs in Bosnia and Herzegovina, final report (March 10, 2008) 4. Politics & Ideas Online Training Course to Strengthen MEL on Policy Influence in Central and Eastern Europe, which was produced by Vanesa Weyrauch with collaboration of Kristie Evenson and Dena Lomofsky http://www.politicsandideas.org/?page_id=2303 5. Fred Carden, (2009). Knowledge to Policy: Making the Most of Development Research. New Delhi: Sage and Ottawa: IDRC. http://idlbnc.idrc.ca/dspace/bitstream/10625/37706/1/IDL-37706.pdf 6. Evert Lindquist (2001). Discerning Policy Influence: Framework for a Strategic Evaluation of IDRC￾Supported Research. University of Victoria: School of Public Administration. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.497.5165&rep=rep1&type=pdf 7. Table: Stakeholder Government Representatives which are not members of Sectorial Networks 8. Table: Stakeholder Members of Sectorial Networks 9. Monitoring Matrix on Enabling Environment for Civil Society Development Regional Report for 2014 (May 2015) 10. Strengthened Partnership between CSOs and other stakeholders 11. CSSP Partners List 12. Sector Member Contacts for Survey 13. CSSP Annual Report: Project Year 1 (September 2013 - September 2014) 14. CSSP Annual Report: Project Year 2 (October 2014 - September 2015) 15. CSSP Work Plan: Project Year 1 (September 2013-September 2014) 16. CSSP Work Plan: Project Year 2 (October 2014-September 2015) 17. CSSP Work Plan: Project Year 3 (October 2015-September 2016) 18. CSSP Y1Q1 Quarterly Report 1 (September 1-December 31, 2013) 19. CSSP Y1Q2 Quarterly Report 2 (January-March 2014) 20. CSSP Y1Q3 Quarterly Report 3 (April-June 2014) 21. CSSP Y2Q1 Quarterly Report 5 (October-December 2014) 22. CSSP Y3Q1 Quarterly Report 9 (October 1 -December 31, 2015) 23. CSSP Y3Q2 Quarterly Report 10 (January - March 2016) 24. CSSP Q10 M&E Report 25. CSSP Q10 Media Coverage Report 26. Mid-term Organizational Capacity Assessment Draft Report (May 2016) 27. USAID/Civic Society Sustainability Project (CSSP) Survey data sets 12 April, 2015 28. USAID/Civic Society Sustainability Project (CSSP) Program Description 29. USAID/Civic Society Sustainability Project (CSSP) Program Description (Recipient’s Radical Transparency in Flood Recovery) 30. USAID/Civic Society Sustainability Project (CSSP) Statement of Work (SoW) 77 ANNEX IV DATA COLLECTION INSTRUMENT – KEY INFORMANT SEMI-STRUCTURED INTERVIEW PROTOCOLS Interviews will be held with nine different stakeholder groups: 1. USAID/BiH (2 interviews) 2. CSSP implementers (referred to as Prime Implementer and Prime Partner in CSSP Program Description) CCI and CPCD (3 interviews) 3. CSSP CSO Sector Leaders/Co-Leaders (19) 4. Sector network members/participants (14 interviews) 5. Technical assistance providers (2 interviews) 6. General government officials relevant for CSO legislation, policy environment, and general views of the program (2 interviews, however 4 of the interviewed sector network members from government/public institution were from the institutions relevant for this aspect of CSSP’s work so they also provided information) 7. Other (non-USAID) donors providing assistance to CSOs (2 interviews) 8. CSOs which are not CSSP beneficiaries (4, 1 of which used to be CSSP CSP Sector Co-Leader but no longer is) 9. Other donor CSO intervention (1 interview) We here present detailed semi-structured interview guides for the two main stakeholder groups – CSSP CSO Sector Leaders/Co-Leaders and sector network members/participants. Interviews with the remaining stakeholder groups were based on the guides presented here, but adjusted for the specificities of each stakeholder group’s relation with CSSP. 78 CSSP EVALUATION INTERVIEW INSTRUCTIONS The interview guides are intended to serve as semi-structured guides for conversations with key stakeholders of CSSP. Do not read the questions or probes word for word. Instead, adapt the wording to match the phrasing used by the respondent and ask only those questions which have not been already addressed by the interviewees during earlier part of the interview. Take notes on key terms or phrases used by the respondents that may be helpful in coding the interview data. Ask for clarification and definitions as needed. Familiarize yourself with the interview protocol guides in advance of your meeting. Skip questions that are not relevant given the interviewee specificities. Highlight the questions you will prioritize if the respondent’s time is limited. Be respectful of the respondent’s time and keep the interview to the agreed length of time. Follow up by phone or email for more information as needed. In addition:  Take notes during the discussion. To ensure we accurately report what is discussed during the interview, we will record this session as well.  As necessary, tailor all questions to fit the individual stakeholders’ relationship with CSSP.  Keep the discussion under sixty minutes.  The Evaluation Team will ensure that the information shared through these interviews remain strictly confidential. 79 USAID/BiH CSSP EVALUATION INTERVIEW GUIDE CSSP CSO Partners Thank you for agreeing to speak with us today. Our names are will be taking notes while we talk. With your permission, we would also like to record this session so that we can refer to the audio to clarify our notes later if necessary. Do we have your permission to begin recording? Do you have any questions before we start? I BACKGROUND AND CONTEXT First I’d like to start off with some general questions to give us a little bit background.  How would you describe your organization? o PROBE: What is its vision and history?  How many employees/associates are there in your organization?  What are the main sectors/areas in which your organization targets? o PROBE: Who are your main counterparts (including government institutions, media, and private sector)? o PROBE: How does your organization interact with citizens?  What is your current position in the organization and role in the work related to CSSP? II CSSP IMPLEMENTATION PROCESS Next, we have a set of questions that related to your experience with USAID and CSSP. General Cooperation with the CSSP Implementing Partners  How did you start your cooperation with the CSSP? o PROBE: How did you first hear about CSSP? o PROBE: Have you worked with CSSP Implementers (CCI and CPCD) before CSSP? o PROBE: Have you received USAID and/or other donor assistance before CSSP? 80  Why did your organization apply to CSSP open call for Sector Leaders/Co-Leaders? o PROBE: How have your first heard about the CSSP’s Open Call for Sector Leaders?  If you are one of the seven CSOs that are sub-grantee partners (rather than a single Sector Leader), describe how you divide the work with your partnering sub-grantee?  What type of CSSP grants have you applied for and received so far, what was the process, and how effective was it in your opinion? Thematic Partnership and Networking Efforts  Describe for me your vision for a partnership network in your sector? o PROBE: What types of engagement did you initiate to bring these actors together? o PROBE: Which ways of engagement seemed to be most effective? Why? o PROBE: Which actors did you think are particularly key for developing/expanding this network?  Beyond the grant support, describe how CSSP was involved in this process. o PROBE: Were there mechanisms for networking that they provided or facilitation/introduction (including to foster relationship with the media and private sector)? o PROBE: If your organization is a sector lead, how effective were CSSP’s initial trainings of the Sector Leads in stakeholder management and building and strengthening networks in your opinion? o PROBE: To what extent were these tools appropriate to your needs and plans? o PROBE: In which other CSSP activities relevant for partnership and consensus building you participated (e.g. trainings, coordination meetings, communication events, events targeting partnership building with media, private sector, citizens, and/or government), and in your opinion what was the quality of these activities were (what were their outcomes)?  Are CSSP activities/events general in line with the needs of your organization and the sector/area in which your organizations targets? o PROBE: Why/why not? o PROBE: Please provide examples.  Please describe the process of creating a priority list in your sector network? o PROBE: How was citizens' input taken into account? o PROBE: What was CSSP Implementing Partners' role in the process of creating a priority list in your sector network?  What types of lessons have you learned from working with key sector network stakeholders?  How do you see the network further developing/expanding? o PROBE: Which factors do you think will influence this (financial support, successful outcomes, key actors, other)? Development of Policy Advocacy Strategies and Campaigns  Tell me about your organization’s decision to pursue policy advocacy strategies and campaigns in the particular issue area you work in. o PROBE: What was the process implemented by your organization to ensure that the identified issue area reflect the real problems of BiH citizens? o PROBE: What have or do you hope to specifically achieve with your current efforts?  Are there any particular challenges that have come up or which you did not anticipate? How did you address these? o PROBE: Capacity, resource, or environment related?  What do you anticipate being able to achieve by the end of the CSSP support?  How do you envision continuing efforts in this area after 2018? 81 o PROBE: Which partners do you anticipate continuing this work with you? CSSP Capacity Development and Sustainability Assistance  Tell me about your OCA diagnostic process o PROBE: To what extent did you agree with initial findings? o PROBE: Specifically for finance and communications, what are key areas you are working on?  What type of capacity development activities have you participated in through CSSP? Which have been more or less useful? o PROBE: To what extent did trainings provide you with the guidance you needed? o PROBE: What was your experience with the TA mentor?  To what extent have you been able to implement this learning? o PROBE: How do you anticipate your financial profile will look in 2 years? o PROBE: And your public profile amongst citizens in BiH?  In general, how do you define sustainability from the perspective of your organization?  Would you say that your organization’s sustainability prospects increased in the last couple of years? o PROBE: Why/why not? Provide examples.  From your perspective, how much have CSSP-related activities contributed to sustainability prospects of your organization/sector network and for the civil society in BiH in general? o PROBE: Why/why not? Provide examples. III CIVIL SOCIETY POLICY ENVIRONMENT Now, I’d like to ask you about CSSP’s efforts to strengthen the sector and policy change related to civil society.  Firstly, if you are a part of a CSO, have you used the Resource Center for CSOs? Tell me about the experience.  Next, have you noticed any changes in regulations and policies relevant to civil society in recent years? o PROBE: If yes, to what extent has CSSP contributed to those?  Are you aware of any CSSP advocacy campaigns implemented to improve the legal framework relevant for sustainability of CSOs in BiH, with focus on legislation on individual/corporate philanthropy, social entrepreneurship, self-regulating CSOs' mechanisms, and relations of CSOs with the authorities? o PROBE: If yes, what changes have you noticed?  Are CSSP activities in the area of policy environment in line with the general civil society needs? Why or why not? o PROBE: In your opinion, can CSSP planned/implemented tasks (campaigns to improve CSOs’ working environments through advocating for transparent public financing, creating a CSO Code of Conduct, and advocating a change in tax law to support philanthropy, development of social entrepreneurship) be effective in creating an affirmative work environment for CSOs? o PROBE: In your opinion, can CSSP planned tasks for creation of a public foundation for supporting CSOs be successful? Why or why not?  In your opinion, what are the remaining challenges and what can CSSP or any future interventions do to improve the environment? 82 IV GENERAL CSSP ACHIEVEMENTS AND CHALLENGES Next, let’s discuss some questions that related to additional general CSSP achievements and challenges from your perspective. Note to interviewer – In this section, only ask questions about areas covered here that have not come up in the interviewees’ answers to previous questions.  How do you perceive the quality of CSSP monitoring of the work of BiH authorities and what were the effects of those activities? i. PROBE: Did this contribute to greater transparency in the decision-making process in BiH? ii. PROBE: Did this adequately serve to continuously assess government's work against resources spent and respond to the needs of citizens and the state of EU reforms?  What were the most useful parts and resulting general achievements of CSSP activities in addressing the following main problems: i) lack of government engagement with civil society, ii) inability of CSOs to influence public discourse, iii)lack of professionalism of CSOs, iv) sustainability of the civil society sector, and v) lack of reliable and unbiased information? o PROBE: What general improvements/new activities did your organization/sector network make as a result of CSSP? o PROBE: To what extent do you see the CSSP efforts bringing together the energies and capacities and actors to influence public policy in BiH? Note to interviewer – Get examples whenever possible. As a reminder, the following items are some of the mentioned areas to which CSSP is expected to contribute: − strengthening overall CSO legitimacy in BiH and their responsibility and connection to the citizens (improved image of CSOs in the eyes of BiH citizens and increased motivation of citizens to act); − strengthening competence and professionalism of CSO experts in their sectors and CSO experts’ earning the respect of the stakeholders and authorities with their own quality of work; − improving CSO’s working conditions and policy environment; − increasing the CSOs’ influence on public sector policies and perceptions of the authorities that CSOs are their partners in the policy processes by the authorities (further development of participatory democracy in BiH and increased understanding of the authorities of the benefits of cooperation with CSOs); − increasing CSOs’ influence related in particular to reforms related to the EU integration process; − responsibility providing new knowledge in networking in order to increase the power of advocacy as a path toward the wide CSO network formation at the national level; − improving active public communication to increase government accountability and responsibility; − increasing advocacy for real problems of BiH citizens; strengthening selected CSOs’ integrity and their mutual networking; − strengthening partnerships and consensus of CSSP-supported CSOs; − strengthening CSOs’ accountability and recognizing mutual interests and responsibilities of CSOs rather than focusing on CSOs’ financing; − increased openness of government institutions and increased stakeholder participation in the budgeting process, etc.  What were the least useful parts and challenges encountered how have these challenges been addressed? 83 V RECOMMENDATIONS AND LESSON LEARNT Now that we have an understanding of the CSSP implementation, we have some questions about the recommendations and lessons learned from your perspective.  From your perspective, what are the lessons learned from CSSP and recommendations for remainder of the CSSP intervention? o PROBE: Ask about implementation manner (including small grant mechanism) as well as inputs and general strategy of the program.  What are the main needs of civil society organizations for any future donor interventions? VI CONCLUSION  Is there anything you would like to share that we have not asked or which you wish to further discuss? 84 USAID/BiH CSSP EVALUATION INTERVIEW GUIDE CSSP Sector Network Members Thank you for agreeing to speak with us today. Our names are will be taking notes while we talk. With your permission, we would also like to record this session so that we can refer to the audio to clarify our notes later if necessary. Do we have your permission to begin recording? Do you have any questions before we start? I BACKGROUND AND CONTEXT First I’d like to start off with some general questions to give us a little bit background. 1. How would you describe your organization? o PROBE: What is its vision and history? 2. How many employees/associates are there in your organization? II CSSP IMPLEMENTATION PROCESS Next, we have a set of questions that related to your experience with USAID and CSSP Sector Networks. General Cooperation with the CSSP Sector Network 3. How did you start your cooperation with the Sector Leader CSO in your network? o PROBE: How did you first hear about CSSP? o PROBE: Have you worked with the Sector Leader CSO before CSSP? o PROBE: Have you received USAID and/or other donor assistance before CSSP? Thematic Partnership and Networking Efforts 4. Describe for me your vision for a partnership network in your sector? o PROBE: What are you trying to achieve with the network in your issue area? o Which ways of engagement are most effective in your opinion? Why? o PROBE: Which actors did you think are particularly key for developing/expanding this network? 85 5. What type of CSSP sector activities have you participated in and what is your opinion about their quality? o PROBE: Are these activities in line with your/your organization’s/your sector network’s needs? Why/why not? o PROBE: Please provide examples. 6. Please describe the process of creating a priority list in your sector network? o PROBE: How was citizens' input taken into account? 7. What types of lessons have you learned from working within the key sector network stakeholders? 8. How do you see the network further developing/expanding? o PROBE: Which factors do you think will influence this (financial support, successful outcomes, key actors, other)? o PROBE: What do you think you will be able to influence in your issue area? III CIVIL SOCIETY POLICY ENVIRONMENT Now, I’d like to ask you about CSSP’s efforts on policy change related to civil society. 9. Have you noticed any changes in regulations and policies relevant to civil society in recent years? o PROBE: If yes, to what extent has CSSP contributed to those? 10.Are you aware of any CSSP advocacy campaigns implemented to improve the legal framework relevant for sustainability of CSOs in BiH, with focus on legislation on individual/corporate philanthropy, social entrepreneurship, self-regulating CSOs' mechanisms, and relations of CSOs with the authorities? o PROBE: If yes, what changes have you noticed and how have these changes affected your organization/sector network? Are CSSP activities in the area of policy environment were in line with the general civil society needs? Pease provide examples. 11. In your opinion, can CSSP planned/implemented tasks (campaigns to improve CSOs’ working environments through advocating for transparent public financing, creating a CSO Code of Conduct, and advocating a change in tax law to support philanthropy, development of social entrepreneurship) be effective in creating an affirmative work environment for CSOs? 12. If you are a part of a CSO, have you used Resource Center for CSOs and if yes, how helpful was it for you? 13. In your opinion, can CSSP planned tasks for creation of public foundation for supporting CSOs be successful, and what are success factors? 14. In your opinion, what are the remaining challenges and what can CSSP or any future interventions do to improve the environment? o PROBE: Provide examples. IV GENERAL CSSP ACHIEVEMENTS AND CHALLENGES Next, let’s discuss some questions that related to additional general CSSP achievements and challenges from your perspective. Note to interviewer – In this section, only ask questions about areas covered here that have not come up in the interviewees’ answers to previous questions. 15. How do you perceive the quality of CSSP monitoring of the work of BiH authorities and what were the effects of those activities? o PROBE: Did this contribute to greater transparency in the decision-making process in BiH? o PROBE: Did this adequately serve to continuously assess government's work against resources spent and respond to the needs of citizens and the state of EU reforms? 86 16. Besides CSSP-funded activities related to your sector network, have you participated in any other CSSP activities/events and how useful these events were in your opinion? o PROBE: Please provide examples. 17. What were the most useful parts and resulting general achievements of CSSP activities in addressing the following main problems: i) lack of government engagement with civil society, ii) inability of CSOs to influence public discourse, iii)lack of professionalism of CSOs, iv) sustainability of the civil society sector, and v) lack of reliable and unbiased information? o PROBE: What general improvements/new activities did your organization/sector network make as a result of CSSP? o PROBE: To what extent do you see the CSSP efforts bringing together the energies and capacities and actors to influence public policy in BiH? Note to interviewer – Get examples whenever possible. As a reminder, the following items are some of the mentioned areas to which CSSP is expected to contribute: − strengthening overall CSO legitimacy in BiH and their responsibility and connection to the citizens (improved image of CSOs in the eyes of BiH citizens and increased motivation of citizens to act); − strengthening competence and professionalism of CSO experts in their sectors and CSO experts’ earning the respect of the stakeholders and authorities with their own quality of work; − improving CSO’s working conditions and policy environment; − increasing the CSOs’ influence on public sector policies and perceptions of the authorities that CSOs are their partners in the policy processes by the authorities (further development of participatory democracy in BiH and increased understanding of the authorities of the benefits of cooperation with CSOs); − increasing CSOs’ influence related in particular to reforms related to the EU integration process; − responsibility providing new knowledge in networking in order to increase the power of advocacy as a path toward the wide CSO network formation at the national level; − improving active public communication to increase government accountability and responsibility; − increasing advocacy for real problems of BiH citizens; strengthening selected CSOs’ integrity and their mutual networking; − strengthening partnerships and consensus of CSSP-supported CSOs; − strengthening CSOs’ accountability and recognizing mutual interests and responsibilities of CSOs rather than focusing on CSOs’ financing; − increased openness of government institutions and increased stakeholder participation in the budgeting process, etc. 18. What were the least useful parts and challenges encountered how have these challenges been addressed? V RECOMMENDATIONS AND LESSON LEARNT Now that we have an understanding of the CSSP implementation, we have some questions about the recommendations and lessons learned from your perspective. 19. From your perspective, what are the lessons learned from CSSP and recommendations for remainder of the CSSP intervention? 20. What are the main needs of civil society organizations for any future donor interventions? 87 VI CONCLUSION 21. Is there anything you would like to share that we have not asked or which you wish to further discuss? 88 ANNEX V DATA COLLECTION INSTRUMENT – ONLINE SURVEY OF CSSP SECTOR NETWORK MEMBERS/PARTICIPANTS CSSP Mid-Term Performance Evaluation Mini-Survey of CSSP Sector Network Members Approach and Rationale Social network analysis (SNA) provides a way to examine network members as well as the connections between members (Taylor, Watley & Coffman, 2015)30. CSSP has as an objective of the activity to “increase partnership and consensus between different stakeholders supporting advocacy agendas” through the formation and support of twelve thematic networks (CSSP Sector Networks). While a traditional SNA that reveals detailed mapping of association patterns is not the specific focus of the Mid￾Term Performance Evaluation of USAID/BiH Civil Society Sustainability Project (CSSP), we use this mini-survey to capture some of the detail of such patterns using the elements of SNA approach to provide insights into the strength of CSSP network partnerships and their potential beyond CSSP. Specifically, the survey will capture a snapshot of the functioning of the Sector Networks and their vibrancy, which provide evidence for better understanding the role and limits they could play beyond CSSP. Description of Sector Network 1. Level of previous engagement with key members (previous network or not) 2. Types and frequency of engagement Network Functioning: 3. Level of innovation (of whichever kind in the network) 4. Major ways of managing and communicating (through leads) each other directly 5. Members’ involvement in all levels: identification of problem to strategizing approach to campaigning Network vibrancy 6. Level of benefit from/to the network 7. Sustainability: Further plans for working with issues/actors In addition to providing inputs to analyses of CSSP Sector Networks using the elements of the SNA approach, this survey also collects general feedback from CSSP sector network members on other non-network related CSSP activities. 30 Taylor, M., Whatley, A. & Coffman, J. (2015). Network evaluation in practice: Approaches and applications. The Foundation Review, 7(2), 22-37. 89 SURVEY OF SECTOR NETWORK MEMBERS OF THE USAID/BiH CIVIL SOCIETY SUSTAINABILITY PROJECT (CSSP) Dear CSSP Sector Network members, This Survey has 19 questions and is estimated to take you around 20 minutes to complete. Your input is very important as it will inform USAID’s external Mid-Term Performance Evaluation of the CSSP, which will provide recommendations to USAID/BiH on possible adjustments of the CSSP in the remaining 26 months of project implementation, as well as general recommendations for possible future donor interventions in civil society sector. We greatly appreciate you taking the time to answer this Mini-Survey! Your answers are confidential and you will not be identified by name in any report. The Survey asks for your name only for the purpose of our own verification and better understanding of the context of your responses. IF YOU HAVE ANY QUESTIONS OR REQUIRE ANY CLARIFICATIONS ON THE SURVEY OR SPECIFIC QUESTIONS, PLEASE CONTACT USAID’S MONITORING AND EVALUATION PROJECT (MEASURE-BiH), WHICH IS CONDUCTING THIS EVALUATION AT ncarsimamovic@measurebih.com. _________________________________________________________________________________ 1. In which of the CSSP sector networks do you participate (select all that apply): a) Culture (led by CSO Action and Foundation Media Center) b) Employment and labor market (led by CSO Institute for Youth Development KULT) c) Economic development policies (led by CSO Independent Bureau for Development (NBR) and Association for Entrepreneurship and Business (LINK)) d) Public financial management (led by CSO Public Interest Advocacy Center (CPI)) e) Education (led by CSO Youth Communication Center (OKC)) f) Health care (led by CSO Partnership for Health (PfH) and Initiative and Civil Action (ICVA)) g) Environmental protection and energy efficiency (led by CSO Center for Development and Support (CRP)) h) Agriculture and Rural Development (led by CSO Agency for Cooperation, Education and Development [ACED]) i) Justice sector (led by CSO Forum of Citizens of Tuzla and Helsinki Committee for Human Rights in Republika Srpska) j) Human rights - marginalized groups (led by CSO Foundation for Social Inclusion (FSU) and Initiative for better and humane Inclusion (IBHI)) k) Women's rights (led by CSO Zene zenama and Zene Trnova) l) Fight against corruption (led by CSO Transparency International BiH and Center for Investigative Journalism (CIN)) 2. Please provide your name and the name of your institution/organization: __________ In what capacity are you a member of CSSP network(s)? a) A member of an NGO which gather citizens (e.g. citizen association) b) A member of an NGO which gathers private sector representatives c) A representative of a private sector company d) A representatives of government/parliament/public sector institution e) Media f) Individual 90 3. Have you been involved in projects or other collaboration with the key partners of the network (including Sector Lead/Co-Lead CSO(s) or other network members) prior to CSSP (note that CSSP sector network started operation in mid-2014)? Yes No Please explain briefly:____________________ 4. How often and in which ways CSSP Sector Network leading CSO(s) communicate with the Sector Network membership (select one options of frequency for each row)? Daily Weekly Monthly Quarterly Semi-Annually Annually Never Face-to face network meetings Face-to face meetings of a more narrow working groups formed out of Sector members Inputs of Sector Members gathered by email by the Sector Lead/Co-lead CSO Inputs of Sector Members gathered by phone by the Sector Lead/Co-lead CSO 5. How often and in which ways you personally communicate within the CSSP Sector Network (select one options of frequency for each line)? Daily Weekly Monthly Quarterly Semi-Annually Annually Never Face-to face Sector working meetings Face-to face meetings of a more narrow working groups formed out of Sector members Inputs to Sector Members sent by email to Sector Lead/Co-lead CSO Inputs to Sector Members sent by phone to Sector Lead/Co-lead CSO Directly contacting and cooperating with other Sector Network Members 91 6. To what extent does the network have (select one options of frequency for each row)? Never Rarely Sometimes Often Always Appropriate and well balanced participation by members from relevant stakeholder groups for this sector (e.g. NGOs with citizen membership, NGOs with private sector representatives, representative of government/parliament/public sector institution, media representatives, individual citizens) Opportunity of members to provide inputs and influence sector priority list and chosen advocacy campaigns Members’ participation in sector advocacy campaigns Well-coordinated actions 7. Chosen priorities for advocacy campaigns for each sector are shown below. a. Culture: i) Request for urgent establishment of a single Creative Europe Program Desk, ii) Solving the position of the BiH National Museum, iii) Developing a Strategy for culture at the level of Sarajevo Canton, and initiating development of strategic documents at all other administrative levels (cantons, entities, State), iv) Advocating for depoliticization of managerial functions in the public institutions of culture as part of solving identified need for reform of public sector of culture in BiH. b. Employment and labor market: i) Distributing budget funds intended for programs and active measures to combat unemployment in BiH at the entity level of authority in a rational and efficient manner, ii) Government institutions design and implement employment and economic incentives programs in line with the recommendations for efficient, transparent and effective performance, iii) The employers are exempt from paying social insurance contributions on employees’ salaries during the first year of employment, and pay State value added tax when collecting debts based on which the VAT was accrued. c. Economic Development Policies: i) Acceleration of the business registration process in FBiH, ii) Lowering the number of para-fiscal levies, iii) Improving concrete support to businesses to counter flood damage, iv) Designing policies for development of concrete models and instruments of support to business (e.g. business incubators, issuance of construction and usage permit, credit guarantee funds, etc.). d. Public finance management: i) Amending the Law on Budgets in FBiH, the Law on budgetary system of RS and Law on budget of Brcko District in order to introduce Budget for citizens in the process of budget preparation as a mandatory budget document, ii) Enactment of the amendments to the Law on audit of BiH institutions that would relate to introduction of concrete measures and sanctions aimed at ensuring better implementation of audit findings, iii) Improving implementation of Regulation on rules for participation of interested public in the process of drafting BiH Federation legal documents and other regulations by the FBiH Ministry of Finance. 92 e. Education: i) lack of demonstration classes, ii) inadequate syllabuses/curriculums, iii) non￾compliance of enrollment policy with trends on the labor market (by advocating for these areas within the Education Development Strategy of Republika Srpska 2016 – 2020, Youth policy of Republika Srpska 2016 – 2020, Strategic directions for development of education in BiH Federation 2012 – 2020 (revision), Strategic framework for development of education in BiH 2015 – 2025, Strategic directions for development of higher education in BiH 2016 – 2026). f. Health Care Sector: i) fragmented health care systems in BiH, without institutional responsibility, mechanisms of coordination between different parts of the system and without obligations to harmonize legislative documents and accessibility of health care services in different parts of BiH (entity, canton, region), which leads to unequal access to health care and to discrimination of citizens, who have different access to health care services depending on their place of residence, ii) institutions, primarily Health Insurance Institutes, withhold even mandatory information from the citizens, which leads to many exclusions from the health care system, iii) citizens are not adequately informed about their rights and obligations in the health care system. g. Environment protection and energy efficiency: i) Campaign to increase political will and competencies of the local BiH authorities (municipalities, cities, cantons) for “greening” the local energy sector through planning and realization of the EE&RES measures, ii) A campaign toward state and entity government institutions to clearly define indicative strategic objectives and to adopt accompanying action plan to increase energy efficiency and to use renewable energy sources, iii) Campaign to motivate and educate the citizens to implement concrete EE&RES measures in their households and business operations. h. Agriculture and Rural Development: i) enactment of the Law on subsidies in agriculture and rural development in Republika Srpska, ii) enactment of the changes to the Law on agricultural advisory services in the BiH Federation with regard to putting the rural development under competence of the advisors or solving this issue through adoption of the Advisory Services’ Work Strategy, iii) enactment of a policy that regulates the area of organic food production. i. Justice: i) ensure constitutional and legal assumption for independence of judicial institutions, ii) improve process of appointing judges and prosecutors, and strengthening the role of the HJPC BiH as an independent institution, iii) ensure financial independence and autonomy of judiciary in BiH. j. Human rights – marginalized groups: i) to introduce minimum of social security in both entities and to activate social protection, ii) to improve extra-institutional support to marginalized groups, iii) to create conditions for amendments to the legal framework for improvement of social entrepreneurship, iv) changes to the BiH Election Law in terms of electoral rights of national minorities. k. Women’s rights: i) Violence against women and domestic violence – general social condemnation, ii) Discrimination of women in labor market – mobbing and sexual harassment at work, iii) Gender -budgets for solving women’s problems, iv) Improving position of women in rural areas. l. Fight against corruption: i) lack of transparency in public procurement process and lack of anti￾corruption mechanisms in the legislative framework, ii) limited circle of persons and institutions that are subject of the law on conflict of interests and mild sanctions that do not incite adherence to the law, iii) lack of control over asset declaration forms. For your sector(s), please note to what extent you agree with the following statements (select one option for each row)? Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree Chosen priority advocacy campaigns reflect well real priorities in this sector Chosen priority advocacy campaigns can realistically be addressed through activities 93 within the CSSP during project duration Chosen priority advocacy campaigns are actionable and well-focused BiH authorities would not introduce (or improve) these policies without CSOs’ engagement 8. I have been involved in: Yes No Identifying sector priorities Deciding on advocacy approach Advocacy and campaigning activities 9. Through membership in the CSSP Sector Network(s), I benefited from: Yes No Network media exposure Better access to policy makers Learning about effective network strategies for addressing policy issue Improving knowledge about specific issues areas in my sector(s) Closer/increased cooperation with other network members Other:________________ 10. My contributions to the Sector Network are: Yes No General expertise in issues in my sector(s) Ideas and inputs in selection of specific priorities General contacts of relevant stakeholders in the sector Access to policy makers relevant for the sector 94 Reputational weights Assistance in organization/implementation of campaigns 11. How often are media representatives included in the work of your network and advocacy for the sector priorities (select one option for each row)? Never Rarely Sometimes Often Always Media representatives cover events/identified issues of the Sector Network Media representatives participate in the actual issue—based work of the Sector Network as members 12. For your sector(s), what level of influence in policy change can be achieved by different ways of implementing advocacy/public campaigns (select one option for each line)? No influence Minor influence Moderate influence Major influence Public outreach campaign targeting citizens through media Advocacy efforts targeted at government representatives through direct work with them 13. My general satisfaction with the work of Sector Network(s) is: High Medium Low 14. Do you plan to continue cooperation with network members on concrete issues after CSSP finishes? Yes No Why or why not __________________________________________________ 15. What recommendations do you have for future Sector Network work and support? __________________________________________________ 16. More broadly, CSSP implementers CCI and CPCD (not related to the work of CSSP Sector Networks) also work on advocacy campaigns for a better institutional, legal and taxation framework for development 95 of civil society in BiH. For each of the advocacy campaigns in this area listed below, please select all that apply: I am aware of the activities and campaign of CPCD/CCI in this area Progress is made in this area since the start of CSSP in 2013 Campaigns for a more enabling taxation framework for strengthening individual and corporative philanthropy Advocacy campaign for a transparent system of public funding for CSOs in BiH Advocacy campaign for establishment of functional cooperation mechanisms between the governments and CSOs in BiH Campaign for establishment of public foundations in BiH Campaign for promotion and introduction of the Code of Ethics and good governance for the CSOs in BiH Please write comments, if any______________________ 17. In your opinion, what should be the level of priority for the work of CPCD and CCI within campaigns for a better institutional, legal and taxation framework for development of civil society in each of the following campaigns:_ High priority Medium priority Low priority Campaigns for a more enabling taxation framework for strengthening individual and corporative philanthropy Advocacy campaign for a transparent system of public funding for CSOs in BiH Advocacy campaign for establishment of functional cooperation mechanisms between the governments and CSOs in BiH Campaign for establishment of public foundations in BiH Campaign for promotion and introduction of the Code of Ethics and good governance for the CSOs in BiH 96 In your opinion, is there some other priority on which focus should be in terms of campaigns for a better institutional, legal and taxation framework for development of civil society and what is it?__________________ 18. CSSP implementer CCI (also not related to specific Sector Networks) also continuously monitor sessions of the parliaments and governments at the cantonal, entity and State level aimed at informing the citizens about performance and results of their representatives at the State, entity and cantonal level. Select if Yes I am aware of CCI’s monitoring reports CCI monitoring reports are useful for informing the citizens about performance and results of their representatives CCI monitoring reports are effective in influencing policy makers Please explain briefly to what extent you believe that CCI monitoring reports are effective in influencing the decision-makers and citizens and why? 19. In your opinion, what should be priorities in the work of CSSP in the next two years? Please also provide any other comments you have: ____________________________ THANK YOU FOR YOUR CONTRIBUTION! 97 ANNEX VI LIST OF KEY INFORMANT INTERVIEWEES # Organization/Institution Type of Stakeholder Stakeholder's Location Date of the Interview 2 Zene zenama CSO Sector Leader/Co-Leader Sarajevo 5/23/2016 3 ICVA CSO Sector Leader/Co-Leader Sarajevo 5/23/2016 6 IBHI CSO Sector Leader/Co-Leader Sarajevo 5/24/2016 7 CPI CSO Sector Leader/Co-Leader Sarajevo 5/24/2016 8 FSU CSO Sector Leader/Co-Leader Sarajevo 5/24/2016 12 Transparency International CSO Sector Leader/Co-Leader Banja Luka 5/26/2016 13 ARD (former ACED) CSO Sector Leader/Co-Leader Banja Luka 5/26/2016 16 CRP CSO Sector Leader/Co-Leader Tuzla 5/27/2016 18 Citizens' Forum CSO Sector Leader/Co-Leader Tuzla 5/27/2016 20 KULT CSO Sector Leader/Co-Leader Sarajevo 5/30/2016 21 Partnership for Health CSO Sector Leader/Co-Leader Sarajevo 5/30/2016 22 Zene Trnova CSO Sector Leader/Co-Leader Sarajevo 5/30/2016 24 CIN CSO Sector Leader/Co-Leader Sarajevo 5/30/2016 27 CRMA CSO Sector Leader/Co-Leader Sarajevo 5/31/2016 36 Akcija and Media Center CSO Sector Leader/Co-Leader Sarajevo 6/2/2016 37 Media Center CSO Sector Leader/Co-Leader Sarajevo 6/2/2016 38 NBR CSO Sector Leader/Co-Leader Modrica 6/3/2016 40 OKC CSO Sector Leader/Co-Leader Banja Luka 6/3/2016 44 LINK CSO Sector Leader/Co-Leader Mostar 6/7/2016 30 Ministry of Justice of BiH Government /public institution Sarajevo 6/1/2016 45 BiH Directorate for European Integration Government /public institution Sarajevo 6/8/2016 1 CCI and CPCD Implementer Sarajevo 5/23/2016 19 CCI Implementer Tuzla 5/27/2016 32 CPCD Implementer Sarajevo 6/1/2016 11 Foundation for Local Democracy Non CSSP CSO (originally CSO Sector Leader/Co-Leader) Sarajevo 5/25/2016 26 XY Non-CSSP CSO Sarajevo 5/31/2016 29 GEA Non-CSSP CSO Banja Luka 5/31/2016 31 Foreign Policy Initiative Non-CSSP CSO Sarajevo 6/1/2016 47 European Union Delegation Other donor Sarajevo 6/10/2016 49 SWISS Agency for Development and Cooperation BiH Other donor Sarajevo 6/13/2016 28 TACSO Other donor CSO intervention Sarajevo 5/31/2016 4 Association for Democratic Initiative (ADI) Sector Network member (CSO) Sarajevo 5/24/2016 98 5 Svjetlo Sector Network member (CSO) Sarajevo 5/24/2016 23 CURE Sector Network member (CSO) Sarajevo 5/30/2016 41 MDP Inicijative Sector Network member (CSO) Doboj 6/6/2016 42 Justicia Sector Network member (CSO) Srebrenik 6/6/2016 14 Ministry of Family, Youth and Sports of RS Sector Network member (government /public institution) Banja Luka 5/26/2016 33 Cabinet of the Prime Minister of FBiH Sector Network member (government /public institution) Sarajevo 6/2/2016 39 Ministry of Finance of FBiH Sector Network member (government /public institution) Sarajevo 6/3/2016 48 BiH Parliament Sector Network member (government /public institution) Sarajevo 6/10/2016 10 Center for Education of Judges and Prosecutors in FBiH Sector Network member (government /public institution) Sarajevo 5/25/2016 15 PREDA Sector Network member (government /public institution) Prijedor 5/26/2016 17 Ministry of Education, Science, Culture and Sport in Tuzla Canton Sector Network member (government /public institution) Tuzla 5/27/2016 25 FBiH Development Planning Bureau Sector Network member (government /public institution) Sarajevo 5/31/2016 34 Aida Vezic Sector Network member (individual) Sarajevo 6/2/2016 43 John Snow Technical Assistance Provider Split 6/6/2016 46 Faculty of Economics of University of Split - Niksa Aliferovic and Maja Cukusic, Ljiljana Najev Technical Assistance Provider Split 6/9/2016 9 Selma Sijercic USAID/BiH Sarajevo 5/25/2016 35 Selma Sijercic USAID/BiH Sarajevo 6/2/2016 99 ANNEX VII CSSP OCA AREAS AND INDICATORS Area 1 – Strategic management Area 7 – Governance and structure Indicator 1 .1 Organizational mission and vision Indicator 7.1 Organizational structure and its functionality Indicator 1.2 Strategic approach in planning Indicator 7.2 Functionality of the Governing board Indicator 1.3 Integrated annual planning Indicator 7.3 Criteria for forming and establishing of the Governing Board Indicator 1.4 Financial sustainability Indicator 7.4 Regulating rights, responsibilities and structural communication. Indicator 1.5 Impact assessment Indicator 7.5 Participatory decision making Area 2 – Relations with membership and target groups Area 8 – Human resources Indicator 2.1 Managing membership and target group/beneficiaries relations Indicator 8.1 Procedures and practices for selection and employment Indicator 2.2 Needs identification Indicator 8.2 Job systematization and description Indicator 2.3 Outreach and communication mechanisms Indicator 8.3 Staff salaries and rewards Area 3 – Project management Indicator 8.4 Administrative and management capacities of the staff members Indicator 3.1 Project planning capacities Indicator 8.5 Building staff expertise Indicator 3.2 Capacities for project implementation Indicator 8.6 Performance management Indicator 3.3 Writing effective project proposals Indicator 8.7 Mechanism for attracting and managing volunteers Indicator 3.4 Project M&E practices Area 9 – Finance and administration Area 4 – Advocacy capacities and monitoring of public policies implementation Indicator 9.1 Organizational policy and written procedures for financial and administrative work Indicator 4.1 Advocacy expertise and skills Indicator 9.2 Archiving and storing documents and data Indicator 4.2 Image/position of the organization in terms of advocacy Indicator 9.3 Technical, space and IT equipping Indicator 4.3 Capacity for monitoring and evaluating the public policies implementation (Watch dog) Indicator 9.4 Travel policies and procedures Area 5 – Service delivery and beneficiaries care Indicator 9.5 Procurement policies Indicator 5.1 Capacity for service delivery Indicator 9.7 Financial planning and budgeting Indicator 5.2 Quality assurance in service delivery Indicator 9.6 System for financial reporting Indicator 5.3 Client satisfaction survey Indicator 9.8 Financial audits Indicator 5.4 Service commercializing Area 10 – Media and public relations Indicator 5.5 Image/position of the organization in terms of service delivery Indicator 10.1 Public relations strategy Area 6 – Cooperation, partnerships and networking Indicator 10.2 Public relations capacities Indicator 6.1 Cooperation with public institutions on a national level Indicator 10.3 Web page Indicator 6.2 Cooperation on a local level Indicator 10.4 Other tools for e-communication with public and program work Indicator 6.3 Networking and coalition-building with NGOs Indicator 6.4: International cooperation and partnership Indicator 6.5 Cooperation with organization from other sectors (businesses, educational and research centers etc.) 100 ANNEX VIII PURPOSE OF GRANT FOR CAPACITY BUILDING AND SUSTAINABILITY SECTOR CSO SECTOR LEADER/CO￾LEADER PURPOSE OF GRANT FOR CAPACITY BUILDING AND SUSTAINABILITY CULTURE AKCIJA (Sarajevo) and MEDIA CENTAR (Sarajevo) 1) Development of management and organizational structure, including decision-making, administration, financial management, mechanisms of internal control, human resource management, and project management; 2) Increasing the financial sustainability through a systemic approach of diversifying funding sources, including: development of services to be offered for a fee, seeking grants from BiH authorities, offering service contracts expert services, etc.; 3) Development of a strategic approach to cooperation, partnership and networking with key stakeholders in the cultural sector; 4) The development of thematic areas and programs that address the key needs of the cultural sector in BiH and the region; 5) Improving the operational capacity (technical equipment / resources, office space, etc.). 6) Development of research and advocacy capacity of the organization, 7) Development of publishing activities; 8) internal and external communications through the implementation of the Communication Strategy. ZENE ZENAMA (Sarajevo) and ZENE TRNOVA (East Sarajevo) Zene Zenama: 1) The simplification of the management and organizational structure 2) Creating new and strenghtening the existing internal rules and procedures for building management capacity for planning, development, implementation, and monitoring of projects; 3) Strengthening the capacity of existing staff and volunteers and functions of human resource management; 4) Development of commercial services through the offering psycho-social assistance to women; 5) Branding and building reputation; and 6) Active development of cooperation and partnership with the media, government institutions, the business sector and the academia. Zene Trnova: 1) Improvement of the management and organizational structure; 2) Strenghtening human resources management; 3) Increasing the capacity of staff to manage the project cycle; 4) Development of a new set of services that the organization will provide for a fee; 4) Building financial and administrative capacity; 5) Introduction of social entrepreneurship through the establishment of agricultural cooperatives; 6) Building partnerships and networking with individuals and organizations from all three sectors of society; 7) Internal and external communication through the implementation of the Communication Strategy. AGRICULTURE AND RURAL DEVELOPMENT ARD (Banja Luka, formerly part of ACED) 1) Strengthening the internal capacity by strengthening human and organizational capacity and securing financial and technical resources for operations; 2) Diversification of funding sources through social entrepreneurship, the development and implementation of quality programs and projects and offering services on a commercial basis; 3) Positioning as the leading organization in the field of rural development through networking and connecting with all relevant stakeholders in the sector on the domestic and international levels; 4) Improvement of internal and external communications through implementation of the Communications Strategy. PUBLIC FINANCE CPI (Sarajevo) 1) Promotion of human resources in areas such as strategic planning, management, administration and finance; 2) Development of research and advocacy capacity of the CPI; 3) Building of organizational and institutional capacity through the development and introduction of missing internal procedures and regulations; 3) Strengthening the financial and technical conditions; 4) Strengthening the financial sustainability through diversification of funding sources, the creation of new commercial products and services (consulting on the budget for the citizens, consultation of civil society organizations and individuals about the creation and funding of sectoral public policies, training of local governments on improving financial management, maintenance, processing and distribution data, video productions, TV series about the financial literacy of citizens, Concept Store), proactive fundraising focused on domestic donors / sponsors; 5) Networking and building partnerships at national and international level through capacity building and strengthening the impact of existing local and regional networks and forming / joining a new and developing closer links with decision-makers; 6) Improvement of internal and external communications through implementation of the Communications Strategy. WOMEN RIGHTS 101 SECTOR CSO SECTOR LEADER/CO￾LEADER PURPOSE OF GRANT FOR CAPACITY BUILDING AND SUSTAINABILITY ENVIRON-MENT PROTECTION ANC ENERGY EFFICIENCY CRP (Tuzla) 1) Strengthening the human capacity through the acquisition of advanced knowledge, skills and appropriate recognized certificatss by existing employees, as well as the necessary rejuvenation of the organization and strengthening of attracting new, young professionals with expertise in the relevant areas of environmental protection and energy efficiency, 2) Strengthening of financial and technical conditions of work, including the purchase of equipment and IT technology necessary to expand the scope of existing and new types of services and products to current and planned categories of users, 3) Organizational and institutional strengthening through (a) the improvement of existing and creation of missing procedures for all areas of work; (b) an effective physical and electronic storage and updating of all institutional, program and material and financial documents, in a way that ensures its quick availability; 4) Strengthening of capacities of Enkon (Energy Consulting), the newly formed company as the main mechanism for the commercialization of services in order to increase the financial sustainability of CRP; 5) Engaging additional human resources (temporary and permanent CRP employees and external experts including members of network); 6) Development of new and improvement of existing products and services in the field of environmental protection and energy efficiency; 7) Strengthening the financiall and technical conditions for the realization of high-quality commercial services offered by Enkon; 8) Improvement of internal and external communications through implementation of the Communications Strategy. CULTURE Citizens' Forum (Tuzla) and Helsinki Committee for Human Righths (Bijeljina) Citizens' Forum Tuzla: 1) An integrated approach to strategic and operational planning through the adoption of long-term and medium-term strategy and annual plans; 2) Building relationships with customers and other stakeholders; 3) Developing functional management and organizational structure through the adoption of appropriate internal policies, procedures and other documents; 4) Improving the policy and practice of human resource management, including professional capacity building of staff and volunteers, particularly in the area of project cycle management and quality management projects and services; 5) Strengthening the financial and administrative capacity; 6) The development and marketing of new, "commercial" services; 7) Diversification of funding sources through the development of annual plans of raising funds and the introduction of alternative sources of financing, primarily grants of government institutions at all levels of government, providing services to government institutions, individual philanthropy, etc .; 8) Building partnerships and networking; and 9) Implementation of the Communications Strategy. Helsinki Committee: 1) Strengthening the advocacy position and capacity in key areas of the organization by defining advocacy strategy, identifying five key areas of advocacy, regular monitoring and analysis of the situation in selected areas of advocacy; 2) Implementation of the Communications Strategy including the establishment of a system of regular communication with the public, target groups, stakeholders, donors, and increasing the visibility of the organization; 3) Monitoring and evaluation of achieved results and impact; 4) Capacity building of personnel in the field of advocacy and lobbying; 5) The development of new services and projects that the organization will offer for a fee; 6) Diversification of funding sources through the development of a strategic approach to fundraising organizations and the development of a strategy of fundraising and staff training; 7) Construction of the system of monitoring and assessing the performance of staff and introducing a system of rewards. HUMAN RIGHTS - MARGINALIZED GROUPS IBHI (Sarajevo) and FSU (Sarajevo) FSU: 1) Strengthening human resources; 2) Strengthening the reputation of the organization, 3) Creating and strengthening cross-sector partnerships, 4) Advocacy and policy influence; 5) Project financing of civil society organizations in BiH; 6) The introduction of social entrepreneurship and the improvement of market-oriented cooperation, 7) Improvement of fundraising and 8) Social awareness on issues of marginalized groups, social inclusion and poverty reduction. IBHI: 1) Capacity building in the area of preparing proposals for EU funds, 2) Regional networking for joint applications to public calls and implementation of projects, 3) Scientific and applied research in social policy, civil society and social development, 4) Human resources - systematization of jobs, wages and benefits, performance management, attracting volunteers 5) Development of social policy in BiH: strengthening and capacity building of stakeholders in civil society, 6) Establishment of mechanisms for attracting non-project funds (with marketing, administration and ICT support) and IPA funds and creating fundraising strategies and 7) Advocacy and influence on public social policies aimed at reducing poverty and exclusion. 102 SECTOR CSO SECTOR LEADER/CO￾LEADER PURPOSE OF GRANT FOR CAPACITY BUILDING AND SUSTAINABILITY EMPLOYMENT AND LABOR MARKET KULT (Sarajevo) 1) Strengthening the capacity for the establishment of a framework for the implementation of participatory strategic and operational planning 2) Preparing necessary rulebooks for operations of the organization and its social enterprise, 3) Capacity building of staff and job systematization, 4) Establishing an effective organizational structure, 5) Quality assurance in the provision of services, 6) Establishment of a mechanism for attracting and engaging volunteers, 7) Strengthening the capacity for providing services and high-quality and accountble implementation of program and project activities, 8) Management Institute has recognized the benefits of starting a social enterprise to ensure income for the implementation of projects for which securing other funds is difficult and fir financing administrative and operational costs of KULT, 9) Commercialization of its services as a specific form of social entrepreneurship, 10) Identifying and building partnerships with potential international partners, 11) Building coalitions with local NGOs and stakeholders from the government and the business sector, 12) Implementation of the Communications Strategy. ECONOMIC DEVELOP-MENT POLICIES NBR (Modrica) i LINK (Mostar) NBR and Link: 1) Further strenghetning of the management and organizational structures, 2) Improvement of strategic and operational planning; 3) Strengthening financial and administrative capacity; 4) Improving human resource management and capacity building of staff and volunteers; 5) Increasing quality and portfolio of services; 6) Promotion of key results of projects and socio-economic impact of their organizations; 7) Diversification of funding sources; 8) Creating and using innovative methods and tools to support users; 9) Formation of a think-tank for local / regional economic development; 10) Improving media-positioning of the organizations and sector as a whole; 11) Implementation of new Communication Strategies. EDUCATION OKC (Banja Luka) 1) Strengthening organizational capacities, including the redesign of the organizational structure, analyses and redesign of existing internal documents and procedures and development of new ones, analyses of the operational plans of the organization, setting up tracking software to measure the performance of the organization and staff, renovating existing space and procurement of equipment; 2) Establishing a system of lifelong learning for employees with a clear definition of the priority areas for staff developmentm such as consulting and coaching services in the field of financial and accounting services, training to prepare for the labor market; human resources management, fundraising and project planning, marketing development, psychoanalytic organizational consulting and others 3) Improvement of existing programs and the creation of new projects and /or activities as part of the same; 4) The development of the financial capacity of the organization, including creating a position of organizational program manager for the development, preparation and implementation of the plan of fundraising; preparation and submission of applications for local sources of financing, designing and putting into operational use of alternative sources of funding, and procurement and putting into operation of adequate and improved accounting software, 5) Implementation of the Communications Strategy. HEALTH PfH (Sarajevo) and ICVA (Sarajevo) PfH: 1) Analysis of potential clients' needs and development of business plan with a list of services and price list for commercial services; 2) Building in the field of social entrepreneurship and marketing; 2) Creating the web platform; 3) Delivering commercial services; .4) Getting accreditation to provide HIV testing services through mobile teams; 5) Establishment and promoting e-clinic through creating a business plan and other related activities such as launch of the website, identification of partners and modalities of providing services, media presentation of the e-clinic, development and preparation of promotional materials, identification and recruitment of experts for writing of articles and promotional campaign. ICVA: 1) Development of strategic management and planning, including holding annual workshops for strategic planning and evaluation of the impact/effects; 2) Strengthening of human resources especially in the area of strategic planning, impact assessment, communication skills, monitoring and evaluation of staff performance and motivation); 3) introduction of dynamic website, printing and distribution of informational materials in order to promote ICVA in public; 4) Building cooperation with journalists through regular meetings and thematic workshops with journalists; 5) Finding new, alternative sources of funding and the development of a set of at least three new commercial services as a possible source of financing, while constantly attracting donors (specialist trainings and other educational events on topics such as recognition and the fight against discrimination in a particular area/institution targetted at public institutions, advocacy/policy/law changes towards parliaments targetted at NGOs, and development of research/analyeis on the legislation implementation or or other specificities of interest targetted at government institutions and donors). 103 SECTOR CSO SECTOR LEADER/CO￾LEADER PURPOSE OF GRANT FOR CAPACITY BUILDING AND SUSTAINABILITY ANTI￾CORRUPTION Transparency International (Banja Luka) and CIN (Sarajevo) TI: 1) System of strategic planning; 2) Organizational and technical capacity that will allow the organization to successfully meet the demands and needs of the users; 3) Human resource management; 4) Quality of the research and studies; 5) The presence and visibility of the organization among the citizens and the local community; 6) Diversification of the donor funding; 7) Commercialization of expert services the system of financial planning; 8) Media relations; 9) Repositioning of the organization within the user groups, and 10) Internal and external communication through the implementation of the Communications Strategy. CIN: 1) Capacity building in the field of management and organization; 2) Improving technical capacity of the newsroom operation and the organization as a whole; 3) Development of high quality multimedia projects; 4) Increasing quality standards of journalist work; 5) Maintaining the current level of diversification of donor funds and increasing the inflow of funds from commercial sources; 6) Branding 7) Building relations and cooperation with educational institutions, and 8) Internal and external communication through the implementation of the Communications Strategy. 104 ANNEX IX CATEGORIZATION OF NETWORKS’ CAMPAIGNS BY POLICY LEVEL 1 Unequal access to health care and discrimination of citizens, who have different access to health care services depending on their place of residence Mapped available health care institutions and services across BiH and created interactive portal with that information (zdravljezasve.ba). The campaign has focused on building a shared understanding of all stakeholders (including the public and government officials) on the need for access to health insurance. Broadening policy horizons 2 Exclusion of citizens from health care system by withholding information Conducted street action campaign, shared information on website. Conducted a petition supported by 7,000 citizens. In negotiations with Tuzla Canton, and City of Tuzla, affected the decision makers to cover health insurance for 32 children victims in Tuzla Canton. In collaboration with network member Zemlja djece from Tuzla, advocating for the Cantonal budget to allocate funding for consistent implementation of the legislation that guarantees free health insurance to the vulnerable groups. Arranged for a thematic session on this issue to be held by the Halthcare Committee of the FBiH Parliament with a presentation of recommendations by the sector leader and co-leader. In addition to the general efforts to get the insitutions to implement the Law on Health Insurance, the campaign managed to affect a short term solution for 32 children in Tuzla Canton to receive health care coverage. Affecting decision regimes 3 Citizens uniformed of their rights and obligations in the healthcare system Education of citizen through media campaigns and website, as well as thorugh the other two campaigns in street actions. This campaign is closely linked with the other two sector campaigns, but rather than being targgeted at the legislation, it is focused on raising awareness of citizens and increase the level of common understandings for demanding these rights. Broadening policy horizons 4 Violence against women and domestic violence Conducted media/visibility campaign (meetings with journalist, radio jingles, TV ad, posters, pens, bags, t-shirts). Held three panel discussions, several meetings with the stakeholders (government officials, representatives from public institutions relevant for gender equality, citizens, members of civil society organizations, media, private sector), as well as street-actions for Global Day of Women. Additional street actions held in seven cities, promoting the results of the report "Monitoring of Human Rights of Women". Submitted a proposal for amendment of adopted legislation on protection against domestic violence to regulate founding, financing, and operations of safe houses that was endorsed by several relevent institutions such as Social Work Centers. Also submitted the initiative to Federal Ministry of Interior for harmonizing the Rulebook on Conduct of Police Officers in Domestic Violence Cases and Enforcement of Protective Measures with the Council of Europe Convention on Preventing and Combating Violence Against Women and Domestic Violence. This campaign has taken on broad public awareness raising and submitted two policy proposals. Broadening policy horizons 5 Discrimination of women in labor market Conducted petition signed by 6,000 citizens against discrimination against new mothers in FBiH during the street actions in seven cities. Street actions: walk with new mothers, mothers walked through towns to draw public attention to their problems and talk to women MPs and citizens. Communicating with Women MP Caucus, and linking campaign with Women MP Caucus initiative which included proposal to equate maternal compensation in FBiH. Proposal to equate maternal compensation in FBiH was approved at the 9th regular session of the House of Represenatives of the FBiH Parliament. The iniative proposed by Women Caucus on the equating material compensation across FBiH has been approved by the FBiH Parliament. Affecting decision regimes 6 Gender sensitive budgets N/A 7 Improving position of women in rural areas N/A Women's rights Campaign not started Campaign not started Sector Priorities for Advocacy Campaigns Main Activites/Evidence, based on information provided by sector leaders/co-leaders in KIIs and CSSP reports Analyses of Policy Influence Level Assigned Policy Influence Level Health 105 8 Lack of transparency in public procurement process and lack of anti￾corruption mechanisms in the legislative framework (campaign aimed at improving the legislation on public procurement) Conducted media campaign based on analytical products (analysis of Public Procurement Law) and investigative reports on public procurement and links between public officials and private companies. Published online database with 40.000 entries on public procurement (2011-2014). Submitted proposed amendments of the to the Public Procurement Law to BiH Parliament. Held meetings with the BiH Agency for Prevention of Corruption and representatives of the BiH Parliament and advocating for changes. Members of two political parties in Parliament of BiH publicly pledges to send amendments into Parliamentary procedure. This campaign has managed to get its proposal known and heard by key government institutions. Broadening policy horizons 9 Limited circle of persons and institutions that are subject of the legislation on conflict of interests and mild sanctions that do not incite adherence to the legislation (campaign aimed at improving the legislation on conflict of interest) Worked together with Agency for Prevention of Corruption and Coordination of the Fight against Corruption of BiH and continued to point to the shortcomings in legislative framework and dysfunctionality, that is, a long-standing blockade in the implementation of the conflict-of-interest regulations. Submitted a set of recommendations for improvement of the legislative framework to remove obstacles to the implementation of the Law on Conflict of Interest at the BiH level. Meetings were held with the Conflict of Interest Committee. The Council of Ministers of BiH adopted necessary internal documents of Agency for Prevention of Corruption and Coordination of the Fight against Corruption of BiH. Legislation at FBiH level was sent into parliamentary procedure. Submitted recommendations to the FBiH Parliament for the amendments of the legislation on conflict of interest. Held a conference on conflict of interest and property cards and long-term solutions for institutional and legislative framework (aattended by 50 participants: institutions, media CSOs, MPs). Presented the Monitoring Report on the Whistleblower Protection. The proposals and consultation processes described are linked to the adoption of necessary internal documents of the Agency for Prevention of Corruption and Coordination of the Fight against Corruption of BiH by the Council of Ministers. Affecting decision regimes 10 Lack of control over asset declaration forms (campaign aimed at establishing mechanism for monitoring accuracy of elected officials’ asset declaration forms and responsibility for reporting false and incorrect information in the asset declaration forms) Held a conference on conflict of interest and property cards and long-term solutions for institutional and legislative framework (aattended by 50 participants: institutions, media CSOs, MPs). Monitored the activities regarding the amendments to the BiH Election Law and sent to the Interdepartmental Working Group a set of recommendations on amendments, including on provisions that regulate the obligation to file and control property statements. Draft amendments that are in adoption procedure took into account the recommendation to limit the obligation for filing property statement only to the elected representatives in order to facilitate the collection and control process; however, the recommended provisions regulating the oversight are not taken into account. Furthermore, based on sector leader's initiatve, the Agency for Protection of Personal Information issued official statement saying that there are no obstacles to releasing information and property cards of public officials. The campaign garnered an official statement from the Agency of Protection of Personal Information as a change in how the existing law is interpreted and implemented. Submitted recommendations partially adopted in the official draft legislation. Affecting decision regimes 11 Request for urgent establishment of a single Creative Europe Program Desk Campaign to create a coordinating body Creative Europe Program Desk for the use EU-Community program funds. Campaign was based on comparative analysis of Croatian and Slovenian case. Ministry of Civil Affairs of BiH adopted a Decree by which two CSOs (including Akcija) are jointly named the Creative Europe Program Desk for 2015-2017. The campaign succeeded in establishing the Creative Desk Europe and being a partt of its management. Affecting decision regimes 12 Solving the position of the BiH National Museum Established long term cooperation with the National Museum prior to public campaign, and used various technics to draw attention to the Museums status, including research, portraits of workers, concerts, short videos etc., culminating with the Museum Watch Action. Following the large Museum Watch campaign, with public figures participating, public events, conferences, Ackija proposed a temporary bridging financing model for Museum, which was adopted by relevant government institutions. As a result, BiH National Museum reopened after three years. The campaign effort resulted in adoption of a temporary bridging solution for the financing of the museum. Affecting decision regimes 13 Developing Culture Strategy in Sarajevo Canton, and initiating development of strategic documents at other levels Worked with officials and organized thematic public meetings to discuss the importance of the development of strategic documents in culture. Held a meeting with the Minister of Education, Science, Culture and Sport of Central Bosnia Canton, which resulted in the initiation of the development of a strategy for Culture for this Canton and a pledge that the the network would be involved in the development of the strategy. This campaign specifically focuses on building the technical skills of officials to develop a strategic approach for cultural policy and planning. Expanding policy capacity 14 Depolitization of managerial functions in the public institutions of culture as part of reform of public sector of culture in BiH N/A Anti-cor-ruption Culture Campaign not started 106 15 Introducing minimum standard for social security in both Entities and activating social protection Prepared policy analysis on priority problems in the sector of human rights of marginalized groups, which were presented at several meetings and roundtable discussions (including with the representatives of government/public institutions). FBiH Government refers to plan to establish a common basic social protection rights and minimum standard for social protection in its 2016 Work Plan. This campaign describes inclusion of their recommendations in the 2016 FBIH Work Program. Affecting decision regimes 16 Improving extra-institutional support to marginalized groups Held a public advocacy including the following elements: three public debates attended by public officials, series of media-stories "Why am I at the margin?", documentary film, two street actions conducted, media appearances, advertisements and social media outreach. Advocacy campaign influenced the FBiH Government Work program for 2015-2018 to include improvement and establishment of regulation of extra-institutional support to the marginalized citizens. Participation of network members in the public consultations on the draft legislation on professional rehabilitation, vocational training, and employment of disabled persons in RS, with the network's recommendations taken into account in the final adopted legislation. The campaign describes infuencing the 2015-2018 FBiH Government Work Plan as well as legislation on professional rehabilitation, vocational training, and employment of disabled persons in RS. Affecting decision regimes 17 Amending the legal framework for improvement of social entrepreneurship Produced Policy brief „Why do we need social entrepreneurship in BiH?" and analysis on "Extent of development of social entrepreneurship in BiH - main stakeholders and legal framework". The FBiH Ministry of Development, Entrepreneurship and Crafts started to work on drafting a set of incentives for development of social entrepreneurship based on the documents and recommendations proposed. The FBiH Ministry of Development, Entrepreneurship, and Crafts took into consideration network's recommendations. Broadening policy horizons 18 Amending the BiH Election Law in terms of electoral rights of national minorities Conducted street action "Being a minority does not mean being less important". Petition for support to national minorities signed by 800 people. Established cooperation with National Minority Councils at the state and Entity levels. Held conference with OSCE on changes to Election Law with regards to minority rights (attended by 88 participants). Sent recommendations to amend the Election Law to the interdepartmental group working on amendments. The campaign is working with key institutional actors on recommendations to the Election Law amendments. Broadening policy horizons 19 Increasing political will and competencies at municipal, city, and cantonal level for “greening” the local energy In RS the campaign is to motivate municipalities to develop energy efficiency (EE) strategies, as the RS Law on EE is adopted. For FBiH, the campaign advocated on adoption of EE Law at the Entity level FBiH. Engaged network members in preparation of request sent to the FBiH Parliament to put the EE Law back into adoption procedure and engaged with the media representatives to write about the initiatives. Municipalities and cities are approaching the sector leader and the network to receive technical support, while the international development agencies (UNDP, GIZ) are approaching the sector leader to be engaged in their programs for provision of technical support to the cities, municipalities and cantons in the process of developing EE strategic documents. Held meetings and workshop with key employees of local administration to motivate municipalities and cities to develop EE action plans. Produced analysis of current progress of all BiH local and cantonal administrations in increasing energy efficiency and utilizing renewable sources of energy as measures for environment protection and sustainable development and used the analysis to inform stakeholders. This campaign is attracting interest of multiple levels of government and creating will to work together on energy efficiency. Broadening policy horizons 20 State and Entity government to define strategic objectives and adopt action plan to increase energy efficiency Advocated for return of the Draft Law of Energy Efficiency (EE) into the FBIH Parliamentary adoption procedure and submitted proposals for improvement of this legislation. Established communication with the FBiH Government Secretary, who confirmed that the Law will be on the agenda of the Parliament soon. Established communication with thee Office of the President of the FBiH Parliament's HoP. Initiative for urgent adoption of the FBiH Law on EE by the House of People of the FBiH Parliament signed by more than 40 public institutions (municipalities, towns, ministries). Secured support from external experts/members of network in the process of developing EE strategies on local levels. Network members are involved in the development of Action Plan for implementation of FBiH Law on EE (Action Plan to be adopted upon adoption of the EE Law). Additional municipalities adopted Energy Efficiency Action Plans. The campaign is advocating toward multiple stakeholders to ensure adoption of the EE legislation and action plans. Several municipalities adopted the action plans. Affecting decision regimes 21 Educate and motivate citizens to implement concrete measures for energy efficiency and renewable energy sources Held eco workshops "Energy evenings with citizens", 14 workshops were held in 2015 and 13 in 2016 so far. Communicating with and education citizens through Facebook and audio-visual media. This campaign has focused on educating citizens to be aware of energy efficiency issues and priorities. Broadening policy horizons Human rights Enviroment protection and energy efficiency 107 22 Rational and efficient allocation of budget funds to combat unemployment at the Entity level Assessed audit reports and public calls for allocation of budget funds for entrepreneurship development programs across BiH, assessed potential strategic sectors of economy based on Entity development strategies, and conducted survey with 40 government institutions. Presented these analyses to public in June 2015. Conducted Media outreach including advertisement of findings from analyses, video production, and conference. Held roundtable discussion to collect proposals for completion of recommendations for improvement of subsidy awarding procedures. This campaign engages key stakeholders from the sector in the discussion and common understanding of the problems. Broadening policy horizons 23 Government institutions design and implement employment and economic incentives programs in line with the recommendations for efficient, transparent and effective performance Presented key sector problems to the public through more than 100 reports on unregulated financial support to economy, poor measures for development of entrepreneurship, employers burdened by inadequate tax collection system and high contributions paid on salaries of new employees. Una-Sana Canton Assembly adopted conclusions for improvement of the system of economic incentives as a result of data analysis produced by the sector leader (data was requested from the sector leader by a Parliamentarian who then presented it to the Parliament). Also held meeting with the FBiH Minister for Development of Entrepreneurship and Crafts. The campaign describes the Una-Sana Canton level adoption of conclusions for improvement of the system of economic incentives. Affecting decision regimes 24 Exempting the employers from paying social insurance contributions on employees’ salaries during the first year of employment, and postponing payment of VAT until payment is received for invoices based on which VAT was accrued Network announced in November 2014 that they will work on exempting employers from paying social insurance contributions on new employees, and the new FBiH Government adopted the Decree on Employment Incentives based on network's initiative. The Decree includes subsidies for employers that employ unemployed individuals with no work experience and long-term unemployed individuals. Over the next two years, the FBiH government will allocate around 70 million KM to the private sector, with 50 million KM for employment of young people. Sector leader reported that it "made sure that the Government of Canton Sarajevo secured 6 million for Employment program - Cofinancing programme in 2015". The Campaign for amending the VAT payment schedules received letters of support from four chambers of crafts, three chambers of economy, and many individual businesses, while four political parties endorsed the initiative. And event was also organized to present this initiative. This campaign describes influencing the FBIH adoption of the Decree on Employment Incentives and Sarajevo Canton employment incentives. Affecting decision regimes 25 Enactment of the Law on Agricultural Subsidies Rural Development in Republika Srpska Held several meetings with the representatives of the Ministry of Agriculture, Forestry, and Water Management that is preparing the Law on Agricultural Subsidies with thee aim to include improvement to the Draft Law. This campaign has engaged with key officials to improve Draft Law on Agricultural Subsidies. Broadening policy horizons 26 Enactment of the changes to the Law on Agricultural Advisory Services in FBiH Held discussion to raise the awareness that the adopted Law on Agricultural Advisory Services from 2013 does not make the advisory services responsible for rural development. Prepared studies and analysis on this subject. This campaign has spred information about the need for advisory services being responsible for rural development in FBiH. Broadening policy horizons 27 Enactment of a policy that regulates the area of organic food production Held working meetings between network's Working Group for organic production and relevant Ministries, presenting them with initial plan for the development of the Road Map. Obtained agreement that all three relevant Ministries at State and Entity level would submit comments on the network's recommendations based on which the sector leader would prepare final recommendations to be sent into the procedure as an initiative for the development of Road Map. Conference on Organic food production held at Foreign Trade Chamber (during the conference it was decided that a Road Map for development of organic food production should be drafted and submitted to CoM for adoption). This campaign has been working with the relevant Ministries to advocate for regulating organic food production. Broadening policy horizons Employmeant and labor market Agriculture and rural development 108 28 Ensure constitutional and legal preconditions for independence of judicial institutions Introduced professional organizations to network and mobilized them to participate on policy enviroment analysis, through Work Group activities on drafting proposals for changes to criminal legislation and changes to the Law on HJPC. Engagement of justice sector members, judicial office holders, lawyers, associations (professional organizations and stakeholders from the judiciary system participate in the network). Held meetings with professional associations (judges, prosecutors, Bar, etc.). Held a meeting with the Chief Disciplinary Prosecutor of HJPC. Network appearances in the media and distribution of findings from network's survey on perception of justice sector stakeholders of main obstacles affecting effectiveness and independence of judiciary. Held a two day conference with 54 participants to discuss survey findings and priority issues. TV program about judicial reform also prepared and broadcasted. This campaign has primarily focused on getting institutional experts together to define common priorities for reform. 29 Improve process of appointing judges and prosecutors and strengthening the role of the HJPC BiH as an independent institution Same as above, led by a Working Group for this campaign. The Working Gorups are coordinating the activities between each other. This campaign has primarily focused on getting institutional experts together to define common priorities for reform. 30 Ensure financial independence and autonomy of judiciary in BiH 31 Amending the Law on Budgets in FBiH, the of RS and Brčko District to introduce Budget for Citizens Meetings with government officials and relevant Parliamentary Committees to familiarize them with the concept of a Budget for Citizens. Brcko District Government members pledged that the initiative for introducing Budget for Citizens would be accepted. Held meetings with the World Bank and with ten key decision makers in DB. Organized a roundtables discussion in Banja Luka. Members participated in drafting proposals for the amendments to laws: FBiH Law on Budgets; RS Law on Budget System; DB Law on Budget. Prepared analysis on the introduction of Budget for Citizens in BiH context, obstacles and challenges as information tool for stakeholders and the public to support the initiative. The campaign has focused on familiarizing institutions and citizens with the concept of budget for citizens. 32 Amendments to the Law on Audit of BiH Institutions related to measures and sanctions for implementation of audit findings 33 Improving implementation of regulatory framework on rules for participation of interested public in the process of drafting BiH Federation legal documents and other regulations Held several meetings with the BiH Ministry of Justice, Federal Ministry of Finance, Assistant Secretary of the FBiH Government and the coordinator for reform of public administration in the FBiH. This campaign has focused on bringing together key actors on the side of government and civil society to discuss improving public participation in the drafting of FBIH legal documents. Public finance Campaign not started Justice Campaign not started 109 34 Acceleration of the business registration process in FBiH Prepared a Map for Faster Business Registration. The Map was presented to key stakeholders with recommendations for the improvement of Law on Business Entities and Law on Business Registration in FBiH were produced. Network's proposals for simplified and cheaper registration process are inlcuded in the adopted Law on Business Entities of FBiH. This campaign influenced new legislation on simplified registration process in FBiH. Affecting decision regimes 35 Lowering the number of parafiscal levies Conducted research and created a Map of Parafiscal Taxes. Held expert meetings with Ministries in Tuzla and Mostar to discuss the map. Based on the Map of Parafiscal Levies of the North-East Bosnia, Municipality of Srebrenik formed a Business Council to woek on reduction/elimination of levies. Established cooperation with the Foreign Trade Chamber and USAID FIscal Sector Reform Activity. The campaign resulted in the decision of Municipality of Srebrenik to form a Business Council to address the problem of parafiscal levies. Affecting decision regimes 36 Improving concrete support to business to counter flood damage Held focus groups and roundtables with members of Government and private sector to address joint measures for SME flood recovery. Held bilateral meetings with government institutions (FBiH Ministry for Development, Entrepreneurship and Crafts), providing them with the analysis of SME flood damage. Analyzed program of urgent SME flood damage recovery and support to recovery of businesses in the flood affected areas, based on 10 focus groups/170 participants in 10 municipalities. Held regular meetings with government representatives to inform them of analyses (SME-Flood) findings and needs. This campaign has inform the various government stakeholders on SME flood recovery needs. Broadening policy horizons 37 Designing policies for development of concrete models and instruments of support to business (e.g. business incubators, issuance of construction and usage permit, credit guarantee funds, etc.). Updated and submitted database of business incubators to FBiH Ministry for Development, Entrepreneurship, and Crafts, along with model for business/agro incubation. Model of business/agro incubator was developed and presented to the public at a conference. FBIH Ministry for Development, Entrepreneurship and Crafts included the model as special category for allocation of subsidies in 2015. Municipality of Konjic included the Model of Business/Agro Incubation into its work program. Ministry for Development and Entrepreneurship of Tuzla Canton expressed readiness to also include it in its plans. This campaign describes concrete influences on strategic documents at multiple government leveles. Affecting decision regimes 38 Lack of demonstration classes in schools Broadening policy horizons 39 Inadequate syllabuses/curriculums Broadening policy horizons 40 Poor alignment of enrollment policy with trends on the labor market Participated in Working Groups for drafting RS Education Strategy of RS and RS Youth Policy. Contributed to making youth employment and linking labor market and education top priorities of the RS Youth Policy Strategy. Participating in the drafting of the Action Plan for the Youth Policy. Participated in drafting of the final version of 2016-2020 Strategic Directions for Development of Higher Education in BiH. This campaign describes participation in drafting of two different strategies. This issue is included as a priority of the RS Youth Policy based on network's advocacy. Affecting decision regimes Economic development policies Education Held meetings with faculties and universities to introduce, promote and advocate for establishment of councils for alignment with labor market (HUB) at each university. All rectors supported the proposals. Held roundtables/conferences. Collaborated with the Agency for Development of Higher Education (HEA). Campaigns are focused on gathering together key stakeholders around the need for demonstration classes and curriculum changes. 110 ANNEX X DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Naida Carsimamovic Vukotic Title Acting Chief of Party Organization IMPAQ International, LLC Evaluation Position? Team Leader Team member (Methods Lead) Evaluation Award Number (contract or other instrument) Monitoring and Evaluation Support Activity (MEASURE￾BiH), implemented by IMPAQ International, LLC, Contract No. 168-C-14-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Civil Society Sustainability Project (CSSP), USAID-BiH, implemented by Centers for Civic Initiatives and Centar for Promotion of Civil Society, Award No. AID168-A￾13-00005 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date April 11th, 2016. 111 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date April 11th, 2016. Name Kristie Evenson Title External expert (Independent evaluator), STTA Organization IMPAQ International, LLC Evaluation Position? Team Leader Team member (Technical Lead) Evaluation Award Number (contract or other instrument) Monitoring and Evaluation Support Activity (MEASURE￾BiH), implemented by IMPAQ International, LLC, Contract No. 168-C-14-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Civil Society Sustainability Project (CSSP), USAID-BiH, implemented by Centers for Civic Initiatives and Centar for Promotion of Civil Society, Award No. AID168-A￾13-00005 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I was contracted by CCI in 2013 to provide some guidance on helping them think about designing a M&E plan for the CSSP activity This included some feedback to them on the proposal design stage and ways to present the general M&E approach, not on indicators as they were already discussed and decided by CCI and USAID, but in thinking about how to 1) design a system to capture these based on the logframe they designed and 2) some guidance for how to train the staff to do this internal M&E process. My guidance was sporadic in that it was around the work of creating the annual plan in the fall of 2013, feedback on their selection of an external polling agency and then in creating several guides for staff in early 2014 for implementing the process. I received a contract formalizing this arrangement in the fall of 2014 for 4 working days. Net amount of contract at that time was 1,276.18 EURO. I have not been in contact with CCI regarding the project since the end of 2014. 112 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date April 11th, 2016. Name Emina Cosic Puljic Title Research Analyst Organization IMPAQ International, LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Monitoring and Evaluation Support Activity (MEASURE￾BiH), implemented by IMPAQ International, LLC, Contract No. 168-C-14-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Civil Society Sustainability Project (CSSP), USAID-BiH, implemented by Centers for Civic Initiatives and Centar for Promotion of Civil Society, Award No. AID168-A￾13-00005 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. 113 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date April 11th, 2016. Name Anesa Hadzic Title Analyst Organization IMPAQ International, LLC Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) Monitoring and Evaluation Support Activity (MEASURE￾BiH), implemented by IMPAQ International, LLC, Contract No. 168-C-14-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Civil Society Sustainability Project (CSSP), USAID-BiH, implemented by Centers for Civic Initiatives and Centar for Promotion of Civil Society, Award No. AID168-A￾13-00005 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 19. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 20. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 21. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 22. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 23. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 24. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. 114 ANNEX XI MEASURE-BIH RESPONSES TO COMMENTS ON DRAFT CSSP PERFORMANCE EVALUATION REPORT September 18th, 2016 # COMMENT MEASURE-BIH RESPONSE 1 USAID/BiH: The first impression of the report is that it is mostly negative. We appreciate a critical review of the project. In order to try and correct the course of the project, it would be helpful if the report outline specific recommendations for eliminating or adjusting negative trends. In addition, if the evaluation team found that there were positive examples and successes of different activities, then we ask that those be highlighted more fully in the report as well, so that we could see what is working and come up with a plan for adjusting the project. The report outlines significant progress for each evaluation question, highlighting trends in the different components as well as the ways to build on these trends. In the revised report, we have further broken several findings to separate the positive from negative findings more clearly. Findings 1, 2, 9, 10, 12, 14, 15, 22, 28, 29, and 32 in the revised report all highlight positive aspects of the intervention. Our recommendations, as defined by evaluation question 4, focus on suggestions for adjustment of CSSP in the remaining two years of implementation and for any potential future interventions in the civil society sector. As such, our recommendations highlight intervention practices that should continue or be intensified, as well as actions that should be added or further considered. The recommendations address all findings (the evaluation team has presented connections between specific findings and recommendations to USAID/BiH). Report revisions add further positive examples or trends found in each of the components, with a special focus on component 3, providing concrete examples linking organizations and best practices. Our conclusions and recommendations also reflect these revisions. 2 USAID/BiH: The recommendations are not specific per sector. Instead, most of the recommendations are focused on management of the activity, including need for improved coordination and collaboration. Only one recommendation mentions the need for campaign refinement. In looking at ways to adjust the project, it would be extremely helpful to have more sector specific information and how some may be stronger or weaker in their organization and activities. Our Report provides summary findings disaggregated by sector for each evaluation question: 1) composition of network; 2) surveyed members’ satisfaction, 3) involvement and opinion of priority adequacy; 4) methods for priority identification; 5) alternative sources of funding per CSO; and 6) each campaign’s policy influence level. This evaluation was designed to evaluate CSSP as a whole against its designed objectives, as opposed to providing sector assessments. In order to evaluate sectors and provide sector specific recommendations, a larger set of KIs would need to be interviewed for each sector and all information would need to be verified with all relevant government counterparts in each sector. In other words, in order to provide official recommendations for each sector and evaluate sectors as strong or weak, twelve sector evaluations would need to be carried out and clear evaluation criteria would need to be defined (e.g. importance of the sector for BiH society, sector expertise of lead CSO, policy influence level, citizen engagement, tackling the most important sector problems vs. choosing “easy wins”, sustainability etc.). 3 USAID/BiH: The finding and recommendation related to public protest campaigning is well taken. USAID agrees that not all public advocacy is meant to be expressed on the streets. However, 115 it is concerning that civil society sector leaders and the most prominent local NGOs still exclusively focus on conferences and round tables as the major advocacy tool. It is obvious that the entire space of best practices between these two options is rarely considered by BiH civil society in order to increase citizen participation in decision-making processes. Even the report says that many campaigns are missing “strategic and tactical thinking that provides a backbone for such implementation best practices. Neither CSSP sub-grant structure nor management has helped very much…Overall, campaigns are achieving what they can where they can, but there is potential for improvement.” The Evaluation Team recognizes the media exposure CSSP campaigns are receiving and has elaborated on this in the revised report (in Findings 3, 22, and 24 and relevant conclusions). While this is still mostly presentation format (roundtables, press conferences etc.), it is a point from which other more innovative approaches could be developed. There is space for peer-learning from media expert partners and innovative campaigns targeting public engagement. The peer-learning needs to be facilitated and it should be understood that policy advocacy should not be limited to outreach towards the citizens, but also on building partnership and legitimacy with the authorities. Our recommendations on considering other types of civic engagement, campaign nature and maturity, and success factors seen so far also address this issue. 4 USAID/BiH: Discussion of the Resource Center is missing from the evaluation report. USAID is interested if the Resource Center is achieving results, and what role this Center is playing in building the collaboration, providing necessary information and possibly resources to local civil society organizations. Revised report includes Finding 18 and Recommendation 12 related to the Resource Center. 5 CCI: The CSSP team states that it is possible that there was some misunderstanding of the term newly formed networks and the networks that existed in some form prior to CSSP. Considering this evaluation report and its interpretation, the CSSP team confirms that total of eight new sector networks were formed, plus a support group in the anticorruption area. Hence, sector networks in the areas of Economic policy development, Agriculture and Energy efficiency, already existed in some form (but they have been additionally and substantially expanded under CSSP), while in the anticorruption sector there was already ACCOUNT network, because of which TI BiH and CIN did not create a new network. Instead, these two organizations created a Support group with the goal to identify key problems, to monitor progress in implementing anticorruption reforms, and to link with the existing ACCOUNT network. This Support group, acknowledging all existing initiatives and networks of anticorruption organizations, will seek to provide a wide basis for cooperation in order to ensure social consensus in identifying and overcoming problems regarding anticorruption efforts and strengthening responsibility in this country. Evaluation Team appreciates the confirmation of our finding. Revised report in footnote 12 clarifies that our finding was subsequently confirmed by the CSSP implementers. 6 CCI: Although there was no baseline assessment that would explicitly show „increased partnership among stakeholders“, we want to emphasize that at the start of CSSP most of the sector partners did not have any stakeholders network (only 3 of 12 to some extent had an established network) as a platform for joint courses of actions on finding solutions to sector problems (addressed by the finding 1 of the Evaluation Our Report already notes implied increased partnership in current Findings 1 and 2. We have added the point on formal mechanisms of communication installed within the CSSP in Finding 1. 116 report). Even the three networks that existed usually did not include government or media representatives, instead they were mainly composed of civil society representatives. Half way through the implementation of CSSP, despite this situation, stakeholders networks were established in all 12 sectors and their members were actively engaged in identifying sectors priority lists and partly in advocating for these priorities. This clearly points to an increased partnership between sector leaders ad interest groups, and is an indicator of advocacy legitimacy that sector partners have. In the end, it should be added that formal mechanisms of communication, as one key assumption for strengthening partnership among sector networks' members, was established through the component -sector network. This includes regular meetings (all networks), web pages and accounts on social networks (Facebook and Twitter), web platforms for internal communication among network members (Network for creation of a better business environment in BiH), newsletter and periodic bulletin (Coalition of marginalized groups in BiH – KOMA, Network for creation of a better business environment in BiH and Justice network). In sum, CSSP team believes that all this clearly points to progress in strengthening communication, trust, joint work and partnership between stakeholders and CSSP sector partners. 7 CCI: CSSP team believes it would be desirable to additionally explain/formulate a part on insufficient network membership involvement in the campaigns, given that, in terms of our continued efforts, the formulation (not sufficient) raises a question of what eventually would be sufficient and whether this is a quantitative measure (for example number of network’s members), or qualitative, that is, in terms of their stronger/more specific engagement in advocacy efforts. Conclusions cited here refer to Finding 4 in the revised report which suggests that there is room to further utilize membership talents, views, and facilitation efforts. While there is no specific level or number identified by the evaluators that would be deemed sufficient, the survey and KII data suggest that there is opportunity and interest to further work with members in many ways. This is reflected in Recommendation 2. Language in the conclusions has been revised to further reflect this in revised report. 8 CCI: The CSSP team cannot agree with the finding and conclusion that we forced opening of private companies to achieve sustainability of involved CSOs. This topic was discussed at consultations and individual meetings and joint coordination meeting; this option was listed as one of potential alternatives, in addition to other option such as (local budgets, membership, donations from private companies, commercialization of certain services through CSOs, etc.). The CSSP team did not force or favor any potential source of funding, and sector partners themselves defined the forms of Our report clearly states that this finding (Finding 16 in revised report) is based on perception of sub-grantees. Regardless of what was stated in meetings and documented by the implementers, the fact is that eight sub-grantees feel a pressure to register an enterprise regardless of their CSO type. We have adjusted the language in Finding 16 to further emphasize this and to note that the CSSP implementers explain that they did not favor any of the alternative funding options nor put forward any preferred options to the sub-grantees in documents and during meetings and that the sub-grantees themselves defined their forms of funding diversification. We have also expanded Finding 16 to note that the resistance to the idea of registering an enterprise by some CSOs in part stems 117 diversification of funding in line with their sustainability strategies created with CSSP support. CPCD: The finding, which, I would say, is based only on statements of some grantees, which says that CSSP implementers forced i.e. favored one alternative source of funding – starting up a company, and that promotion of good practices that some grantees utilized was frustrating to others, is unacceptable. It seems that besides statements from grantees there was failure to consult other documents that could dispute this subjective opinion. This is primarily because it is stated in one place that around half of grantees either formed a company or are considering that possibility. In addition, in grantees' statements one can recognize some misunderstanding of difference between social enterprise and traditional private company that has only one goal- profit. Expression of that misunderstanding is a statement that „by forming a company (presumably a for-profit enterprise) one organization would lose its character and purpose of its existence as civil society organization. It should be note that whenever we mentioned and promoted formation of enterprises, we always emphasized that it was about social enterprise, an enterprise that uses profits to fund the mission of a civil society organization as its founder. In further implementation of the project, this could mean that we should invest more efforts in raising awareness among grantees about the concept and purpose of social entrepreneurship. A planned informational and consultation training session on this topic that will be held in September will certainly be beneficial. With regards to the finding that we as implementers in advocating for possible alternative sources of funding forced i.e. favored a creation of a company , it is true that in all direct, individual and group meetings with grantees (consultations with all grantees implemented by the Capacity building manager from December 2015 to January 2016, consultation meetings in Konjic and Sarajevo), and in appropriate documents (let’s just mention the Matrix for monitoring progress in achieving expected results in the area of financial sustainability), always and equally treated and promoted all alternative sources, always stressing that selection of a specific form of financing would depend on the type of organization and its thorough assessment. It is also not true that we failed to pay sufficient attention to the type and different missions of CSSP beneficiaries while advocating for alternative sources of funding. This attitude implies that we meant a formation of a (for-profit) company, providing paid services... We only said that it would be up to the organization itself to choose its alternative ways of funding. We did not necessarily imply a creation of a company or providing services to government (advocacy org, watchdog from undeveloped regulation that would distinguish social enterprise from traditional for-profit enterprises in BiH (reflected later in Finding 31 and Recommendation 21). 118 etc.). Hence it was not about imposing any forms of funding that are not foreign donations, but it was about encouraging CSSP grantees to start thinking about alternative sources and start applying them in their fundraising practices. Many advocacy and watchdog organizations in the world are funded from the funds that are not foreign donations (individual philanthropy, sponsorship, membership fee…). 9 CCI: It is CSSP team's general assessment that CSSP's implementers role in some processes and stages of CSSP is to some extent devalued, especially in the component Campaigns and monitoring and Sustainability, and that „hands￾off“ approach and implementers ' administrative and technical role in sub granting is overemphasized. Several practices in which CSSP team was substantially involved refer to the process of advising and facilitating and proactive activity toward sector partners, starting from planning of the project activities and their further development (based on inputs and comments from CSSP team) in project proposals (with all sector partners and for all approved project proposals), consultation assistance for preparation and realization of individual activities (especially public events/appearances) down to detailed elaboration of the plans of the campaigns in eight sectors. Additionally, these claims with regards to „hand-off“ approach in the implementation of CSSP are at odds with the reports of grantees' perception of CCI's and CSPC's work in the implementation of CSSP. In the 2014 report, only 1.8% of the surveyed grantees (via anonymous online google form platform) stated that CCI/CSPC, rarely or are not at all involved in the activities of the grantees. The same report for 2015 shows that only 0.55 % of the surveyed grantees repeated that claim (negative trend). Contrary to this, 76% of the grantees in 2014 and 90% in 2015 stated that CCI/CSPC were sometimes or moderately involved in the grantees' work (reports submitted to USAID and available upon request of the Evaluation team). In line with the findings, CCI's/CSPC's approach was to collaborate with the grantees and provide support, giving them at the same time room to make their own decisions and strengthen their internal organizational responsibility. With regards to the campaign design, in the second half of 2015 (at the end of program year 1 of the implementation since the start of sub-grantees in the area Campaigns and monitoring), the CSSP team, using its own monitoring process, identified a problem associated with the lack of clear campaigns in some cases (as pointed by the finding/conclusion of the evaluation). In October 2015, the CSSP team started the practice of introducing plans of the campaigns/action plans in the sectors where it deemed most necessary. In that sense, the CSSP team responded to the observed problem and introduced a new tool (although it was not initially planned by the project), which has proved beneficial in our advocacy experiences and sought to transfer knowledge and CCI described their management style to the evaluation team as not involved in substantive aspects of campaigns in order to allow for ownership and creativity by the partners. Most importantly, we would like to emphasize the need to differentiate between different levels of management involvement - i.e. administrative and operational guidance vs. strategic guidance. The strategic guidance refers to sector-specific advice and structured and consistent criteria for definition and implementation targeted campaigns and tasks and reporting on them. We found insufficient evidence supporting the existence of such strategic guidance based on interviews and documents. The information about the new campaign management tool was not offered previously to the evaluation team by the CSSP implementers and none of the KIIs mentioned this. It is encouraging that CCI sees the need for a clearer framework and tool, but its utility is still to be seen. Revised report shows adjusted language in line with our responses in Findings 5, 25, and 26 (including footnote 27) and relevant conclusions. 119 experiences to sector partners. During this process, the CSSP team and sector partners held more than 20 individual consultation meetings; provided assistance during development of the plans of the campaigns; transferred knowledge and experiences in the area of advocacy and at the same time in return it got a management tool spoken about in the finding 25. Most of these plans of the campaigns started in 2016, so we believe that potential effects of improvement of the campaign planning system may be expected in the upcoming period. This type of assistance and facilitation, in this way, through the plans of the campaigns, was completed with eight sector partners. Also, this tool will be used in the upcoming period and with involvement of other sectors that so far have shown more extensive experience and capacities in the area of advocacy. Zene Zenama: The only thing that we do not agree with relates to the finding concerning the role of the main implementers of the Program – it is indicated in several places that their role is technical and administrative and not strategic and advisory. Because of our experience, we cannot accept such statements because we have had advisory support primarily, but also administrative and strategic in all stages of the preparation, implementation and reporting on all three components of the Program. Sometimes it seems that Program management staff is too involved and that they monitor every detail regardless of whether it is about an activity in the components networking or sustainability and monitoring. 10 CCI: The CSSP team cannot agree with the finding that some sector partners were under pressure to run negative campaigns in the streets/media and organize protests. On the other hand, our team insisted on visibility of the campaigns, some activities and implementing organizations, public mobilization and raising of the problem to public level. This is particularly important in some phases of the campaigns when there are efforts to mobilize public and create advocacy legitimacy. Street events, do not have to be, and mainly they have not been so far, a reflection of negative campaigns, but they have been used to mobilize public, to draw attention to the problem, to distribute promotional materials, to utilize performances. However, CSSP team does believe, and this was discussed with sector partners while assisting them to define the plans of their campaigns, that in case government ignores or fails to intervene and find solutions to the problem, that other tactics such as negative campaigns should not be avoided and government or individuals should be called upon their responsibility. We believe that this only reflects a principled CSO approach and that any other approach would be characterized as a failure to persist on finding solutions to Our report states that this finding (Finding 24 in revised report) is based on perception of sub-grantees and we have further used that language in the revised report. Regardless of what was communicated to the sub-grantees by the implementers, almost all of the sub-grantees expressed that they feel pressure to focus on public campaigning and street actions to voice public criticism of authorities, irrespective of their campaign focus, progress, and maturity. Sub-grantees’ increased media engagement and exposure is a good step in the direction of being more innovative in messaging and engaging target audiences, as is noted in Finding 22, but CSSP management’s way of using street protests/campaigns as the best examples for public exposure frustrated sub-grantees while not pushing them to focus on potential alternatives to get media and visibility that would assist their efforts. 120 sector priorities, which could additionally bring lack of confidence and endanger NGO image among citizens. Further on, in terms of visibility of the campaigns, i.e. some activities such as street actions (but not the ones with negative sign), they are used as tools to strengthen recognition of organizations as sector leaders and key actors in the sectors in which they work. This, again, is directly related to the goals of sustainability, because we believe that strong organizations that are recognized in their sectors can provide alternative sources of funding more easily (philanthropy, local budgets, etc.). It should be emphasized that most of grantees before CSSP did not use at all (or use till some level) media tactics in order to communicate the problem with a wider community. It is visible from the report on media appearances that now all organizations have knowledge and that they much more utilize media as tactics in their campaigns (even those that had the knowledge) in relation to the start of CSSP. For example, during the quarter 1 at the start of advocacy projects (last quarter of 2014) all sector partners registered 82 press statements while the quarter 1 of 2016 saw 476 press reports covering the activities of the sector partners. In sum, insisting on street events/protests with a goal to criticize government and run negative campaigns has never been CSSP team's requirement or ultimatum, although during consultations and work on designing campaigns we did frequently encourage efforts to improve visibility of the campaigns and introduce innovative advocacy tools as opposed to previously more frequently used tactics, such as roundtables, conferences, etc. 11 CCI: Although it is not completely clear about what type of overstated impact it is about, the CSSP team wants to emphasize that eventual insufficiently precise reporting on the impact has not been done consciously or on purpose. As the findings in the evaluation report state, changes and impact on modification of public policies are complex social and political processes where it is sometimes extremely difficult to identify a domain of the campaigns implemented by sector partners on CSSP. The CSSP team believes that this management tool - the plans of the campaigns- will additionally specify sector partners’ interventions in policy issues, and therefore significantly reduce potential risk in terms of formulations in the report. We believe that this tool will be beneficial in the context of having a more concise reporting on reach and impact of advocacy activities on CSSP. The evaluation team recognizes that reporting policy influence is a challenging endeavor, as we acknowledge throughout the findings related to evaluation questions 3 (in Findings 19, 21, 22, 24 in the revised report). However basic reporting practices could have early on assisted (beyond the now noted new campaign management tool which in any case does not display significantly different reporting yet for quarterly reports in 2016) to make the annual and quarterly reports documents clearer. It appears that both implementer and partners were somewhat inundated with reporting efforts and submitted reports with very uneven descriptive accounts, unsubstantiated impact claims and often incomplete information on all the campaigns they were implementing. More systematic reporting mechanism is welcome, however to date the questionable impact claims described in the reports were pervasive, regardless of the intentions. . Finding 26 in the revised report is slightly revised to elaborate in line with our responses. 12 CCI: Even though the project initially planned a somewhat stronger and clearer link between advocacy efforts and issue areas We recommend to include the EU-integrations dimension where possible - based on what was envisaged by the CSSP design - and not to focus exclusively on it. Finding 23 in the revised report has been adjusted to clarify that only a couple of CSSP campaigns are 121 related to EU reform processes, some priorities that did not have such pronounced direct relationship with these processes were identified together with advocacy priorities of the sector network. Given that this concept of the project clearly highlights the role and consensus among networks on the issues that they will be tackling, we believe that it would be counterproductive to suggest and influence sector networks to focus exclusively on this type of reforms. On the other hand, we believe that some other advocacy initiatives are related to EU reform policies such as the initiative to modify VAT Law (Employment and Labor market sector) or the initiative to modify Law on Business Registration in order to simplify and reduce the costs of business registration in FBiH (Economic development policy sector), the initiative to establish social minimum for FBIH and census for defining real social need (Human rights and marginalized groups sector) the initiative to modify Law on Conflict of Interest (Fight against corruption sector), the initiative to form Councils for linking with labor market at 80 faculties of six public universities in BiH (Education sector). Parallel with this, CCI in consultations with DEI representatives created a comprehensive analysis of EU integrations, in a wider sense, and utilization of IPA funds. KULT: Comment refers to the Finding 21 – page 42, where KULT has not been stated as an organization whose initiatives promote EU integration solutions. The initiative to modify VAT Law, that is, introduction of payment of VAT upon paid invoice is the practice used in EU countries and something that has been stipulated by EU directives. We strongly emphasized this in our advocacy. We would appreciate if this fact was included in the document. related to the most immediate EU integrations requirements, with a few other campaigns that could be more broadly connected to some longer-term EU integration requirements. The campaigns that are mentioned in CCI’s and KULT’s comments are not related to the immediate EU integrations requirements. The mentioned comprehensive analysis of EU integrations is not done within CSSP (based on the evaluation team’s understanding) and the interviewed DEI representative notes no contact related to CSSP. 13 CCI: The CSSP team has been proactive in fulfilling its responsibilities to monitor the realization of sub grantees. It also provided support to grantees (general and tailored) in planning and implementing their advocacy and organizational development activities (for example „Women's rights sector, Health care sector, Culture, Agriculture) in line with the needs of different stages of the project. With regards to this, functions of technical assistance or grant management were somewhat more emphasized in the early stages of the project, primarily through selection of sector partners, basic training sessions, approval of applications on project component, establishment of grant management system, etc. Because of this, the roles were probably more positively assessed, while some other roles, like facilitating role, are to be more strongly demonstrated. Despite the fact that most of the partner organizations actively participated in the campaign aimed to open museum in Sarajevo, as a form of articulated cooperation among grantees, the CSSP team believes that in the upcoming period there will be a need to facilitate relations Our findings suggest that management of the CSSP has primarily or initially been more administrative and this is confirmed in this comment by CCI. Efforts to expand on this are noted, but the room for further management evolution to focus on smart facilitation, strategic guidance etc. is present and should be considered. The conclusions for question 3 in the revised report reflects slight revisions in relation to this. 122 to engage in information sharing/cooperation among grantees, in line with the development of their advocacy campaigns. 14 CCI: This explanation refers to the Finding 3 of evaluation report, but more directly to the statement of representatives of CSOs quoted. CSSP team wants to point out the fact that in implementation of the project more joint meetings of implementers and partners were organized than the quoted statement says (two meetings). Five coordination meetings with partner organizations were held, one of which was with the participation and the invitation of the Director of the USAID Mission. These coordination meetings were held on May 5, 2014 at the Hotel Hollywood in Ilidza, March 11 and 12, 2015 at the Hotel City Garden in Konjic (two-day meeting), on October 14, 2015 at the Hotel Hollywood in Ilidza, November 10, 2015 at the Hotel Hollywod in Ilidza, and at the invitation of the Director of USAID March 3, 2016 at the Hotel Bristol in Sarajevo. In addition, after completion of the selection process of sectoral partners, on June 27, 2015 in Sarajevo (UNITIC) was held kickoff event on which the public was presented with all organizations, sectors and objectives of the CSSP. Finally, the two joint multi-day training sessions for the organizations, held in May (20-24) and June (23-26) 2014 at the Hotel Hollywood in Ilidza, also provided an opportunity to CSSP partners to discuss possible directions of action and common perspectives on the project CSSP. Report revisions (Finding 5 and footnote 13) reflect CCI’s comment on the number of the meetings/joint trainings. However, not all of these were facilitation meetings, and even facilitation meetings were described by sub-grantees primarily as presentations rather than facilitated discussions. Clearly, the responsibility for such coordination and sharing rests with both partners and implementers, but the space for implementers to improve the likelihood of better coordination should be utilized. 15 CPCD: Generally, I believe that many findings refer to the concept of CSSP, not to the way and quality of its implementation. This specifically relates to the findings that address the scope and type of technical assistance that CSSP implementers were asked to provide according to the project and to the structure and purpose of three grants of the project. I believe that this did not receive sufficient attention and that the team did not consult all the documents it had available. Therefore, it was the implementers' obligation to provide basic training and ensure realization of three OCA processes, to facilitate Public call procedure and select external consultants to provide professional assistance in development and subsequent implementation of sustainability and communication strategies (this was the main purpose of the first and second part of the sustainability grant). Financial support through sustainability grant aimed to ensure that grantees had funds to finance the scope and technical assistance according to their needs, which they had the chance to (self) assess through the OCA process that was facilitated by external consultants and with participation of the grantees. These needs they included in their sustainability and communication strategies. In that context, the assessment that the grantees did not receive technical The performance evaluation task is broader than simply reviewing implementation process, as it requires a look at both design and implementation factors affecting the progress in achieving outcomes. The evaluation team reviewed all documents received from the CSSP implementers and USAID/BiH and also requested additional documents from the implementers to better understand specific components of CSSP – i.e. format of the grant structures etc. However, review of the documents did not provide good information due to unclear reporting and generalist formats. Also note that we were told that individual sub-grantee documentation is confidential. In terms of CPCD’s comment about the sub-grantees receiving technical assistance for building sector-specific technical capacities, evaluation team did not state that such assistance could not be provided under CSSP. We rather state that the grantees have mostly not used the opportunity for such assistance so far, as they have mostly focused on internal organizational procedures and operational capacities (Findings 10, 11, and 12 in the revised report). In terms of CPCD’s comment on flexibility and freedom, we state in the report positive and negative sides of the flexibility. Negative 123 assistance in building their organizational and technical capacities specific to the sector in which they work simply could not be accepted. In that sense, CSSP implementation, in line with project's design was flexible and it left grantees with full freedom to spend the grants they were awarded. In respect to this flexibility and freedom, it is noticeable that there is a certain contradiction that explicitly or implicitly runs throughout this report. On one hand, this flexibility and freedom of grantees is listed as quality of the project, while on the other hand, this characteristic of CSSP approach was not sufficiently considered while evaluating grantees' views expressed through semi-structured interviews and survey. Of course, one should acknowledge and appreciate grantees' views and opinions, however I believe that some of their views have come out as a result of the ways in which the questions were defined (contents of the questions). We do not know the contents of the survey nor are we aware of the questions from the protocol for the semi-structured interview, but it is my impression that the formulation of the questions did not pay enough attention to CSSP design, including the type and scope of assistance that grantees were supposed to receive from the implementers. I am sure that the grantees were aware of this from the very beginning of their participation on the project, which could not be said from looking at some statements listed in the report. This way we have appraisals for CSSP flexibility and freedom it offers to its grantees, but the conclusions or recommendations do not mention grantees' responsibility for purposeful spending of the funds awarded for achieving the goals and expected results defined in their sustainability and communication strategies, their OCA Capacity Building plans. Along with this, the report suggests changes in CSSP design, especially in terms of the scope and contents of technical assistance without paying attention to budget limitations in that sense and purpose of the awarded grants. Personally I could accept that we as implementers should provide more assistance to grantees in the future through increased Manager's personal assistance, through mentorship and consultations, through different and more efficient utilization. Other options for more substantial assistance are to have a more purposeful utilization of the awarded grants. It would not be possible to accept the assessment that grantees did not receive adequate assistance in development of sustainability and communication strategies. Especially because of the role we as implementers had in that process. I believe that this is about certain level of grantees' dissatisfaction with the quality of service that was provided to them by experts that they employed. I would agree with the fact that the situation on the market in terms of supply of expert services, particularly in the areas of concept and practices of sustainability, is not satisfactory and that there are not that many local expert organizations that could sides did not need to preclude the positive ones - i.e. the sub￾grantees could have been instructed to plan sector-related capacity building and cross-sector cooperation, leaving exact modules to them to define. In terms of CPCD’s comment about the assessment that grantees did not receive adequate assistance in developing sustainability and communication strategies, our report says that assistance was provided but is perceived by eight sub-grantees as not equally useful and applicable for all CSOs that participate in CSSP, depending on their targeted purpose (Finding 16 in revised report). We agree that this could potentially be due to lower quality of experts they selected, however, we do not have enough evidence to explicitly note that as a finding. In terms of CPCD’s comment on our evaluation suggesting changes in design without paying attention to budget limitations and purpose of the awarded grants, we note that all recommendations are focused on concrete and actionable ways for management to adjust its focus to enhance outcomes in the three component areas. These adjustments are not cost intensive, but require additional management focus (e.g. in the case of network facilitation) or require use of existing funds in more focused ways (e.g. in relation to emphasis on M&E for capacity development or use of other partners’ skills to enhance specific skill sets). We did not have access to detailed budgets of either implementers or the grantees (beyond the 3 component breakdown of the approximate $400k grant funds), but we would assume that there is some flexibility in how remaining funds can be focused to achieve enhanced outcomes (particularly since the sub-grant budgets themselves appear to be quite flexible). Edits in the revised report preceding the recommendations includes this assumption. Finally, in terms of the CPCD’s comment about not knowing the contents of evaluation team’s survey questionnaire and KII protocols, we would remind that the data collection instruments for both KIIs and survey have been provided in the Annexes of the Evaluation Report and took design into account. Six sub￾grantees explicitly described criteria for tasks that can be supported under CSSP sub-grant as not being clear to them. 124 provide these services. Sustainability that CSSP promotes is a new area for them. 16 CPCD: In the end, which again relates to the contents of the questionnaire and protocol for semi-structured interview, I am surprised that the findings do not recognize that some views of the grantees on which the report relies are direct results of the fact that CSSP project is the first project that has been promoting a different, comprehensive and strategic approach to thinking about CSO sustainability. NGOs are not used to this new approach, which includes changes in opinions and behavioral patterns, which is not always an easy thing to do and which sometimes faces conscious and unconscious resistance. However, the evidence that the situation is changing as a result of CSSP is in the fact that all grantees introduced or are ready to introduce one or two alternative sources of funding and that sustainability through capacity building is no longer perceived as CSSP project goal but as conditio sine qua non without which it would be difficult to imagine long-term sustainability of NGOs. As explained above, all data collection instruments for both KIIs and survey have been provided in the Annexes of the Evaluation Report. Our report already in the section on CSSP background clearly notes the uniqueness of CSSP in their focus on sustainability, as well in findings and conclusions which state that the sub-grantees appreciate CSSP’s focus on sustainability (e.g. Finding 12 in the revised report). We have included language to elaborate further on this in Finding 9 and in conclusions in the revised report notes that, given the environment in which CSOs operate, often juggling among different projects and chasing funding, all CSSP CSOs appreciated the process that allowed them to devote specific time to internal organizational analyses and strategic planning. 17 KULT: The first relates to the assessment that KULT does not have media representatives in the net-works. This is officially true, there are no signed documents. However, our media reports and more than 200 media appearances over the past six months show that the media have been covering us, that they have shown up and that we have had excellent collaboration. When asking media to be-come our official members we usually encountered resistance, because they do not want to formalize the cooperation. It is different in practice and they do support us. The evaluation team acknowledges that increased media coverage is positive and has articulated this more clearly in the revised report, as per our responses outlined in this document under comments 3 and 10. However, CSSP design envisaged an actual deeper engagement of media representatives in network work. 125 U.S. Agency for International Development/Bosnia-Herzegovina Robert Frasure Street #1 71000 Sarajevo Bosnia and Herzegovina