September 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Andrew Green and Sanja Nikolin through Social Impact. Performance Evaluation of Civil Society Advocacy Initiative (CSAI) in Serbia i | Page Cover photo: Corporate volunteers give back to their community. Photo selection courtesy of Andrew Green. Translation of cover photo text: “Through corporate volunteering, companies motivate their employees to be volunteers and offer their time, knowledge and skills for the benefit of others and the community.” ii | Page FINAL PERFORMANCE EVALUATION OF CIVIL SOCIETY ADVOCACY INITIATIVE (CSAI) IN SERBIA Draft: June 21, 2013 Revised: August 2, 2013 Second Revision: August 23, 2013 Final Submission: September 6, 2013 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii | Page CONTENTS Contents .......................................................................................................................... iii Acronyms ........................................................................................................................ v Executive Summary ........................................................................................................ 1 Evaluation Purpose and Evaluation Questions ......................................................................................... 1 Project Background ................................................................................................................................... 1 Design, Methods and Limitations .............................................................................................................. 2 Conclusions ............................................................................................................................................... 3 Recommendations .................................................................................................................................... 6 Evaluation Purpose & Evaluation Questions ................................................................... 7 Evaluation Purpose ................................................................................................................................... 7 Evaluation Questions ................................................................................................................................ 7 Project Background ......................................................................................................... 8 Context ...................................................................................................................................................... 8 Project Purpose ....................................................................................................................................... 10 Evaluation Methods & Limitations ................................................................................. 15 Findings, Conclusions & Recommendations ................................................................. 18 Findings and Conclusions ....................................................................................................................... 18 Overall Conclusions ................................................................................................................................ 33 Overall Recommendations ...................................................................................................................... 33 Annexes ........................................................................................................................ 35 Annex I: Evaluation Statement of Work .................................................................................................. 35 Annex II: Evaluation Methods and Limitations ........................................................................................ 39 Annex III: Data Collection Instruments .................................................................................................... 42 A. Evidence Matrix ...................................................................................................................................... 42 B. Semi-Structured Interview Questions ..................................................................................................... 44 C. BCIF Grantee Mini-Survey Questions and Summary of Results, English ............................................. 48 D. Mini istraživanje dobitnika BCIF grantova i sažetak rezultata, srpski .................................................... 49 E. MAS Mini-Survey Questions and Summary of Results, English ............................................................ 50 F. Mini istraživanje MAS i sažetak rezultata, srpski .................................................................................... 51 Annex IV: Sources of Information ........................................................................................................... 52 Persons Interviewed .................................................................................................................................... 52 Documents Reviewed ................................................................................................................................. 55 Annex V: Disclosure of Conflict of Interest .............................................................................................. 60 Annex VI: Coverage Tables .................................................................................................................... 62 Table 1. Comparison of Total Grants to Evaluation Interview Grants, in dollars ........................................ 62 iv | Page Table 2: Comparison of Total Grants to Evaluation Interview Grants, in #Grants Covered by Interviews (Not #Interviews) ......................................................................................................................................... 63 Table 3: Comparison of Total Grants to Evaluation Interview Grants by Year, in dollars .......................... 64 v | Page ACRONYMS ACAP Advanced Community Advocacy Program AOR Agreement Officer’s Representative AP Advocacy in Practice ARPers Advocacy Resource Persons BCIF Balkan Community Initiatives Fund CI Civic Initiative COR Contracting Officer’s Representative CRDA Community Revitalization through Democratic Action CSAI Civil Society Advocacy Initiative CSF Civil Society Fund CSO Civil Society Organization CSR Corporate Social Responsibility DCOP Deputy Chief of Party DG Democracy and Governance DP Democratic Party DSS Democratic Party of Serbia ECNL European Center for Not-For-Profit Law EU European Union EurInt European Integration FH Freedom House GL Green List GoS Government of Serbia GRAG Grassroots Grants ICTY International Criminal Tribunal for the former Yugoslavia IF Innovation Fund IPA Instrument for Pre-Accession Assistance IQC Indefinite Quantity Contract ISC Institute for Sustainable Communities JA Junior Achievement JI Joint Initiative M&E Monitoring and Evaluation MP Member of Parliament NDI National Democratic Institute OCCS Office for Cooperation with Civil Society PCAP Proficient Community Advocates Program PMP Performance Monitoring Plan POC Point of Contact SAA Stabilization and Association Agreement SF Successful Fundraising SG Success Grants SI Special Initiatives SK Smart Kolektiv SNS Serbian Progressive Party SSI Semi-Structured Interview SOW Statement of Work USAID United States Agency for International Development 1 | Page EXECUTIVE SUMMARY Evaluation Purpose and Evaluation Questions The objective of this performance evaluation is to conduct a full, evidence-based, and independent review of the CSAI program implemented by the Institute for Sustainable Communities (ISC), and to produce a report that provides a qualitative as well as quantitative evaluation of project activities. This final evaluation of CSAI addresses four evaluation questions covering aspects of both performance and process: a) What are the core achievements and legacy of CSAI? This can include achievements in comparison to the workplan as well as unintended results. b) What lessons can ISC learn from the management and implementation of CSAI? c) How effective was CSAI in strengthening civil society’s capacity for and engagement in advocacy? d) What were the obstacles or difficulties ISC encountered during implementation of the activity? The key stakeholders for this evaluation are USAID/Serbia and Montenegro’s Democracy and Governance Office, USAID/Washington, ISC/Serbia, and ISC/USA. The findings and recommendations will inform implementation of the newly awarded Civil Society Forward program, which is also implemented by ISC. Project Background Context CSAI was implemented during a time of volatile politics and state relations: extraordinary elections, changes in the ruling coalition, party-switching Members of Parliament (MPs), and the divisive issue of cooperation with the International Criminal Tribunal for the former Yugoslavia (ICTY). At the same time, the state itself was being redefined as Montenegro split off, and then Kosovo unilaterally declared independence. The allure of European Union (EU) accession both constrained politics and split political parties, leading to an extensive reform agenda that could be derailed at any time by shifting political alliances. The global recession beginning in 2008 further complicated the development of civil society and democracy because it cut short a promising period of economic growth. Civil society has also undergone significant changes, but public perception of the sector has remained distressingly negative. The sector is largely split between organizations pursuing recreational and other non-political activities, and those pursuing political reform and policy advocacy (of which many are rooted in anti-Milosevic activities). Philanthropy and corporate social responsibility (CSR) are taking root in Serbia and, on the whole, the enabling environment has improved. As foreign donors begin reducing their activities in Serbia, diversification and sustainability remain problematic since the EU accession process targets funding in very different parts of civil society than does traditional democracy assistance. Support for Civil Society Development USG support for civil society has been extensive, including inter alia Freedom House programs and the large USAID-funded Community Revitalization through Democratic Action (CRDA) program. In 2006, USAID launched the Civil Society Advocacy Initiative (CSAI) as a $12 million, five-year program to help civil society organizations (CSOs) better represent the needs of ordinary people, and for civil society to 2 | Page become a more influential and trusted partner of businesses and the government. After three additional ceiling increases during 2008-10 and an extension of two years, the program ended in May 2013 as a seven-year, $27.5 million initiative. The theory of change that appears to have motivated the design of CSAI is that fundamental cultural and political change results from both citizens who are empowered to act and organizations that are capable of leading them. CSAI promoted civil society advocacy through grants, training and tailored technical assistance to develop individual, project and/or institutional capacity to exert influence and represent constituency interests. CSAI also undertook efforts to strengthen the civil society sector by removing obstacles in the operating environment and helping NGO coalitions to improve the overall framework for action. CSAI framed debates at the national level on issues of broad societal concern. In this way, CSAI contributed to attitudinal change, drawing attention to new issues, and affecting the awareness, attitudes or perceptions of key stakeholders. ISC integrated four implementing partner organizations into project activities: • The Balkan Community Initiatives Fund (BCIF), • Smart Kolektiv (SK), • Civic Initiatives (CI), and • The European Center for Not-for-Profit Law (ECNL). Each implementing partner brought strengths to the CSAI effort. The project was structured into three component areas: One for grants and training to develop CSO advocacy capacity; another to improve the relationship and support for CSOs among citizens, corporations, and the state; and finally, one to address the sector enabling environment. CSAI experienced significant change over the lifetime of the project: five Chiefs of Party during seven years, as well as funding and programmatic changes. One constant throughout the program was its emphasis on advocacy at the national and local levels. CSAI has been succeeded to some extent by the subsequent Civil Society Forward (CSF) project, a two-year project also implemented by ISC that reduces its grant making and training roles. CSF enlarges the number of implementing partners to 11 organizations, including CSAI implementing partners BCIF and SK. Design, Methods and Limitations This final evaluation examines a mixture of performance and process, as can be seen in the evaluation questions. The evaluation questions on core achievements and legacies (Question A) and capacity strengthening effectiveness (Question C) address the performance aspect, while questions on management and implementation lessons (Question B) and obstacles to implementation (Question D) address the process aspect. The SI Evaluation Team was composed of Dr. Andrew Green, Senior Evaluation Specialist and Team Leader, with 20 years of experience in post-communist civil societies; Sanja Nikolin, Program Development Specialist and Serbian civil society expert; and Divna Pekic, logistician and interpreter. The Evaluation Team conducted approximately 90 semi-structured interviews (SSIs)of key informants at the implementing organizations, including direct participants, indirect beneficiaries, and external experts in Belgrade, Novi Sad, Zrenjanin, Kikinda, Leskovac, Niš, Pirot, Zaječar, and Majdanpek. The Team conducted interviews with personnel representing: • 65 of the total 237 unique grantee organizations (27%), 3 | Page • 147 of the 396 unique grants (37%), and • Over $9 million of the total $15 million in grants (62%). Every performance evaluation that relies extensively on qualitative data must deal with response, recall, and selection biases. The most effective approach to combating these biases is to use multiple sources of data to triangulate an evaluation issue, to include key informants from organizations that do not directly benefit from the evaluated program, and to use of questions about specific examples of knowledge use. In addition, attribution is difficult to make outside of experimental or quasi-experimental impact evaluations because the research team cannot isolate interventions from similar interventions or exogenous factors. Finally, an unexpected obstacle emerged during the evaluation: the Team learned during the initial meeting with CSAI leadership that ISC had planned two internal evaluations during the time period set for the USAID evaluation. It is not possible to know precisely what effect the presence of a separate evaluation effort had on the information gathered by the Evaluation Team, but it is certain that an element of ‘noise’ was introduced.1 Conclusions Given the complexity and long duration of CSAI, the Team was able to gather a large quantity of information from which to draw extensive conclusions. a) What are the core achievements and legacy of CSAI? The Team concluded that CSAI helped implementing partners BCIF, CI, and SK to grow as organizations. Regular access to CSAI grants gave implementing partners room to develop their organizational capacities and strategic visions. BCIF, in particular, has become a Serbian foundation that can fill a gap in the sector through its grants at the local level, training of its grantees, and focus on connecting communities and CSOs. The long-term access to funding, organization capacity-building, and focus on advocacy by CSAI contributed to this implementing partner growth. The Evaluation Team concluded that CSAI has increased advocacy capacity in grantee organizations, although lobbying and personal connections are still important methods for some organizations. It should be noted that ISC has helped develop an advocacy approach based on an understanding of a more policy-focused and holistic concept of advocacy and community outreach. Advocacy Fellows from organizations involved in ‘joint initiatives’ (JIs) were inspired by their study trip and made concrete adjustments to their advocacy strategies based on new knowledge and skills. CSOs have difficulty attracting significant support from citizens, corporations, and the state, and one factor in the utility and efficacy of public awareness campaigns is the extent to which citizens are receptive to the specific policy issue. Corporate philanthropy is increasing through giving and CSR partnerships. The Evaluation Team concluded that there were successful and prominent grant-funded projects, although this success was not always due to active advocacy approaches. Some of the more prominent projects that achieved some level of success include: • Open Parliament, a project to increase transparency of Government of Serbia (GoS) institutions; • Crno na Belo, which addressed workplace rights; 1 ‘Noise’ is a term commonly used in data contexts. It refers to unknown errors or variation in measurement, e.g. Nate Silver’s “The Signal and the Noise.” Measurement was likely affected in unknown ways by ISC’s internal evaluations. 4 | Page • ‘Oplaneti se!’, an environmental awareness project that mobilized volunteers nationally; • Visibly Better, an effort to improve public perception of CSOs; • The Bill on Social Housing, which included access to social housing for marginalized groups; • The Law on Associations, Law on Endowments and Foundations and Law on Volunteering, which helped reconfigure the structure of the sector; • Ministry of Youth and Sports, a new GoS ministry to address a population segment marginalized previously in state policy; • Support of Junior Achievement, a project to advocate for and support the development of an enabling environment for youth entrepreneurship; • Office for Cooperation with Civil Society (OCCS), a GoS body that serves as an intermediary between civil society and the state; • Share Conference, an effort to demonstrate to CSOs ways in which social media could improve their outreach and activities; and • Business Leaders Forum, a venue for promotion of CSR and CSO-corporation partnerships. The Evaluation Team concluded that CSAI has helped grantees grow and develop as organizations in multiple ways. First, ISC’s training on communications and public relations, as well as on financial management and reporting, challenged grantees to become better organizations and strengthened CSO capacity to reach different audiences and incorporate social media. However, the training was not always appropriate for the CSO context, as is further described in the body of the report. Training on advocacy techniques was also valuable and applied by grantees. Second, ISC grants helped strengthen existing organizations with demonstrated potential, such as Nacionalna Koalicijaza Decentralizaciju and Zajecarska Iniciativa, to develop into more capable organizations with regional capacity. Third, training vouchers were highly valued by all Advocacy in Practice (AP) grantees for their flexibility. It helped them to procure tailored training that made a difference. Fourth, ISC’s Mobile Advocacy Schools also provided needed organizational development training to a set of CSOs that were outside of its existing grantmaking sphere. Finally, BCIF was able to create and provide valued training to meet the current needs of their set of grantees and strengthen organizational capacity. b) How effective was CSAI in strengthening civil society’s capacity for and engagement in advocacy? Civil society’s capacity for, and engagement in advocacy has internal- and external-facing aspects that should be addressed for a proper response to this question. For instance, this type of capacity building requires strengthening CSOs as organizations, and strengthening CSOs as potential partners for citizens, corporations, and the state. The Team concluded that internal capacity for CSAI implementing partners, ISC grantees, and BCIF grantees was strengthened, but is also vulnerable to factors largely beyond the control of any donor-funded program. Grantees’ advocacy capacity was strengthened by ISC and BCIF grants, training, and interaction, but this capacity could be transient due to turnover among grantees. Long-term access to funding is an important factor in successful advocacy. The Team concluded that capacity to engage with external actors in advocacy efforts is improving, yet weak. CSOs are only beginning to understand how to work with potential corporate donors, and CSAI implementing partners do not have constructive relationships with important and relevant state actors beyond personal connections, including at the OCCS. c) What lessons can ISC learn from the management and implementation of CSAI? The Evaluation Team drew conclusions about the ISC-partner relationship, the grantmaking scheme, and program implementation practices on the basis of findings from our key informant interviews and project documents. 5 | Page The Team concluded that the relationship between ISC and the implementing partners was characterized more as that of a directive lead organization prioritizing project implementation, rather than a facilitating partner prioritizing sector development. Communication between ISC and implementing partners was weak and largely limited to bilateral meetings with ISC, and the potential of JIs was diminished by ISC apex coordination. Special Initiatives grants had a separate status and thus a separate dynamic within CSAI. Finally, there is no evidence that training capacity on advocacy (or other topics) in the sector as a whole has improved over the seven years of CSAI, other than what BCIF provides for its own grantees. Rather, the Team’s interviews indicate that pre-existing capacity was partially dismantled, and the subsequent selected trainers (Advocacy Resource Persons) were not used systematically and remain atomized, as is described further in the detailed report narrative. The Team concluded that changes to the grantmaking scheme by both ISC and BCIF were beneficial to CSAI, but could have been more extensive. CSAI experienced changes in grantmaking approach and improvements to past themes, including JIs replacing scattered grants for projects in similar policy issues. The changes in both the ISC and BCIF schemes, however, did not produce any meaningful distinctions between the various new ‘flavors’ of grants. Furthermore, the Team concluded that ISC used both positive and negative program implementation practices. Both ISC and BCIF conducted transparent, fair grant competitions. ISC’s direct engagement with implementing partners and potential grantees on proposals and work plans improved these groups’ strategic visions and outcomes. ISC’s adoption of such innovative approaches as training vouchers and the ‘Mobile Advocacy School’ were valued by recipients. However, the frequent turnover of ISC’s leadership limited the development of connections between ISC and implementing partners, and stunted the relationships with leading CSOs in Belgrade. d) What were the obstacles or difficulties ISC encountered during implementation of the activity? The Evaluation Team concluded that political volatility constrained the ability of CSOs to advocate and mobilize citizen engagement, and that the global economic downturn was problematic for individual and corporate philanthropy. Overall Conclusions The large set of individual and aggregate conclusions above can be synthesized to a set of overall conclusions that have implications for future civil society programming. First, ISC’s implementation of CSAI has been highly professional, from grant making processes to innovative training, to mentoring of organizations, to general organization management. CSAI’s monitoring and evaluation efforts and applied research are of high quality, and should be emulated by other civil society program implementers. Second, CSAI increased the advocacy capacity of Serbian CSOs. With particular success in some of the emerging CSOs outside Belgrade, many CSOs are now capable of playing a broader regional role, including Nacionalna Koalicijaza Decentralizaciju and Zajecarska Inicijativa. CSAI implementing partners like BCIF and SK acknowledged the various ways in which their organizations were challenged to become better and more effective. This training imparted an understanding of a more policy-focused and holistic concept of advocacy and community outreach. The capacity that grantee organizations have developed through training has often been carried forward to subsequent projects, regardless of funding source. Advocacy capacity is vulnerable to factors beyond CSAI’s control, however – which increases the importance of building training capacity and leadership at the sector level. Policy advocacy, it should be noted, is always and everywhere at risk of changing political climates. Third, CSAI has been implemented by ISC as a plan to be executed, rather than as an effort to develop the sector more generally. This can be seen in several ways. The development of sector 6 | Page leaders was hindered because ISC took a directive approach to interactions with implementing partners instead of facilitating partnerships among the organizations. More telling is the lack of any strategy by ISC to reduce its leading role as implementing partners became stronger and more autonomous. Similarly, the lack of a broader vision for the sector’s training capacity will be a significant problem in Serbia going forward, as will be the lack of capacity for grappling with nonprofit law issues. Finally, the lack of a sector leader, whether as an informal coalition or a formal organization, is highly problematic given the withdrawal of donors and EU accession policy windows of opportunity. USAID-funded projects are intended to have higher-level impacts beyond the outcomes expected from clusters of activities, and projects should be designed and implemented with respect to broader development goals. Fourth, improvement in public perception and diversification of funding sources are long-term processes. Serbia has made notable gains from a very low level of individual philanthropy, corporate engagement, and state acceptance, but all three aspects are still weak. Recommendations The Team’s ability to make recommendations applicable for the two-year CSF is constrained by the fact that this project is already underway and the implementing partners were selected ahead of time. The new CSF project already includes a more robust version of training vouchers, which would have been a Team recommendation. However, based on the overall and the earlier detailed conclusion, the Team would like to present recommendations for consideration by USAID for the Civil Society Forward project: • The structure of CSF is perhaps analogous to that of the JIs, which – along with past implementing partner concerns – raises questions about whether ISC will again act as an apex-level coordinator that inhibits relationships across implementing partners. ISC needs to allow more room for the implementing partners to work autonomously and to set an agenda themselves. ISC’s approach of coaching grantees from concept paper to the ultimate proposal submission should remain a part of its practice, but in a more advisory and less directive role. • Attention must be paid to encouraging a sector lead resource organization or network. As noted, the sector faces a daunting set of problems common to most, if not all, organizations - the lack of a coherent sector-wide training capacity, the need to grapple with continual enabling environment issues, and the critical importance of engaging the GoS during the EU accession process. The OCCS is not – and should not be – the aggregate-level representative for the sector. • Develop a strategy for how the sector can deal with long-term problems of philanthropy and public perceptions. The development of a ‘culture of philanthropy’ is a long-term process, and the following lesson of other post-communist states should be heeded: Focus public attention on how CSOs help individuals and communities, move beyond traditional concepts of deductible donations, and political and financial support for the sector will follow. The experience of grassroots grantees and ISC grantees outside Belgrade demonstrate that it is possible to tap into individual resources, usually in the form of volunteers, as well as corporate in-kind donations and state official collaboration. 7 | Page EVALUATION PURPOSE & EVALUATION QUESTIONS Evaluation Purpose The objective of this performance evaluation is to conduct a full, evidence-based, and independent review of the Civil Society Advocacy Initiative (CSAI) program implemented by Institute for Sustainable Communities (ISC), and to produce a report that provides a qualitative as well as quantitative evaluation of project activities. Evaluation Questions This final evaluation of CSAI addresses four evaluation questions covering aspects of both performance and process: a) What are the core achievements and legacy of CSAI? This can include achievements in comparison to the workplan as well as unintended results. Based on the in-brief, the SI Evaluation Team understood this question to mean activities that could be characterized as ‘important,’ ‘significant,’ and/or ‘having potential for future results.’ b) What lessons can ISC learn from the management and implementation of CSAI? Based on the in brief with USAID, the Team has understood this to be loosely defined as ‘lessons applicable to current and future civil society programming.’ c) How effective was CSAI in strengthening civil society’s capacity for and engagement in advocacy? Based on the in-brief, the Team understood that ‘capacity’ should be emphasized. d) What were the obstacles or difficulties ISC encountered during implementation of the activity? Based on the in-brief, the Team understood this to be loosely defined as ‘ambient or exogenous factors that affected the program.’ The audience for this evaluation is USAID/Serbia and Montenegro’s Democracy and Governance Office, USAID/Washington, ISC/Serbia, and ISC/USA. The findings and recommendations will inform implementation of the newly awarded Civil Society Forward (CSF) program, which is also implemented by ISC. 8 | Page PROJECT BACKGROUND Context Volatile Politics and State Relations The political landscape in Serbia was fraught with danger for democracy in 2006. The dysfunctional union of Serbia and Montenegro disintegrated after a referendum in Montenegro. The EU suspended Stabilization and Association Agreement (SAA) talks with Serbia due to a lack of progress in cooperation with ICTY. Serbia adopted a new Constitution along with a much-debated preamble on Kosovo, which was followed by extraordinary parliamentary elections in May 2007. Feudalistic division between government ministries undermined the rule of law, party-switching by MPs was not unusual, and the political scene was further roiled by a series of high-profile disputes. On the economic front, however, Serbia was ranked as the number one business reformer in the World Bank’s Doing Business Report 2006. Proceeds from privatization cushioned the country’s elites and simulated both political and economic reform successes. Much has changed over the last seven years, however. In March 2012, Serbia received full EU candidate status.2 Support for closer relations with the EU, which for many meant the pursuit of EU accession, has been one thread of continuity within changing ruling coalitions led by Kostunica’s Democratic Party of Serbia (DSS) from 2004 to 2008, and then by Tadic’s Democratic Party (DS) from 2008 to 2012. Weaknesses of these coalitions led to exploitation of political spoils, corruption, and feckless administration, rather than a shared action platform to tackle tough policy problems. In October 2008, Seselj and Nikolic supporters within the Serbian Radical Party split over EU policy spurring the creation of the Serbian Progressive Party (SNS). The 2012 parliamentary elections were narrowly won by Tadic, but political maneuvering by party leaders resulted in a coalition government headed by the Socialist Party’s Dacic; Nikolic won the presidential election in a run-off with Tadic. The rise and fall of politicians over the last seven years should have come as no surprise, given that public opinion had not changed very much – unemployment, poor living standards, and corruption remain key issues for citizens. Indeed, the EU continues to have a significant influence on the domestic and foreign policies of Serbia. Many legislative and regulatory changes are not necessarily positive, as Serbian governments have claimed that ‘the EU is making us do it’ to justify unpopular measures. This, in turn, feeds cynicism and dissatisfaction about democracy among citizens with low information levels. As of the 2012 elections, only the shrinking DSS opposed Serbia’s progress toward accession candidate status.3 Civil Society and Political Change According to the Center for the Development of the Nonprofit Sector, there were over 20,000 registered – but only 1,935 active – NGOs in Serbia in 2005.4 The sector consisted of two large groups of organizations that stemmed from very different sources: 2 Radovan Karadzic was arrested in Belgrade in July 2008 and Ratko Mladic was extradited to ICTY on 31 May 2011. 3 The official policy and main political parties both stand in favor of Serbia’s military neutrality, i.e. they dismiss the idea of NATO membership for the time being. 4 CRNPS Directory 2005. 9 | Page a) Traditional humanitarian, faith-based citizens’ and professional associations, such as for persons with disabilities, firemen and fishermen, or for the preservation of culture and tradition, or sports associations. These groups were tied to professional support and solidarity, or rural community self￾help, and were rooted in either Christian values or the Socialist era inclination to support organized leisure time and non-political activities; and, b) NGOs that emerged in the late 1980s with the introduction of a multiparty system, and evolving as anti-war civic opposition to Milosevic’s regime with a focus on social and political change.5 The latter ‘were a powerful force for bringing democratic transition to Serbia.’6 For over a decade prior to CSAI, the structure and tools of these NGOs were influenced by the American model of civic activism,7 based on the idea of a constant and productive tension between the state and civil society. For a number of reasons, donors believed that the civil society sector had matured and moved beyond a need for capacity building, broadly defined. After all, this sector played a considerable role in bringing down the undemocratic government in 2000, it spearheaded initiatives that were helping to normalize relations between Serbia and Montenegro, Croatia, and Bosnia and Herzegovina, and it contributed to progress on transitional policies and the visibility of Serbia’s human rights and democracy advocates. Public perception of civil society organizations was predominately negative. On a fundamental level, the term ‘NGO’ was closely associated with the groups that helped bring down Milosevic, but the term ‘civil society organization’ (CSO) was not in broad public use. The public image of anti-Serbian, overtly political organizations, working on behalf of outside interests rather than the Serbian people8, was fueled in no small part by the significant amount of foreign funding and technical assistance received by Otpor, Humanitarian Law Center, Lawyers’ Committee for Human Rights, CeSID and other prominent CSOs. Youth and women’s groups called for solidarity with survivors of the Srebrenica massacre – thereby creating powerful enemies of CSOs who were willing to engage in counter-campaigning work. Finally, citizens perceived NGOs to be relatively ineffective in problem solving. Despite internal weaknesses, including in CSO leadership and the emergence of ‘CSOs brands’ controlled by one person, many Serbian CSOs felt ‘entitled’ to international support. Yet donors felt less and less inclined to fund ‘brand’ CSOs, and they gradually shifted attention to organizations better equipped to respond to the new needs of Serbian citizens. Seven years after the start of CSAI, there are over 18,000 registered CSOs in Serbia, awareness of civil society has somewhat increased, and negative attitudes have decreased slightly. Aspects of the enabling environment, including the Law on Associations and the Law on Endowments and Foundations, have improved substantially. Civil society is now a social structure that is recognized by policy makers. The Office for Cooperation with Civil Society provides an institutionalized opportunity for an improved and sustained relationship with the state. However, civil society capacity remains limited. Organizations are donor-dependent and operate on a project by project basis. Donors and CSOs agree on the need to reinforce CSO outreach and representation to build legitimacy and credibility, and to strengthen governance and planning capacity in order to increase effectiveness and impact. 5 Milivojević, Zdenka, (2007), ‘Civicus Civil Society Index - A Report for Serbia,’ http://www.crnps.org.rs/wp￾content/uploads/civicus_sr2007.pdf. 6 Freedom House (2001), ‘Annual Report,’ http://www.freedomhouse.org/sites/default/files/inline_images/2001.pdf 7 Main supporters included Freedom House, Fund for an Open society, Charles Stewart Mott Foundation, Rockefeller Brothers’ Fund, along with other bilateral and multilateral donors. 8 CSAI Visibly Better Campaign. 10 | Page The ability to diversify funding is recognized as a key aspect of CSO sustainability. This includes accessing state funding at all levels through a transparent process, as well as corporate and individual philanthropy development, and adjustments to the fiscal framework to remove disincentives. As noted in the Civil Society Assessment Report,9 the handling of 481 funds – a line item in national/local government budgets for grants to CSOs – should be included as an indicator of good governance, and consideration should be given to supporting the monitoring efforts of civil society and oversight bodies such as the Anti￾Corruption Agency. Exogenous Factors The financial crisis and global recession in 2008 had a delayed but strong impact on the internal crisis in the Serbian economy. Excessive government spending and a drop in public revenues, late repayments of the obligations (arrears) at different levels of government (estimated at 0.5% of GDP),10 and a failure to engage in public administration and pension reform all contributed to a growing deficit. This resulted in a lower quality of life for citizens, which reduced public interest in issues perceived to be unrelated to household survival, employment, and socio-economic issues. As a potential candidate country, Serbia received Instrument for Pre-Accession Assistance (IPA) funding support for transition and institution-building and cross-border cooperation for the period from 2007 to 2013. This funding targets the GoS as the main assistance beneficiary. Although it is increasingly significant in financial terms, EU support to civil society is based on different parameters than the USG model. From the CSO perspective, IPA funding’s emphasis on partnership with the state, service provision, and support to institution building often results in formal and dysfunctional relationships that undercut CSO impact. This is frustrating for many CSOs, especially human and minority rights organizations. Project Purpose Prior USG-Funded Programs Since 1990, the USG-supported Freedom House (FH) Visiting Fellows Program has helped equip reformist forces in the region with the experience needed to develop the infrastructure upon which a democratic society can be built.11 From 1999 to 2006, FH supported efforts to promote democratic reform and change in Serbia. It then supported a democracy-building program consisting of grants, technical assistance, and training, as well as assignments for American volunteer experts. These volunteers provided support for public education and awareness activities, civic mobilization, and regional initiatives that transfer expertise and create networks among democratic activists in Serbia and elsewhere in Central and Eastern Europe. The FH program fostered early wins in advocacy for democracy and human rights, election monitoring, and reconciliation with the past. The centerpiece of USAID’s strategy to assist the nascent democracy emerging from the downfall of Milosevic was the $200 million ‘Community Revitalization through Democratic Action’ (CRDA) program. This program ran through 2007. CRDA was designed to promote community involvement in identifying social and economic needs, and then supply the funding to meet those needs. Infrastructure needs were quite important following years of neglect, investment shortages, and conflict, but eventually 9 Barnes, Catherine (October 2011), ‘Civil Society Assessment Report,’ prepared for USAID Serbia and Montenegro. 10 Fiscal Council, Republic of Serbia, http://fiskalnisavet.rs/en/. 11 Freedom House (2001), ‘Annual Report 2001.’ Note also that USAID provided support for civil society activities to inter alia NDI, Institute for Democracy in Eastern Europe, and the German Marshall Fund. 11 | Page income-generating projects came to dominate in response to continuing low levels of economic development outside major urban areas. USAID felt that support was needed to promote more inclusive democratic structures in order to reduce the risk of political instability.12 The initial intention for a civil society program was to support key political institutional reforms vis-à-vis advancement toward EU accession candidate status. As the subject of much internal USAID debate, the actual CSAI design that was developed placed much more emphasis on reform advocacy generally via civil society than on institutional reforms specifically via key political actors. CSAI Budget(s) and Timeframe(s) In 2006, USAID launched CSAI as a $12 million, five-year program to help civil society organizations (CSOs) better represent the needs of ordinary people, and for civil society to become a more influential and trusted partner of businesses and the government.13. The program ended on May 31, 2013 as a seven-year, $27.5 million initiative. Three core budget ceilings were approved: An additional $2.2 million for Special Initiatives funding in 2008; $4.3 million in 2009 as a response to a need for additional resources in the sector; and a $9 million, two-year period of performance extension in September 2010. The total ceiling increase amounted to $15.5 million – and exceeded the original total program value, but the program scope and objectives were not substantively modified. The Theory of Change The theory of change that appears to motivate the design of CSAI is that fundamental cultural and political change results from both citizens who are empowered to act and organizations that are capable of leading them. CSAI promoted civil society advocacy through grants, training, and tailored technical assistance to develop individual, project, and/or institutional capacity to exert influence and represent constituency interests. CSAI also undertook efforts to strengthen the civil society sector by removing obstacles in the operating environment and helping NGO coalitions to improve their overall framework for action. CSAI framed debates at the national level and through decentralized activities on freedom of association, waste management, socio-economic rights, and government transparency and accountability, and the program exerted pressure that led to inclusion of these issues on the political agenda. In this way, CSAI contributed to attitudinal change by drawing attention to new issues and affecting the awareness, attitudes, or perceptions of key stakeholders. Various sector-leading CSOs have attempted to encourage discursive commitment from NGOs and policy actors by replacing the term ‘NGO’ with the term ‘CSO,’ and affecting language and rhetoric in 12 USAID/Serbia and Montenegro designated CSAI as one of the core programs under Strategic Objective (SO) 2.11 (‘Risk of Political Instability Reduced’) within the Mission’s strategy for 2006 to 2010. The SO focused on the support of key democratic structures and processes at all levels nation-wide to increase political stability; it also looked to improve conditions in specific, vulnerable areas, by increasing economic opportunities, civic participation and community interventions. In 2010, USAID/Serbia and Montenegro amended the strategy to cover the period of 2011 to 2015, although the Mission’s overall strategic objective of supporting Serbia in its vision to be ‘democratic, prosperous and moving toward Euro-Atlantic Integration’ remained the same. 13 Institute for Sustainable Communities (2013), ‘Civil Society Advocacy Initiative: Who We Are,’ http://csai.iscserbia.org/who_we_are/. 12 | Page order to both promote greater recognition of the civil society sector and mitigate negative associations implicit in the term NGO. There is no evidence that this approach generated the anticipated returns. CSAI, through ECNL and Civic Initiatives, sought to secure policy and procedural changes, including opening new spaces for policy dialogue. At the same time, the program exerted influence on policy content. The CSAI approach evolved over time, and it finally led to ‘Coalition’ Theory, or the Advocacy Coalition Framework,14 in which policy change happens through coordinated activity among a range of individuals with the same core policy beliefs. CSAI Structure ISC integrated four implementing partner organizations into project activities: the Balkan Community Initiatives Fund (BCIF), Smart Kolektiv (SK), Civic Initiatives (CI), and the European Center for Not-for￾Profit Law (ECNL). Each partner brought strengths to the CSAI effort. BCIF was ISC’s implementing partner for grantmaking, but was also expected to play a leading role in promoting individual and corporate philanthropy. The relatively young Smart Kolektiv organization specialized in communications and social marketing, and was expected to play a role in numerous activities, as appropriate. CI was a well-established organization that had long been a sector resource center, and focused on national advocacy initiatives and youth. CSAI has three main components (or results): Result 1. CSAI grantees advocate effectively about political, economic, and social issues that consolidate Serbia’s democratic transition. There are two distinct yet intertwined activities in this component – grants made by ISC and sub-grants by BCIF, and the training that both organizations offered to grantees. ISC grant types were:  Implementer Grants, intended for the four implementing partner organizations  Advocacy in Practice (AP) Grants, intended for existing or emerging CSOs engaging in policy advocacy on targeted issues  Innovation Fund (IF) Grants, intended for CSOs engaging in policy advocacy on new issues or using new techniques to increase citizen participation  Success Grants (SG), intended to assist existing ISC grantees with resources to raise awareness of the CSO and its noteworthy project achievements  European Integration (EurInt) Grants, intended for CSOs engaging in public advocacy on the advantages of EU accession  Joint Initiative (JI) Grants, intended for coalitions of CSOs to advocate for specific policy reforms in government accountability and transparency, socio-economic rights, and environmental protection  Special Initiatives (SI), intended as a flexible and opportunistic funding resource for CSOs engaged in political reform advocacy 14 Stachowiak, Sarah (2007), ‘Pathways for Change: 6 Theories about How Policy Change Happens,’ Organisational Research Services, http://www.organizationalresearch.com/publicationsandresources/pathways_for_change_6_theories_about_how_p olicy_change_happens.pdf 13 | Page BCIF sub-grant types were:  Grassroots Grants (GRAG), intended for small and even informal CSOs engaged in policy advocacy at the local level  Green List (GL) Grants, intended for small CSOs engaged in policy advocacy on environmental protection issues at the local level  Advanced Community Advocacy Program (ACAP) Grants, intended for medium or more established CSOs to engage in policy advocacy at the local or regional levels  Proficient Community Advocates Program (PCAP) Grants, intended for a select group of existing grantees to engage in follow-up projects  Successful Fundraising (SF) Grants, intended to challenge a select group of existing grantees to raise funding from individual, corporate, or state sources that would be matched ISC offered training to grantees on public advocacy techniques and strategies, financial management and reporting, communications and public relations, media outreach, budget analysis and advocacy, among others. ISC also instituted innovative ‘Mobile Advocacy Schools’ that offered a variety of training topics to small and informal organizations throughout Serbia that were not grantees. BCIF offered training to grantees on public advocacy, financial management and reporting, project management, and fundraising, among others. Result 2. Citizens, businesses, and government view CSOs as reliable partners in building a more democratic and prosperous Serbia. For this Result there were a number of distinct activities that were carried out in four areas:  CSO sector visibility, most notably the ‘Visibly Better’ campaign to raise awareness and positive citizen perceptions of CSOs  Promoting individual philanthropy, involving public awareness campaigns and legislation advocacy  Promoting corporate philanthropy and partnerships, involving the promotion of strategic corporate philanthropy promotion, high profile public events, and the establishment of a business-led forum to promote best practices in corporate-CSO partnering  Enhancing state-CSO cooperation, involving advocacy for government units dedicated to CSO engagement, transparency in state funding for CSOs, and more active engagement between government agencies and CSOs Result 3. An enabling environment exists for CSOs to advocate effectively for reform issues. Serbia lacked significant or adequate legislation supporting the development of a civil society sector, such as laws on associations, foundations, tax deductibility, economic activity concessions, and other related topics. CSAI efforts to improve the enabling environment built on past activities of implementing partners, e.g., CI actions to revise the Law on Associations and ECNL technical assistance under previous USAID programs. Project Implementation CSAI experienced significant change over the lifetime of the project. One important area of change was in the project’s leadership: CSAI had five Chiefs of Party during seven years, including one with no prior experience in post-communist settings. At the same time, the CSAI approach changed over time to accommodate funding and environment changes. One constant throughout the program was the emphasis on advocacy. The program supported large and small organizations that focused on local and 14 | Page national issues. Over time, the program started to evolve with grantees as it enabled them to grow, develop and form partnerships. CSAI has been followed by Civil Society Forward (CSF), a two-year project also implemented by ISC. CSF enlarges the number of implementing partners to 11 organizations, including CSAI implementing partners BCIF and SK, as a bridge to potential future direct support. The goals of CSF are to (a) support national-level advocacy efforts on issues critical to EU accession; (b) promote local advocacy and civic engagement at the grassroots level; (c) strengthen the sustainability and effectiveness of the implementing partners; and (d) increase the level of local philanthropy. 15 | Page EVALUATION METHODS & LIMITATIONS This final evaluation examines a mixture of performance and process, as can be seen in the evaluation questions. The evaluation questions on core achievements and legacies (Question A) and capacity strengthening effectiveness (Question C) address the performance aspect, while questions on management and implementation lessons (Question B) and obstacles to implementation (Question D) address the process aspect. The SI Evaluation Team is composed of Dr. Andrew Green, Senior Evaluation Specialist and Team Leader; Sanja Nikolin, Program Development Specialist and Serbian civil society expert; and Divna Pekic, logistician and interpreter. The evaluation of CSAI started with a Team Planning Meeting on 14 May, followed by the review of important documents and drafting of a desk study prior to beginning fieldwork. Expatriate team member Andrew Green arrived in Belgrade on 19 May and participated in an in-brief with USAID/Serbia & Montenegro staff. Following these meetings, the Evaluation Team began data collection, which included the use of SSIs with key informants, group interviews, materials review, and mini-surveys. The relevant protocols and planned mix of evidence sources are detailed in the annexes (refer to Annex II and Annex III). The Team collected data for approximately 14 business days, and then analyzed the data, prepared a report outline, and produced a draft presentation for USAID/Serbia & Montenegro. Site visits were conducted in Novi Sad, Zrenjanin, Kikinda, Leskovac, Niš, Pirot, Zaječar, and Majdanpek in addition to Belgrade. The sites were selected non-randomly based on USAID guidance on the first evaluation question. The Evaluation Team presented its findings to USAID in an out-brief meeting on 7 June, before Andrew Green departed Belgrade on 8 June. The Evaluation Team conducted approximately 90 SSIs of key informants with the implementing organizations, direct participants, indirect beneficiaries, and external experts. The Team conducted interviews with personnel representing:  65 of the total 237 unique grantee organizations (27%),  147 of the 396 unique grants (37%), and  Over $9 million of the total $15 million in grants (62%). The interview set was selected non-randomly, and the Team endeavored to conduct a set of interviews that was representative of the total grant amount, number of grants, and annual grant amounts (see Tables 1-3 in Annex VI). In addition, the Team sent mini-surveys by email to non-interviewed BCIF grantees and Mobile Advocacy School participants. The essence of evaluation is comparison—typically across time or geography, better still across both. An impact evaluation further attempts to compare across units that have or have not received some sort of programming treatment. While CSAI has collected extensive monitoring and evaluation data, including applications of scorecards at multiple points in time, the extensive coverage, long timeframe, and other activities of partners of the program excluded the possibility of even ex post facto quasi-experimental approaches. As with any evaluation based predominantly on qualitative data, there are some noteworthy limitations on comparison, and hence attribution, that should be anticipated. 16 | Page First, recall bias may be present, for example, as grantee CSO staff respond to Evaluation Team questions with answers related to grants from another donor. A similar problem is that participants in multiple training activities may blend their experiences into a composite memory or response (such as CSO staff who may have received training on advocacy and on fundraising), and subsequently do not distinguish between them as separate activities. Second, response bias is a common problem for program evaluations. For example, CSO grantees may give the interviewer positive remarks about the project because s/he would like to receive another grant in the future. The Evaluation Team fully expects that CSO grantees, trainees, and implementing partners may understand that a negative evaluation could mean the end of project opportunities, or may fear that their comments may be identifiable and thus harm ongoing relationships. Third, selection bias in the form of contacts provided by the implementers can mean that the Evaluation Team only hears from people with positive experiences. The most effective approach to combating these biases is to use multiple sources of data to triangulate on an evaluation issue, as is often accomplished through qualitative reliability matrices. Combining information found in documents or interviews from multiple sources prevents any one piece of biased data from skewing the analysis. Another approach that pertains specifically to interviews is including key informants from organizations that do not directly benefit from the evaluated program, and asking questions about specific examples of knowledge use. Attribution is difficult to make outside of experimental or quasi-experimental impact evaluations, because research teams cannot isolate interventions from similar interventions or exogenous factors. For this evaluation of CSAI, the Evaluation Team notes that during the seven years of the program, Serbia welcomed civil society programs by other donors, and experienced significant political volatility, economic recession, and re-shaping of the state borders. For a program designed to address issues of advocacy for policy change by civil society and citizens, these are variables likely to affect the performance of the program. The Evaluation Team’s biggest concern stemmed from the availability of contacts at BCIF grantees, as well as ISC grantees in the first years of CSAI. The Evaluation Team anticipated that it would be difficult to schedule time with the staff from smaller, volunteer-dominant CSOs, and so the amount of evidence that could be collected from any one key informant would be less than desirable. Similarly, the Team anticipated that response rates to a mini-survey request might be low. While BCIF grantees and early ISC grantees were more available than the Team had anticipated, the response rates for the mini￾surveys of BCIF grantees and Mobile Advocacy Schools was, in fact, low at 21% (15/71) and 27% (51/193), respectively. Unlike the standard evaluation context, the Evaluation Team learned during the initial meeting with CSAI leadership that ISC had planned two internal evaluations during the time period set for the USAID evaluation, in full knowledge of the timing and high priority of the USAID evaluation. Both internal evaluation topics were covered also by the USAID evaluation: One evaluation focused on ISC’s Mobile Advocacy Schools, and was conducted before the Evaluation Team began field work; the other evaluation focused on ISC’s advocacy approach, and was conducted wholly during the Evaluation Team’s period of field work. Indeed, for the latter, the internal team’s desired interviewee list overlapped to a significant extent with the Evaluation Team’s, which produced confusion and irritation among some of the desired interviewees the Team contacted, as well as a competition for availability and resulting logistical difficulties for the Evaluation Team. The main methodological concern, of course, is that the Team does not (and cannot) know how this affected responses by mutual interviewees:  Recall bias – A prior interview with the internal team, or even the scheduling of such an interview prior to an interview with the USAID Evaluation Team, may have improved recall, may have focused 17 | Page interviewee recall on a more limited set of aspects than the Team desired, or may have induced interviewee fatigue on the topics generally.  Response bias – It is possible that a prior interview by the internal team provided an opportunity for interviewees to rehearse responses to certain topics, or that the USAID Evaluation Team’s interview provided that opportunity for interviewees interested in maintaining good relations for the follow-on CSF program. Ultimately there is no way to know precisely what effect the presence of a separate evaluation effort had on the information gathered by the Evaluation Team, but it is certain that an element of ‘noise’ was introduced to the data collection context.15 15 ‘Noise’ is a term commonly used in data contexts. It refers to unknown errors or variation in measurement, e.g. Nate Silver’s “The Signal and the Noise.” Measurement was likely affected in unknown ways by ISC’s internal evaluations. 18 | Page FINDINGS, CONCLUSIONS & RECOMMENDATIONS Findings and Conclusions This section presents the Team’s findings and conclusions based on document review, SSIs, mini-surveys, and other research results. Note that this report first groups together the two performance evaluation questions, and then the two process evaluation questions. Evaluation Question 1: What are the core achievements and legacy of CSAI? This can include achievements in comparison to the workplan as well as unintended results. Based on the in-brief, the Team understood this to mean activities that could be characterized as ‘important,’ ‘significant,’ and/or ‘having potential for future results.’ The Evaluation Team concluded that CSAI helped implementing partners BCIF, CI, and SK grow and develop as organizations:  Regular access to CSAI grants gave implementing partners room to advance as organizations.  BCIF, in particular, has become a Serbian foundation that can fill a gap in the sector through its grants at the local level, training of grantees, and focus on connecting communities and CSOs. CSAI’s advocacy focus contributed to this growth. The Team found that BCIF, CI, and SK acknowledged becoming more capable organizations with respect to financial management, internal procedures, and/or communications and public relations. Although CI was a leading resource center at CSAI inception, it took the first program year to align their work plan and reporting with USAID requirements. Smart Kolektiv was the newest organization, and it experienced many staff changes among non-key personnel. Thus, Smart Kolektiv required very close monitoring by ISC in order to fulfill both the administrative and programmatic requirements. Implementing partner ECNL started off as a niche, expert organization, independent in its operation, and it did not require much support from ISC. BCIF had previous grantmaking experience but benefitted from ECNL expert analysis and support in advocacy and lobbying for the Law on Endowments and Foundations. Alone, BCIF was not able to access the Ministry of Finance, and with CI and ECNL support it could. These three implementing partners, the Team found, valued how the long-term access to funding provided room for them to develop as organizations, and exposed them to new ideas and approaches over a longer period of time than a normal grant project. For example, while BCIF has long preferred to work at the grassroots level, CSAI’s clear focus on advocacy for legal and policy changes shifted BCIF’s grantmaking in a new and complementary direction. In addition, ISC worked with BCIF to expand and enhance training capacity, particularly with respect to advocacy, for its grantees. The prior BCIF advocacy program established with Freedom House and BTD support targeted smaller organizations with an emphasis on civic participation and awareness-raising. ISC developed BCIF’s capacity to work 19 | Page with mid-level capacity organizations – and improved their ability to deliver capacity building services,16 including programmatic, fundraising, administrative and outreach support to grantees. Public advocacy and successful fundraising training programs are now part of a standard BCIF offer. At the same time, the Team found that CSAI funding was only a small percentage of the three organization’s funding, with the exception of Smart Kolektiv. CSAI funding only made up about 20% of BCIF’s total funding, but reliable funding access allowed BCIF to think more strategically about activities lasting longer than one year. Implementing partner ECNL is a Budapest-based international organization that specializes in nonprofit law, providing technical assistance in this niche area for partners in post-communist countries throughout the region. The Evaluation Team concluded that CSAI has increased advocacy capacity in grantee organizations, although lobbying and personal connections are still important methods for some organizations, and CSOs face difficulties attracting support for their advocacy efforts beyond a shrinking set of foreign donors:  ISC has helped develop advocacy capacity, as well as a more policy-focused and holistic concept of advocacy and community outreach.  Advocacy Fellows from the JIs coalition CSOs were inspired by the study trip and made concrete adjustments to their advocacy strategies on the basis of new knowledge and skills.  CSOs have difficulty attracting any type of support from citizens, corporations, and the state.  Corporate philanthropy is increasing through giving and CSR partnerships. The Team found that interviewed grantees understood advocacy to be grounded in engagement of citizens through a wide variety of passive and active means. Interviewees reported utilizing such techniques as street events, roundtables and multi-stakeholder public discussions, appearances in local and national media, and new media campaigns. However, organizations as diverse as the Narodna Koalicija, Resource Center Majdanpek, Center for Regionalism, and CeSID and CRTA reported that lobbying and personal connections remained key advocacy strategies for policy changes. The Team interviewed nine Advocacy Fellows from CSOs participating in JIs, and found that the Fellows valued the study trip for the training on advanced advocacy techniques by Olga Gladkikh, and the opportunity to meet counterparts at US organizations. Grantees like the Humanitarian Law Center and the Autonomous Women’s Center pointed to a political environment that was not receptive to messages about Serbian war crimes or even gender￾based violence. For them, public awareness campaigns were not assessed to be useful in the foreseeable future. Advocacy capacity includes knowing how to reach out to citizens and the private sector for political and financial support. Indeed, the Team found that all but two interviewees explained the poor public perception of CSOs as stemming from a lack of philanthropy culture and the tainted ‘NGO’ term. The graph in Figure 1, from a 2009 CI survey,17 demonstrates that positive and negative perceptions of the 16 In parallel, BCIF was implementing a Cooperative Netherland Foundation-(CNF) funded social transition program that constituted 30% - 40% of BCIF budget, while ISC contributed 20% on average. CNF grants ranged from 15,000 – 20,000 Euro. 17 Civic Initiatives (2009), ‘Public Perception and Attitudes Towards NGO Sector in Serbia in 2009,’ ISC and USAID. 20 | Page sector are of similar weight among citizens and did not shift much between 2006 and 2009. Further, as Figure 2 shows, CSOs are not viewed as particularly effective problem solvers in absolute or relative terms, which suggests that much of this particular view of the sector is driven by factors beyond the sector’s control. Unfortunately, there is no comparable recent research to extend this timeline, although a 2013 CRTA/NDI study18 indicated that more than a third of Serbian citizens feel NGOs are foreign agents. The Team also found from a BCIF 2012 study on philanthropy in Serbia and neighboring states that Serbia has the lowest level of giving.19 Almost 73% of citizens think that people in Serbia set aside too little for public good, a level that increased from 66% in 2009 research. There remains strong consensus among companies that philanthropic giving is not incentivized adequately through tax policy – increasing from 81% in 2009 to 90% in 2012 among companies with an existing practice of giving. The main reasons for the low level of giving are poor economic conditions (56%), lack of awareness on public good (22%), and lack of interest (22%). Companies and citizens agree that the business sector is insufficiently involved in corporate giving. Every fourth company says that the business sector is not included enough and 74% think that it is included to an extent, but not as much as it should be. The main reasons include a widely spread lack of interest within the society (65%). Companies, more than citizens, see poor financial conditions (57%) and unfavorable legislation (56%) as leading reasons for lack of corporate involvement. A significant number of companies also emphasize insufficiently transparent flow of money (45%) and lack of trust in organizations that conduct actions for public good (39%). The Team found that corporate philanthropy has been increasing, but in nascent or unclear ways. The BCIF-established VIRTUS Award for corporate philanthropic giving, based on the Czech model, may have encouraged a competitive dynamic at participating corporations: the team had a single report from a former key personnel at Holcim recounted that the Serbian director of Holcim closely monitored activities that would help the company stand out in its CSR efforts. Additionally, BCIF encouraged the development of the Serbian Philanthropic Forum, which is supported and run primarily by business leaders from participating corporations. Corporate giving may be increasing, but data are lacking because corporations are either not transparent about giving or they include such funding under marketing expenses. However, CSOs have demonstrated the ability to raise funds from corporations when the CSOs are trained and incentivized. For example, BCIF’s Successful Fundraising grants, which matched private donations, granted $36,097.81 and equivalent cost-sharing. The Evaluation Team concluded that there are grant-funded projects that were successful and prominent, although not always through active advocacy approaches: CSAI awarded 396 grants totaling almost $15 million over a seven-year period. Not all grant-funded projects were successful, but the Team found that almost all of the non-SI projects were required to explicitly incorporate advocacy strategies in their project plans. 18 Supported by British Embassy Belgrade and conducted by Ipsos Strategic Marketing, February 2013. 19 BCIF, ISC and USAID (December 2012), ‘Individual and Corporate Philanthropy in Serbia: Practices and Attitudes of Citizens and Company Representatives,’ Ipsos Strategic Marketing, http://www.tragfondacija.org/media/PDF/BCIF%20-%20Research%20on%20philanthropy.pdf. 21 | Page Figure 1. Findings from the 2006 and 2009 Civic Initiatives (CI) survey of public perception and attitudes toward NGO sector in Serbia Figure 2. Civic Initiatives (CI) survey of public opinion on institutional effectiveness 22 | Page  Open Parliament Open Parliament is a project mainly manifested in the form of a website that brings together digitization of parliamentary transcripts and cross-comparison of MP seating and votes in order to increase Parliament’s transparency. This has been an important achievement, especially since it contributes to the work of journalists and advocacy groups. This ISC-initiated coalition of five NGOs dropped to four CSOs with complementary skills. Since June 2012, the Open Parliament web site has had 160,000 hits. After their study tour to the US, the coalition sought to increase impact by putting more effort into educating stakeholders about the use of the mechanisms created.  Crno na Belo This effort supporting workplace rights is in some senses successful from the mere fact that this issue had largely been ignored by media, CSOs, and the state, and yet addressed workplace issues that are widely known and experienced in Serbia. Crno na Belo was a multifaceted project combining public awareness campaigns, street events, and direct assistance to workers through monitoring materials and free legal aid. The inability to engage trade unions under the ISC grant was one of their greatest frustrations, however. The study trip to the US helped them to convince ISC of the necessity for legal aid and to learn about effective constituency building.  Oplaneti se! Oplaneti se! was implemented from 2009 to 2012 as a national ecological campaign aimed at increasing recycling, solving the garbage disposal problem in Serbia, and improving primary waste selection. This campaign was very successful in mobilizing volunteers through 16 participating CSOs.  Visibly Better The ‘Visibly Better’ project was a public awareness campaign intended to provide citizens with a more accurate view of the sector and its activities. While ISC views the ‘Visibly Better’ campaign as a success, and it was in the sense of pushing key actors in the sector to engage in a public awareness campaign, the Team could not locate evidence it had any impact on public attitudes generally. The starting point could be seen in the 2009 CI survey discussed above. According to a recent CRTA/NDI survey,20 there has been little change in public attitudes about CSOs over the last seven years. Only 17% of citizens believe that active citizens can change things in their local communities, and only 10% think that it is feasible at the national level. One particularly unfortunate perception that persists in a substantial portion of the Serbian population is shown below in Figure 3 – that NGOs are, in essence, agents for foreign influence. 20 Supported by British Embassy Belgrade and conducted by Ipsos Strategic Marketing, February 2013. 23 | Page Figure 3. Results from a February 2013 National Democratic Institute (NDI) survey on public attitudes toward CSOs. It is possible that public opinion did not change on net, as increases in positive perceptions from the public awareness campaign may have been offset by increases of negative perceptions from the perceived corruption of party-affiliated CSOs.  Bill on Social Housing Group 484 used CSAI as an opportunity for a strategic shift from their focus on refugees and IDPs to migrants and vulnerable groups. The law passed in 2009 did not result entirely from CSAI-funded effort, according to Group 484. However, key provisions resulted from CSAI advocacy, such as a list of vulnerable groups. Unfortunately, the law has still not been implemented and agencies for social housing are not functional yet. Group 484 is no longer focusing on social housing, but is now engaged on the process side at the local level.  Laws on Associations, Endowments and Foundations, Volunteering The Team found that the 2009 Law on Associations was a clear improvement on the existing outdated and inadequate law, and emerged from a process begun prior to CSAI under the leadership of CI. The law allows for much easier registration and more accurate sectoral data. Unlike many other laws in Serbia, this one is consistently implemented. This is due at least in part to ECNL training for Business Registry personnel and Ministry of Public Administration and Local Self-Government. However, many respondents now see a need to modify the Law. 24 | Page General dissatisfaction levels with legal regulations were captured by a 2011 study by Civic Initiatives and the Office for Cooperation with Civil Society.21 Figure 4. Opinions of CSO Respondents on Legal Regulations, 2011 Civic Initiatives Survey The 2010 Law on Endowments and Foundations was also passed through the cooperative efforts of BCIF and ECNL, providing a constructive legal status for foundations for the first time. The Law on Volunteering was passed, but in a form that was significantly different than what nonprofit law experts had proposed. The Team found that the Law’s requirements for corporations or CSOs seeking to sponsor volunteer-based activities are so onerous that organizations actively seek ways to avoid the Law entirely. ECNL actually designed guidelines for CSOs on ways to avoid the Law. As noted by the Novi Sad Humanitarian Center, this may have been an example where advocacy efforts ended too soon – i.e. as soon as the draft was accepted by Vojvodina provincial secretariats and placed on the agenda of the National Parliament. After three years of implementation, there are no data on impact and/or obstacles. However, all respondents agreed that this law needs to be amended soon.  Ministry of Youth and Sports The Ministry of Youth and Sports was also a clear goal of Civic Initiatives, and one that is more clearly attributable to their advocacy efforts. It was created in May 2007, after four years of lobbying and advocacy by CI-led youth groups. The Ministry has been perhaps the most cooperative ministry for CSOs. The GoS National Youth Strategy, National Youth umbrella organizations, Law on Youth and capacity building of 70 +Youth offices complemented this early advocacy success.  Support of Junior Achievement (JA) A CSAI Special Initiatives grant supported JA to help young people become more competitive in the labor market by strengthening their personal and professional skills. Support of the Partnership@Work program also helped to develop cross-border relations between Serbian and German high-school students. JA received multiple grants from different USAID-funded programs. JA is a USAID, and partially CSAI, legacy of accredited entrepreneurship development training through the Ministry of Education, the first and only such program of its kind in Serbia. It continues to successfully operate in Serbian primary and secondary schools and advocate for an enabling environment for entrepreneurship education in Serbia.  Office for Cooperation with Civil Society (OCCS) 21 Civic Initiatives and Office for Cooperation with Civil Society (September 2011), ‘Assessment of Conditions in Civil Society Organizations and the Civil Society Sector in Serbia, September 2011,’ ISC and USAID. 25 | Page After launching an initiative in 2008, Civic Initiative and ENCL claimed the establishment of the government Office for Cooperation with CSOs in 2010 as an advocacy victory. In 2011 the Office received 4 million RSD from the state budget (less than €50,000), but significant funding comes from foreign donors for its operations: the EU Delegation pledged €1.2 million for the three-year period of November 2012-November 2015. The Office aims to expand access for CSOs to interact with government in policy making contribute to enabling environment for CSOs, and allow for greater transparency of financial and policy decision making. The Team found that the Office was often acting on behalf of civil society, e.g., ‘representing’ civil society at ministry working groups, which is not part of its mandate to facilitate CSO engagement with GoS agencies.  Share Conference Within the ‘New World’ capacity building platform for CSOs to learn about effective use of new media and ICT for advocacy, CSAI supported two Share conferences in 2011 and 2012 that were implemented by a spinoff of EXIT Association, Share Foundation. These conferences were among the most important IT industry events in the region and were also transferred as a model to Beirut in 2012. Results include emergence of a new generation of advocacy methodologies, and technology in the service of citizens and communities. Digital activism, open software development, government transparency and accountability, privacy protection, and freedom of speech are emphasized. Through contacts between local professionals with leading companies and institutions such as Google, Facebook, Tweeter, Vimeo, Harvard, MIT, Stanford, Oxford and others, these events inspired audiences beyond civil society. However, sustainability of the initiative between conferences is an issue.  Business Leaders Forum Smart Kolektiv established the Business Leader’s Forum (BLF) to promote corporate-CSO partnerships. The BLF has increased membership, promoted the idea of CSR, and engaged in active sharing of knowledge and best practices. SK’s role as facilitator was highly valued by all interviewed BLF members. BLF Board members view SK’s role as very important to future development of strategic CSR. They perceive the value added of BLF as a more pro-active grouping, compared with the UN Global Compact, for example. The limitations include a low demand for CSR and few actual strategic partnerships between CSOs and corporations. However, BLF is definitely a solid base to build upon in the future. The Evaluation Team concluded that CSAI had helped grantees to grow and develop as organizations:  ISC’s training on communications and public relations, as well as financial management and reporting, challenged grantees to become better organizations and strengthened CSO capacity to reach different audiences and incorporate social media. However, the training was not always appropriate for the CSO context, as is explained in the text below. Training on advocacy techniques was also valuable and was applied by grantees.  ISC grants helped strengthen existing organizations, such as Nacionalna Koalicija za Decentralizaciju and Zajecarska Iniciativa, with demonstrated potential to develop into more capable organizations with capacity to act regionally within Serbia.  Training vouchers were highly valued by all Advocacy in Practice grantees for their flexibility. It helped them to procure tailored training that made a difference.  BCIF was able to create and provide valued training to meet the needs of their set of grantees and strengthen grantees’ organizational capacity. 26 | Page The Team found that almost all grantees valued the training they received in financial management and reporting, advocacy, and communications and public relations, and applied the skills and knowledge in practice. Incorporating the lessons of trainings was a challenge to organizations, but all interviewed organization leaders pointed to ways in which their organizations became better and more professional. Both ISC and BCIF grantees acknowledged the complexity of financial reporting, for example, but organizations like Group 484, Center for Democracy Foundation, Center for Regionalism, and Zrenjanin’s Educational Center pointed out that they adopted these practices for every project, regardless of donor requirements. Similarly, the Team found that BCIF grantees valued and applied skills from trainings. Two organizations, Nacionalna Koalicijaza za Decentralizaci and Zajecarska Inicijativa, increasingly engaged with partners throughout their region of Serbia, with the latter moving to establish a ‘regional’ foundation. Several respondents, including a women’s group against domestic violence, stated that trainings and coaching conducted by Represent Communications were not always relevant to CSOs. Another grantee made a point about a communication training held by a leading Serbian journalist who ‘was out of touch with NGOs’. Grantees felt that such events were more useful as two-way exchanges with the cultivation of a follow-on relationship where both sides learn about one another. A finding from the Team’s mini￾survey of other BCIF grantees under CSAI was that 80% (12/15) could name specific follow-on projects that benefitted from BCIF training. A related finding is that AP grantees have advocacy skills and are using them in other projects. Organizations like Nacionalna Koalicijaand Educational Center Zrenjanin cited how they used stakeholder analysis in subsequent projects. These and other organizations referred to cases of employing budget analysis to enhance their advocacy for ‘Line 481’ transparency. Resources were dedicated to communications and financial management by such organizations as the People’s Parliament and Nacionalna Koalicija. Tools and techniques benefited even those groups that had some level of advocacy skills already. ISC AP grantees prized the training vouchers, because this mechanism’s flexibility allowed grantees to procure training and technical assistance that really needed, rather than topics simply chosen by an implementer. For example, the Open Parliament coalition procured strategic planning that they claimed made a difference in aligning their partner’s vision and clarifying roles. Center for Democracy Foundation qualified locally procured training and consulting on outreach and communication as ‘probably the most useful support ever’. The Team found that ISC’s Mobile Advocacy Schools provided needed organizational development training to a set of CSOs that were outside of its existing grantmaking sphere. The Team’s mini-survey of MAS participants demonstrated that MAS training served a real need that was not being met otherwise. As many as 35 out of 53 CSO (66%) respondents were able to state a specific follow-on project where they applied new knowledge and skills gained through MAS. Half of the responding participants indicated that they submitted follow-up proposals to BCIF for advocacy projects, with 60% of those succeeding. 27 | Page MAS training support by training area # % Public advocacy 43 81.1% Finance management and reporting 28 52.8% Communication and media 31 58.5% Project management 36 67.9% Proposal development, fundraising 42 79.3% Other, please specify 9 17.0% Figure 5. Types of Training Offered in Mobile Advocacy Schools (MAS) Evaluation Question 3: How effective was CSAI in strengthening civil society’s capacity for and engagement in advocacy? Based on the in-brief, the Team understood that ‘capacity’ should be emphasized. Civil society’s capacity for and engagement in advocacy has internal and external aspects that should be addressed for a proper response to this evaluation question. The Team concluded that internal capacity for CSAI implementing partners, ISC grantees, and BCIF grantees has increased, but is vulnerable to factors largely beyond the control of any donor-funded program:  Grantees’ advocacy capacity was strengthened by ISC and BCIF grants, training, and interaction, but this capacity could be transient.  Long-term access to funding is an important factor in successful advocacy. With the exception of almost all of the interviewed Special Initiatives grantees, both ISC and BCIF grantees reported to the Team specific examples of how their capacity for advocacy has been strengthened, their organizations and activities had changed as a result, and that the skills have been applied to subsequent projects regardless of the funder. Examples include all three joint initiatives’ members, Green List partners, as well as BCIF grantees Plava Ptica, Iskra Loznica, Porecje Vucje, and EP ‘Moravski orašak’ Trstenik. However, smaller organizations cannot sustain this capacity due to turnover and disuse. Capable staff leave the sector for more reliable and higher-paying jobs and organizations chase whatever projects are available in order to survive financially. Sustaining capacity may even be a problem for implementing partner Smart Kolektiv, which has demonstrated strengths in pulling together potential philanthropy partners. The organization itself, however, may be reduced to Neven Marinovic alone, according to two BLF members and two AP grantees. As is true for implementing partners, long-term funding or access to funding on a regular basis is needed for advocacy capacity to be used effectively because advocacy for legislative or other substantial policy change may take longer than one year to work. This issue was raised by the Center for Regionalism, Fokus M, Ck13, Novi Sad Humanitarian Center, and Zrenjanin's Educational Center. 28 | Page The Team concluded that capacity to engage with external actors in advocacy efforts is improving yet weak:  CSOs are only beginning to understand how to work with potential corporate donors.  CSAI implementing partners do not have constructive relationships with important relevant state actors beyond personal connections, including at the OCCS. Efforts by BCIF and SK to promote engagement between corporations and CSOs have produced corporate giving strategies, direct philanthropic giving, and partnerships, which the Team found in project documents, interviews with both organizations, and interviews with individuals from Holcim, B92 Foundation, Erste Banka, and Coca-Cola. However, no interviewed CSO had a clear strategy for how to engage corporations. They merely expressed vague aspirations of contacting unnamed firms to gauge interest in their activities. Moreover, it is not clear that CSOs know how to pursue corporate philanthropy when it is available. An anecdotal example of this low capacity was one BLF member that reported to the Team that a 2012 proposal process demonstrated the low proposal writing capacity of CSOs. In addition, interviewees from two grantees, one corporation, and two external experts cited an ‘arrogant’ approach of CSOs when interacting with corporations and even in public awareness campaigns, i.e., a ‘we’ll show you how this should be done’ approach. Equally important was the Team’s finding that even well-established CSOs, such as CI, YUCOM, Humanitarian Law Center, and even the Autonomous Women’s Center, rely heavily on lobbying and personal connections, not public advocacy. The OCCS should be an intermediary for the sector, but even here the Team found that engagement with OCCS was based primarily on personal connections. Important state actors like OCCS do not have the capacity or resources to cooperate more actively on a general level with CSOs. However, local officials may represent better potential for cooperation if a level of trust and credibility has been established with leading local CSOs. Evaluation Question 2: What lessons can ISC learn from the management and implementation of CSAI? Based on the in-brief, the Team understood this to be loosely defined as ‘lessons applicable to current and future civil society programming.’ The Evaluation Team drew conclusions about the ISC-partner relationship, the grantmaking scheme, and program implementation practices on the basis of findings from our key informant interviews. CSAI had an extensive M&E plan, and gathered considerable data on organizations during the seven years of the program. Unfortunately, frequent changes in leadership and uneven application of scorecards to organizations at different points in time mean that the data are not as useful for analytical purposes as needed. For example, organizations that are scored more than once for the Advocacy Index experienced unexplained setbacks between years, new baseline scores at points, and other discontinuities. The Team concluded that the relationship between ISC and the implementing partners was characterized more as a directive lead organization prioritizing project implementation, rather than a facilitating partner prioritizing sector development:  Communication between ISC and implementing partners was weak and largely limited to bilateral meetings with ISC, which inhibited the development of any true partnerships between implementing partners.  JIs may have been an improvement resulting from a common programmatic goal of the CSO coalition, but ISC apex coordination limited the amount of collaboration among coalition members.  Special Initiatives grants had a separate status and thus dynamic within CSAI. 29 | Page  There is no evidence that training capacity in the sector has improved over the seven years of CSAI. Rather, pre-existing capacity was partially dismantled, and the subsequent Advocacy Resource Persons (ARPers) were not used systematically and remain atomized.22 The Team found that the three Belgrade-based implementing partners, BCIF, CI, and SK, did not regularly meet as a group with ISC. Rather, they would periodically meet on a bilateral basis with key ISC personnel. While some implementing partners did interact with each other, e.g., SK and BCIF, and there was no reason that implementing partners could not pursue deeper relations with other implementing partners, the absence of a project context conducive to partnering is striking for a project ostensibly based on partnership. The Team found that the JIs functioned similarly. As explained by leading members of each of the three JIs whom we interviewed, each JI member organization had their tasks and roles clearly delineated, and any interaction between organizations was based on past relationships. Communication went through ISC, the Team found, and then down to each of the JI organizations. A related finding was that ISC’s construction of the JIs was based largely on the set of past grantees, not on existing relationships between grantees, such that all of the Environment JI organizations reported to the Team that they lost 7-10 months simply working out tasks, coordination, and goals. The JI effort by CSAI, though, had the positive effect of overcoming collective action problems by the JI members stemming from potential internal governance problems and the need to pursue short-term funding. The Crno na Belo coalition struggled with implementation of public events that were jointly funded from coalition members’ discrete budgets but agreed unilaterally with ISC. Horizontal management slowed down the decision-making process, prevented development of quality assurance procedures and, possibly, decreased impact. The Special Initiatives (SI) grants operated within CSAI according to a different dynamic than other ISC grant processes, not surprising given the nature and genesis of those grants. Grantees that never had a SI grant all described interaction with CSAI as including: coaching or mentoring process before submission of a proposal, a large set of training topics on offer, and regular interaction during implementation of the project. Grantees that only had SI grants (for a variety of activities) all described their primary relationship as being with the US Embassy or USAID. Training was limited to financial management and reporting for those that were less established, and interaction only in the form of periodic reporting and ISC attendance at public events. All interviewed SI grantees reported that the projects would likely have been conducted with funding from another donor. Grantees that had both SI and non-SI grants reported these same differences. The end result is a set of organizations that did not have some of the same beneficial experiences with CSAI that other organizations had. The team found from interviews with an implementing partner, trainers, and all of the larger grantee organizations outside Belgrade that the training capacity in the sector changed in significant ways during the period of CSAI. At the start of CSAI, the main training resource was ‘Tim TRI,’ a Civic Initiatives activity that offered training on a variety of NGO-relevant topics. ISC initiated an effort to create a corps of ‘Advocacy Resource Persons’ (ARPers) that would be intensively trained in advocacy practices, as a means to provide training and support to ISC and BCIF grantees. A portion of the ARPers came from the ranks of Tim TRI, however, which had a negative effect on that CI activity. The Team found that ISC itself conducted much of the advocacy training and did not use the ARPers in any kind of regular or systematic way. One ARPer remarked that, ‘It felt like ISC just reached into the supply closet to grab one of us 22 The ARPers are merely individuals, and there’s no collective or association or network – hence they are ‘atomized’ or disconnected. 30 | Page whenever they couldn’t do something themselves.’ No ARPer we interviewed said that there was regular communication between them and ISC or any formal or informal grouping of ARPers, a context that reduces ARPers to an individual rather than a collective resource. All ARPers the Team interviewed valued the training they received on training for advocacy, and used those skills whenever they are asked to train people either by other donor-funded projects or pro bono for their favorite Serbian CSOs. The Team concluded that changes to the grantmaking scheme by both ISC and BCIF were beneficial to CSAI, but could have been more extensive:  CSAI experienced changes in the granting approach and themes that were an improvement on the past.  There appears to be no meaningful distinction between Advocacy in Practice, Innovation Fund, and Success Grants.  BCIF developed a progression of grant levels to encourage well-performing organizations, especially those doing more constituency building, but there appears to be no meaningful distinction between those levels other than funding size. The suite of CSAI grants and themes changed almost annually, according to project reports and the grants database. After the second year of the program, BCIF stopped offering Grassroots grants, which every grantee the Team interviewed valued because they were available to organizations outside Belgrade, including informal organizations. This change came about because ISC wanted CSAI to focus on advocacy and policy change projects, while BCIF’s Grassroots grants were intended to spur citizen action and community mobilization; note that BCIF continued these types of grants in its grantmaking outside the CSAI framework. BCIF created the ACAP grants aimed at slightly larger, formal, and more established CSOs outside Belgrade, which clearly complemented ISC grants and more closely adhered to the approach of CSAI. AP grants were completed and mostly accomplished their results. However, sustainable changes could not really occur over 12 months and topics like environmental protection were being addressed by multiple organizations. These projects achieved results, but the effect on citizens was localized and idiosyncratic. JIs were a different approach begun in the fifth year of CSAI. Based on ISC matchmaking, that were intended to overcome what ISC viewed as limited advocacy for policy change from scattered projects addressing related by different issues. The Team found that interviewed JI members generally valued the new approach because it focused their efforts on an identifiable goal. The Advocacy in Practice JI focused on open government, particularly the successful Open Parliament effort. The Socio-Economic Rights JI addressed policy issues that had largely been ignored by the GoS and the public, and focused on workplace rights for workers, particularly problems with contract labor. The Environmental JI picked up a prior focus on waste management and recycling. The New World and Innovation Fund (IF) grants, begun in 2010, are an example of a clear niche effort by CSAI. The intention was to ensure strategic utilization of new media and technologies for civic activism through training and support to CSOs in use of social media, and introduction to organizations that were strong in doing that. The team found evidence that these resulted in direct increase in adoption of new technologies by CSOs. The Hakaton and the Share Conferences, for example, represent new ground for CSOs. Success grants, given out only in 2010, were intended to give high-performing grantees resources to publicize the organizations and their activities. There is no evidence that this had any effect on public awareness. There appears to be little distinction between Advocacy in Practice, Innovation Fund, and Success grants, other than the label. The Team found the Advocacy in Practice label to be clearly associated with CSAI by all types of grantees, but it was discontinued in 2010. 31 | Page Themes also came and went, depending on high-profile political events like elections or European integration. These projects may have achieved their results, but were largely small one-off efforts. BCIF also experienced changes in its suite of grant opportunities. When ISC and BCIF agreed after the first project year that the Grassroots Grants should be discontinued in favor of slightly larger grants for more capable organizations, BCIF created the ACAP grants. It is important to note that BCIF kept small grants for the Green List environmental action projects by small organizations that were collaborating with Environmental JI members funded by ISC. The new ACAP grants had success, such as the Resource Center Majdanpek’s successful effort to make Line 481 funding from the municipality a transparent and fair competition, and the Roma ‘Osvit’ organization conducting public events in conjunction with the Roma Decade. ACAP grantees reported to the Team that the additional training on advocacy, fundraising, and project management had been used in follow-on projects by their organizations. BCIF created the PCAP grants in the fifth year of CSAI as an additional, higher level for well-performing organizations. A select group of seven past BCIF grantees received intensive mentoring and training on capacity-building. PIRGOS subsequently drew on municipal Line 481 funds, which in a previous project had been made more transparent and fair, to develop a rural development strategy. The Academy for Women’s Entrepreneurship used the advanced skills in a follow on advocacy effort targeting local authorities in Kovacica, with the goal of allocating a portion of the sports budget to women’s sports, instead of funding football and hunting exclusively. Finally, BCIF created the ‘Successful Fundraising’ grants, in which selected organizations were given training in fundraising, and BCIF matched the money they raised with a grant. Seven organizations fundraised over $36,000 from local sources. As with ISC, there appears to be little distinction between ACAP and PCAP grants, other than selectivity and grant size. The shift from Grassroots Grants marked an identifiable change, as did the Successful Fundraising Grants. The Team concluded that ISC exhibited both positive and negative program implementation practices:  Both ISC and BCIF conducted transparent, fair grant competitions.  Implementing partners’ work plans improved because of ISC collaborative engagement.  ISC’s coaching approach before, during, and after grant awards was a management aspect that almost all grantees said made their projects better.  ISC adopted innovative approaches such as training vouchers and the ‘Mobile Advocacy School’ that recipients valued.  CSAI M&E practices were strong and collected useful output and outcome data.  Frequent turnover of ISC leadership limited the development of relationships between ISC and implementing partners, and stunted the connection to leading CSOs in Belgrade. Every ISC and BCIF grantee reported hearing about grant competition from multiple sources such as websites, listservs, and emails. The calls for proposals were also viewed as clear and the competitions as fair. The implementing partners in Belgrade reported to the Team that the annual work plan process with ISC was collaborative and resulted in better projects. The Team also heard from SK and BCIF that the combination of access to funding over a long period of time, and this collaborative process, gave their 32 | Page organizations the room and impetus to plan strategically for more than one year. This indeed may be one of the main legacies of CSAI. One key finding is that ISC was deeply involved in the grantmaking process from concept to implementation, which was true for all grant types other than Special Initiatives. ISC worked closely with grantees to develop a solid concept paper, a realistic work plan, and then a budget grounded in the proposed activities. The Team found that the training vouchers concept and the Mobile Advocacy Schools were new to Serbia. As noted elsewhere in this report, beneficiaries valued both innovations. CSAI maintained an extensive and sophisticated monitoring and evaluation system, the Team found. Project staff were able to quickly pull up spreadsheets that had all indicators, annual targets and achievements, and indicator definitions. One negative aspect of this, however, is that the CSAI suite of grants changed nearly every year, so iterations of indicators related to grantmaking activity lost a portion of their utility. The Team found that the frequent turnover of ISC leadership was a problem for developing an ongoing working relationship between ISC and the implementing partners. The longer term strategic visions of the implementing partners were subject to change based upon the different priorities and experiences of each Chief of Party. This turnover was less of a problem for non-SI grantees, because their main contacts were the individual grant officers who remained more or less constant over time. SI grantees, on the other hand, reported that their lack of regular interaction with CSAI meant that a new Chief of Party had to be educated about their grant-funded efforts, which hindered the development of long-term strategic visions and the development of strong relationships with implementing partners. While an obvious implication for CSAF is to maintain continuity in the Chief of Party position to the greatest extent possible, another conclusion is that ISC can lessen the burden of an unavoidable transition by making greater efforts to educate new leadership internally through a more robust hand-off. Evaluation Question 4: What were the obstacles or difficulties ISC encountered during implementation of the activity? Based on the in-brief, the Team understood this to mean ‘ambient or exogenous factors that affected the program.’ The Evaluation Team concluded that several ambient or exogenous factors had a negative effect on CSAI’s implementation:  Political volatility constrained the ability of CSOs to advocate and mobilize citizen engagement.  The economic downturn is problematic for individual and corporate philanthropy. Serbia has experienced significant political volatility over the last seven years, which has had an effect on civil society and CSAI’s program. First, multiple elections tend to close policy windows either through focused attention on campaigns or turnover in elected officials. This is particularly problematic for organizations and policy issues that depend on personal connections. The Environmental JI is perhaps the best example, as its advocacy for new laws on waste management and recycling fell victim to the 2012 change of government. Additionally, events in Kosovo prevented the Humanitarian Law Center from effectively advocating for war crimes tribunal issues. The economic collapse of Greece and the ongoing recession in Europe have had a negative effect on Serbia’s economic development. The economy had been stagnant and weak for many years prior, but the recession may have served to dampen any enthusiasm that may have still existed in the post-Milosevic era. Low personal incomes reduce the potential for individual philanthropy, and uncertainty may inhibit corporations from acting strategically in philanthropy. 33 | Page Overall Conclusions The large set of individual and aggregate conclusions above can be synthesized to a small set of overall conclusions that have implications for future civil society programming. First, ISC’s implementation of CSAI has been highly professional, from grantmaking processes to innovative training, to mentoring of organizations, to general organization management. The monitoring and evaluation efforts and applied research are of high quality, and should be emulated by other civil society program implementers. Second, CSAI increased the advocacy capacity of Serbian CSOs. With particular success in some of the emerging CSOs outside Belgrade, many CSOs are now capable of playing a broader role within Serbian regions, including Nacionalna Koalicija za Decentralizaciju and Zajecarska Inicijativa. CSAI implementing partners like BCIF and SK acknowledge the various ways in which their organizations were challenged to become better and more effective. This training imparted an understanding of a more policy-focused and holistic concept of advocacy and community outreach. The capacity that organizations have developed through training has often been carried forward to subsequent projects, regardless of funding source. Capacity is vulnerable to factors beyond CSAI’s control, however, which increases the importance of sector training capacity and sector leadership. Policy advocacy, it should be noted, is always and everywhere at risk of changing political climates. Third, CSAI has been implemented by ISC as a plan to be executed, rather than as an effort to develop the sector more generally. USAID-funded programs are intended to have higher-level impact beyond the outcomes expected from clusters of activities. Project designs should reflect strategic consideration of how the set of activities contributes to a broader development goal. The issue can be seen in several ways. The development of sector leaders was hindered because ISC took a directive approach to interactions with implementing partners instead of facilitating partnerships among the organizations. More telling is the lack of any strategy by ISC to reduce its leading role as implementing partners became stronger and more autonomous. Similarly, the lack of a broader vision for the sector’s training capacity will be a significant problem in Serbia going forward, as will be the lack of capacity for grappling with nonprofit law issues. Finally, the lack of a sector leader, whether as an informal coalition or a formal organization, is highly problematic given the withdrawal of donors and EU accession policy windows of opportunity. Fourth, improvement in public perception and diversification of funding sources are long-term processes. Serbia has made notable gains from a very low level of individual philanthropy, corporate engagement, and state acceptance, but all three aspects are still weak. Overall Recommendations The Team’s ability to make recommendations applicable for the two-year CSF is constrained by the fact that the project is already underway and the implementing partners were selected ahead of time. The new CSF project already includes a more robust version of training vouchers, which would have been a Team recommendation. However, based on the overall and the earlier detailed conclusion, the Team would like to present recommendations for consideration by USAID for the CSF project:  The structure of CSF is perhaps analogous to that of the JIs, which – along with past implementing partner concerns – raises questions about whether ISC will again act as an apex- 34 | Page level coordinator that inhibits relationships across implementing partners. ISC needs to allow more room for the implementing partners to work autonomously and to set an agenda themselves. ISC’s approach of coaching grantees from concept paper to the ultimate proposal submission should remain a part of its modus operandi, but in a more advisory than directive role.  Attention must be paid to encouraging a sector lead resource organization or network. As noted, the sector faces a daunting set of problems common to most, if not all, organizations – the lack of a coherent sector-wide training capacity, the need to grapple with continual enabling environment issues, and the critical importance of engaging the GoS during the EU accession process. The OCCS is not and should not be the aggregate-level representative for the sector.  Develop a strategy for how the sector can deal with long-term problems of philanthropy and public perceptions. The development of a ‘culture of philanthropy’ is a long-term process, and the following lessons of other post-communist states should be heeded: Focus public attention on how CSOs help individuals and communities, move beyond traditional concepts of deductible donations, and political and financial support for the sector will follow. The experience of grassroots grantees and ISC grantees outside Belgrade demonstrate that it is possible to tap into individual resources, usually in the form of volunteers, as well as corporate in-kind donations and state official collaboration. USAID should conduct an assessment of best practices from throughout the post-communist states. 35 | Page ANNEXES Annex I: Evaluation Statement of Work Background: The Civil Society Advocacy Initiative (CSAI) was originally designed as a $12 million, five-year program to support Serbian civil society influence public policy, serve as a government watchdog, and conduct sustained advocacy campaigns on a wide variety of reform issues. In 2008, the award's ceiling was increased by an additional $2.2 million for Special Initiatives funding and in 2009 the ceiling of the core budget was increased by an additional $4.3 million, which reflected a need for additional resources in the sector. In September 2010, the ceiling was increased again – by $9 million – and the period of performance was extended for two additional years until May 31, 20I3. As a result, CSAI is now a seven￾year program with a total ceiling of $27.5 million. Overall, the program was designed to increase the quality, quantity, and strategic focus of advocacy initiatives; the range of skills, strategies, and tactics used in advocacy efforts, including educating the public and monitoring implementation of laws or policies; the quality and quantity of public discussions about the need for republic-level constitutional reform, and the implications of proposed constitutional provisions; the number of organizations and coalitions conducting sustained campaigns; and the ability of citizens and the Parliament to oversee the implementation of enacted legislation and appreciate the need for new or amended legislation, and CSO and citizen input into legislation, by-laws and government policy, particularly those related to Euro-Atlantic integration. CSAI provides technical assistance, training, study tours, internships, cross-border contacts, partnerships, and mentoring to:  Strengthen the advocacy capacity of civil society organizations;  Support Serbian NGO advocacy on legislation, policy, and implementation of reforms leading to Euro￾Atlantic integration;  Encourage and equip government, especially Parliament, to work with civil society and perform legislative oversight;  Improve the legal and regulatory framework for NGOs; and  Enhance NGO management capacity and financial sustainability. Activities included:  Awarding Advocacy in Practice (AP) grants and in the final two years of CSAI implementation complex joint advocacy initiatives on issues such as promoting government accountability, increasing environmental activism and improving socio-economic rights;  Supporting the Advanced Community Advocacy grant program implemented by the Balkans Community Initiatives Fund;  Providing capacity building for advocacy in practice and joint initiatives' grantees, including training in networking/coalition building , project cycle management and media relations to complement the current trainings on advocacy and consensus building;  Implementing a Mobile Advocacy School Initiative to provide training around the country for grassroots advocates; 36 | Page  Encouraging CSR – especially employee volunteering – and additional recognition of leaders in corporate philanthropy;  Promoting individual donations and philanthropy;  Increasing the visibility of CSO activities and improving the overall image of the sector through a multi-pronged approach that includes training CSOs in outreach and media relations, technical assistance for individual CSOs' communication efforts, and launching an initiative to introduce public service announcements and social campaigns; and  Supporting a strengthened legislative environment for civil society advocacy. CSAI was USAID/Serbia and Montenegro's primary mechanism for awarding grants to local organizations working on legislative oversight, constitutional reform, and other key areas of democratic and economic reform, particularly reforms necessary for accession to the European Union. The program includes a significant grant component that primarily focuses on one- to two-year, competitively-awarded grants to advocacy -oriented and watchdog CSOs, think tanks, independent analytical centers, citizen­based organizations, and membership-based associations. In line with the overall program, grants support civil society projects that address issues such as government accountability, citizen advocacy, the environment, youth, human rights, access to public information, and cross-border cooperation. The program pays special attention to supporting and developing organizations outside the capital city of Belgrade, particularly those working on cross-cutting issues of interest to USAID, such as gender, youth, cross-border contacts, or transparency in government operations. As CSAI winds down, USAID/Serbia and Montenegro has recently awarded a new civil society assistance program - Civil Society Forward, launched in FY 2013. This two-year program serves as a bridge, providing technical assistance and grants to strengthen Civil Society Organizations' advocacy and build capacity of key local partners to receive direct USAID assistance once the program ends. Findings and recommendations from this evaluation will inform implementation of this new program. SI Responsibilities and Projects: The objective of this performance evaluation is to conduct a full, evidence-based, and independent review of USAID/Serbia and Montenegro's CSAI Program, implemented by ISC, and to produce a report that provides a qualitative as well as quantitative evaluation of project activities. The audience for this evaluation will be USAID/Serbia and Montenegro's Democracy and Governance Office as well as ISC. The findings and recommendations will inform implementation of the newly awarded Civil Society Forward program, which is also implemented by ISC. Evaluation Questions The evaluation will address, at a minimum, the following questions: a) What are the core achievements and legacy of CSAI? This can include achievements in comparison to the work plan as well as unintended results. b) What lessons can ISC learn from the management and implementation of CSA!? c) How effective was CSAI in strengthening civil society's capacity for and engagement in advocacy? d) What were the obstacles or difficulties ISC encountered during implementation of the activity? The report will summarize the findings from field visits, discussions, and meetings with respective reports. Pitfalls and gaps, if any, must be justified and supported by recommendations. The evaluators are encouraged to be as specific as possible in their recommendation s, so as to best inform our Civil Society 37 | Page Forward program. Recommendations must be action-oriented, practical, and specific, with defined responsibility for the action. Methodology The Evaluation Team will first complete a desk study to establish an understanding of CSAI's activities and environment before arrival in Serbia. The desk study, which will help the Evaluation Team become familiar with ISC's work, includes reviewing: • The Cooperative Agreement and all Modifications, with particular attention to Modifications 10 and 12 and the amended/intensified scope of work and technical proposal for two-year cost extension of CSAI; • CSAl 's Mid-term evaluation; • ISC quarterly reports; • ISC's Performance Monitoring Plan; • Surveys of the Civil Society sector in 2005 and 2009; • A survey on the perception of CSOs commissioned by USAID/ISC; • Baseline survey of Serbia civil society following the re-registration process per the Law on Associations conducted by IPSOS Strategic Marketing and Civic Initiatives; • The European Commission Report on Serbia (Serbia Progress Report 2012); and • The NGO Sustainability Index Report for Serbia. Key informant interviews will be the primary data source for this evaluation. The Team must support all findings and conclusions/qualitative analysis with quantitative data, when available, and/or representative examples. The Evaluation Team Leader will provide the interview protocol s to USAID prior to starting interviews. USAID reserves the right to approve these protocols and to add questions as necessary. The Evaluation Team will spend approximately seven days on field visits to CSAI grantees in Serbia. The remaining time will be scheduled as to allow for the team members to meet with key civil society interlocutors and partners in Belgrade, including USAID, the Institute for Sustainable Communities, ISC, implementing partners and grantees. The Team will also meet with key government partners including the Office for the Cooperation with Civil Society, EU Integration Office, and Ministry of Youth and Sports. The LOE allows time in Belgrade to write a Draft Report. The Team will provide an Out-brief to the Mission on key findings. The evaluation must be substantiated by relevant data and information gathered from meetings with program partners and all the relevant stakeholders. The Team must clearly demonstrate links between the recommendations and/or conclusions and data available. Tasks The Evaluation Team will undertake these specific tasks to prepare the deliverables: • Review of all documents listed in the ‘Methodology’ section above; • Meetings with: o Institute for Sustainable Communities' headquarters in Vermont, U.S. (by phone); o USAlD/Serbia and Montenegro's Mission Director , Democracy and Governance Office Director, Program Officer, Project AOR, and field office managers; o Institute for Sustainable Communities' staff in Belgrade; 38 | Page o ISC implementing partners in Belgrade (Civic Initiatives, Smart Kolektiv, Balkan Community Initiatives Fund (BCIF), and European Center for Non-Profit Law); o Project beneficiaries (the list of beneficiaries will be provided along with other documents); o Appropriate Government of Serbia departments; o Other donors, including the EU and the UN. To support the team's initial literature review, USA ID/Serbia and Montenegro will share via Google Docs application, the electronic copies with the contractor of all documents to be reviewed (listed above) one week prior to departure. Special attention during the review must be paid to how well the project contributes to achievement of the Mission's strategic objective. In addition to these approaches, the Mission is looking for new, creative suggestions regarding this evaluation, and it is anticipated that the implementer will provide a more detailed explanation of the proposed methodology for carrying out the work. The Team is expected to provide its insight into the weakness or the strengths of the methodology proposed. Level of Effort LOE Distribution Action/Deliverable Approx. 10.5 days Pre-Departure Planning and Desk Review including Desk Study, draft plan, and trip preparation Approx. 21 days Fieldwork and Data Collection including In-brief to Mission, In￾country meetings and research, and Travel from Serbia Approx. 11.5 days Data Analysis, Reporting, and Dissemination including Finalizing Draft Report, Submitting report, and Revising based on Comments from USAID/Serbia and Montenegro. Timeline Task/ Deliverable Period of Performance Review background documents; preparation work (offshore) 13-17 May Team Planning Meeting 14 May Andrew Green travels to Serbia 18-19 May In-brief with USAID/Serbia & Montenegro; submit draft work plan 20 May Data collection, analysis, annotated report outline, draft presentation 21-25 May, 27 May – 1 June, 3-6 June Evaluation Team submits annotated report outline and draft presentation for review by USAID/Serbia & Montenegro 6 June Presentation and debriefing with USAID/Serbia & Montenegro 7 June Debriefing with key stakeholders TBD Andrew Green departs Serbia 8 June SI submits draft report to USAID/Serbia & Montenegro 21 June USAID/Serbia & Montenegro comments on draft report 12 July SI delivers final report 2 August 39 | Page Annex II: Evaluation Methods and Limitations This final evaluation examines a mixture of performance and process, which is reflected in the evaluation questions. The evaluation questions on (a) core achievements and legacies, and (c) capacity strengthening effectiveness address the performance aspect, while questions on (b) management and implementation lessons and (d) obstacles to implementation address the process or design aspect. The SI Evaluation Team is composed of Dr. Andrew Green, Senior Evaluation Specialist and Team Leader; Sanja Nikolin, Program Development Specialist and Serbian civil society expert; and Divna Pekic, logistician and interpreter. Work Plan The evaluation of CSAI started with a Team Planning Meeting on 14 May, followed by the review of important documents and drafting of a desk study prior to beginning fieldwork. The documents reviewed included the technical proposal, quarterly reports, work plans, M&E plan, a mid-term evaluation, assessments, applied research materials, participant lists, training agendas, training evaluations, toolkits, guides, manuals, website traffic statistics, strategies, action plans, policy briefs, committee briefing books, and policy research reports. Expatriate team member Andrew Green arrived in Belgrade on 19 May and participated in an in-brief with USAID/Serbia & Montenegro staff. Following these meetings, the Evaluation Team began data collection, which included the use of semi-structured interviews (SSIs) with key informants, group interviews, materials review, and mini-surveys. The relevant protocols and planned mix of evidence sources are detailed Annex II. The Team collected data for approximately 14 business days, and then analyzed the data, prepared a report outline, and produced a draft presentation for USAID/Serbia & Montenegro. Site visits were conducted in Novi Sad, Zrenjanin, Kikinda, Leskovac, Niš, Pirot, Zaječar, and Majdanpek in addition to Belgrade. The sites were selected non-randomly due to USAID guidance on the first evaluation question. The Evaluation Team presented its findings to USAID in an out brief meeting on 7 June, before Andrew Green departed Belgrade on 8 June. Evaluation Design The Evaluation Team conducted approximately 90 SSIs of key informants with the implementing organizations, direct participants, indirect beneficiaries, and external experts. The Team conducted interviews with personnel representing:  65 of the total 237 unique grantee organizations (27%),  147 of the 396 unique grants (37%), and  Over $9 million of the total $15 million in grants (62%). The interview set was selected non-randomly, and the Team endeavored to develop and conduct a set of interviews that was representative of the total grant amount, number of grants, and annual grant amounts (see Tables 1-3 in Annex VI). In addition, the Team sent mini-surveys by email to non-interviewed BCIF grantees and Mobile Advocacy School participants. Limitations and Constraints The essence of evaluation is comparison—typically across time or geography, better still across both; an impact evaluation further attempts to compare across units that have or have not received some sort of programming treatment. While CSAI has collected extensive monitoring and evaluation data, including 40 | Page applications of scorecards at multiple points in time, the extensive coverage, long timeframe, and other activities of partners of the program excluded the possibility of even ex post facto quasi-experimental approaches. As with any evaluation based predominantly on qualitative data, there are some noteworthy limitations on comparison, and hence attribution, that should be anticipated. First, recall bias may be present, for example as grantee CSO staff respond to Evaluation Team questions with answers related to grants from another donor. A similar problem is that participants in multiple training activities may be blending their experiences into a composite memory or response, e.g., CSO staff may have received training on advocacy and on fundraising, and subsequently do not distinguish between them as separate activities. Second, response bias is a common problem for program evaluations. For example, CSO grantees may give the interviewer positive remarks about the project because s/he would like to receive another grant in the future. The Evaluation Team fully expects that CSO grantees, trainees, and key implementing partners may understand that a negative evaluation could mean the end of project opportunities or may fear that their comments may be identifiable and thus harm ongoing relationships. Third, selection bias in the form of contacts provided by the implementers can mean that the Evaluation Team only hears from people with positive experiences. The most effective approach to combating these biases is to use multiple sources of data to triangulate on an evaluation issue, as is often accomplished through qualitative reliability matrices. By combining information found in documents or interviews from multiple sources, any one piece of biased data would not skew the analysis. Another approach that pertains specifically to interviews is the inclusion of key informants from organizations that do not directly benefit from the evaluated program and the use of questions about specific examples of knowledge use. Attribution is difficult to make outside of experimental or quasi-experimental impact evaluations, because it is too difficult to isolate interventions from similar interventions or exogenous factors. For this evaluation of CSAI, the Evaluation Team notes that during the seven years of the program, Serbia experienced civil society programs by other donors, significant political volatility, economic recession, and re-shaping of the state borders. For a program designed to address issues of advocacy for policy change by civil society and citizens, these are variables likely to affect the performance of the program. The Evaluation Team’s biggest concern was about the availability of contacts at BCIF grantees, as well as ISC grantees in the first years of CSAI. The Evaluation Team anticipated that it would be difficult to schedule time with the staff from smaller, volunteer-dominant CSOs, and so the amount of evidence that could be collected from any one key informant would be less than desirable; similarly, we anticipated that response rates to a mini-survey request might be low. While BCIF grantees and early ISC grantees were more available than we had anticipated, the response rates for the mini-surveys of BCIF grantees and Mobile Advocacy Schools was in fact low at 21% (15/71) and 27% (51/193). Unlike the standard evaluation context, the Evaluation Team learned during the initial meeting with CSAI leadership that ISC had planned two internal evaluations during the time period in which the USAID evaluation could take place, in full knowledge of the timing and high priority of the USAID evaluation. Both internal evaluation topics were covered also by the USAID evaluation: one evaluation focused on ISC’s Mobile Advocacy Schools, and was conducted before the Evaluation Team began field work; the other focused on ISC’s advocacy approach, and was conducted wholly during the Evaluation Team’s period of 41 | Page field work. Indeed, for the latter, the internal team’s desired interviewee list overlapped to a significant extent with the Evaluation Team’s, which produced confusion and annoyance among some of the desired interviewees the Team contacted, as well as a low level of competition for availability and resulting effect on Evaluation Team logistics. The main methodological concern, of course, is that the Team does not (and cannot) know how this affected responses by mutual interviewees:  Recall bias – A prior interview with the internal team, or even the scheduling of such an interview prior to an interview with the USAID Evaluation Team, may have improved recall, may have focused interviewee recall on a more limited set of aspects than the Team desired, or may have induced interviewee fatigue on the topics generally.  Response bias – It is possible that a prior interview by the internal team provided an opportunity for interviewees to rehearse responses to certain topics, or that the USAID Evaluation Team’s interview provided that opportunity for interviewees interested in maintaining good relations for the follow-on CSF program. It is not possible to know precisely what effect the presence of a separate evaluation effort had on the information gathered by the Evaluation Team, but it is certain that an element of ‘noise’ was introduced unnecessarily.23 23 ‘Noise’ is a term commonly used in data contexts. It refers to unknown errors or variation in measurement, e.g. Nate Silver’s “The Signal and the Noise.” Measurement was likely affected in unknown ways by ISC’s internal evaluations. 42 | Page Annex III: Data Collection Instruments A. Evidence Matrix Documents/Research Semi-Structured Interviews Mini-Survey Activities & data sources are illustrative, not comprehensive Donor Implementer Secondary USG Personnel Implementer Staff Direct Participants or Beneficiaries Indirect Beneficiaries External RESULT 1: CSAI grantees advocate effectively about political, economic, and social issues that consolidate Serbia’s democratic transition Implementer Grants  ISC, BCIF, CI, SK, ECNL   BCIF, CI, SK, ECNL Advocacy in Practice Grants  ISC, BCIF Media, GoS   Grantees, Fellows BTD, FOS Grassroots Advocacy Grants, Advanced Community Action Program, Proficient Advocacy Community Grants  ISC, BCIF Media, local govt   Grantees, Trainees Local govt GoS Leaders, trainees at CSOs not interviewed Strategic capacity building; Advocacy Resource Persons; Mobile Advocacy Schools  ISC, BCIF Grantee materials   Trainees, ARPs Local govt BTD, FOS Leaders, trainees at CSOs not interviewed Innovation Fund, Success Grants  ISC Grantee materials   Grantees Special Initiatives  ISC Grantee materials   Grantees Local govt Youth Program  CI GoS   Min Youth RESULT 2: Citizens, businesses, and government view CSOs as reliable partners in building a more democratic and prosperous Serbia CSO Sector Visibility (Vidljivo bolje)  ISC, SK, CI Media, polls   CSOs BTD, FOS Leaders at CSOs not interviewed Promoting Philanthropy (Mala pomoć za puno srce)  ISC, BCIF Media, stats   Relevant CSOs CSOs Leaders at CSOs not interviewed CSO-Private Sector Partnerships (VIRTUS, BLF)  ISC, SK, BCIF Media, stats   CSOs, companies CSOs BTD Leaders at CSOs not interviewed Enhancing State-Civil Society Cooperation  ISC, CI, EU   GoS Leading BTD, Leaders at CSOs not 43 | Page ECNL CSOs FOS interviewed RESULT 3: An enabling environment exists for CSOs to advocate effectively for reform issues Law on Associations, other legislative efforts  ISC, ECNL European Foundation Centre   Leading CSOs GoS BTD, FOS 44 | Page B. Semi-Structured Interview Questions Implementer Direct Participants Indirect Beneficiaries Externals RESULT 1: CSAI grantees advocate effectively about political, economic, and social issues that consolidate Serbia’s democratic transition Grants To what extent have appropriate organizations been targeted by [grant]? Specific examples of successful grant-funded advocacy projects? Unsuccessful? Why? How should grant-making be done differently? How were the dynamics of national and local different? Regional disparities and rural/urban spread? Gender equality? Coalition building? Examples of successful coalitions? Specific examples of built capacity for advocacy? Examples of improved prominence of local actors? Is CSAI still needed? If so, has the need changed and how (address demand and supply side)? In what ways has it affected citizens’ engagement? Environmental Joint initiatives and visibility? Government Accountability and Transparency? Soc-Econ Rights? Coalitions and coordinating partners with different visions, goals? The proposal requirements? The review process? The reporting requirements? Org needs assessment? Problems with progression? Specific example of how your capacity improved with CSAI support/training? Had your org done anything like this before? How did your org decide on advocacy issue? What effect did the advocacy campaign have? In what ways has it affected citizens’ engagement? How could this be done in the future without CSAI? How did you find out about the grant competition? Coalition building? Examples of successful coalitions? Could it have been done better/differently? Is CSAI still needed? Were the proposal requirements clear, vague, simple, too complex? Has your org written proposals before? Reported progress? What needs did org identify before training offered? Specific example of advocacy project? In what ways has public opinion improved? In what ways has it affected citizens’ engagement? Specific example of advocacy project? In what ways has public opinion improved? Specific example of improvement from legislation? Specific example of improved capacity/engagement for advocacy? Specific example of effective advocacy campaign? 45 | Page Training Was it a balanced progress on advocacy skills development and organizational development? Advanced advocacy training -partnership development, media promotion, community mobilizing? Technical assistance and support lessons learned? Peer learning and exchange? Core curriculum and its future? Has it integrated gender considerations? Tradeoffs between repeating training program cycles to expand reach and offering new programs for more in depth knowledge? Specific examples of low capacity that improved? Capacity high enough to do without CSAI? Specific example of effective advocacy campaign? Specific examples of orgs capable of advocating without CSAI technical assistance? Has the concept worked? Is there still a need for ARPs? What would you do differently with regard to their role/training/approach? Coordinating partners with different visions, goals? Impact of use of new media in fundraising? Examples of increased regional cooperation? Increased CSO capacity at grassroots levels? Improvement of the reputation and image of CSOs? How difficult was it scheduling MAS? Capacity high enough to do without CSAI? Examples and impact individualized training/consulting? Impact of use of new media in fundraising? Examples of increased regional cooperation? Increased CSO capacity at grassroots levels? Improvement of the reputation and image of CSOs? Specific example of how MAS skills helped org? Examples of increased regional cooperation? Increased CSO capacity at grassroots levels? Improvement of the reputation and image of CSOs? Examples of increased regional cooperation? Increased CSO capacity at grassroots levels? Improvement of the reputation and image of CSOs? 46 | Page RESULT 2: Citizens, businesses, and government view CSOs as reliable partners in building a more democratic and prosperous Serbia CSO Sector Visibility How successful was the project in improving public perception of CSOs (mention NDI/CRTA survey)? Impact of grantee capacity building for effective media outreach? Law on associations/volunteers? Specific example of improvement from legislation? How was this coordinated with ISC and other partners? Were there any specific fact ors that may have limited the effect of an awareness campaign? Specific example of improved capacity/engagement for advocacy? Specific example of effective advocacy campaign? Were there any specific factors that may have limited the effect of an awareness campaign? How to coordinate CSOs to work together well? Other ways to raise visibility? Were there any specific factors that may have limited the effect of an awareness campaign? Promoting Individual Philanthropy History of philanthropy and research into philanthropy and web portal? Cultural issues with individual philanthropy? Value vs. time? Were there any specific factors that may have limited the effect of an awareness campaign? Why do individuals give value or time? Specific example of improved capacity/engagement for advocacy? Specific example of effective advocacy campaign? Were there any specific factors that may have limited the effect of an awareness campaign? CSO-Private Sector Partnerships What successes in engagement of private sector partners? Business Leaders Forum accomplishments? Integration of social, ethical, environmental aspects in operation of businesses? Multi-stakeholder forums? Database on CSR practice, cases? Cooperation with SIPRU and SIEPA? Awards? Why do corporations participate? Why do corporations give value? Why does your firm participate in partnerships or strategic giving? How has this changed the way your firm views and engages with civil society? How to coordinate CSOs to work together well? Other ways to raise visibility? Were there any specific factors that may have limited the increase in corporate engagement? Enhancing State-Civil Society Cooperation Office for Cooperation with Civil Society – what impact on CSO-state relations? Advocacy on financial stability of CSOs? Monitoring of line 481? Were there any specific factors that may have limited the development of this engagement? Office for Cooperation with Civil Society – what impact on CSO￾state relations? Advocacy on financial stability of CSOs? Were there any specific factors that may have limited the development of this engagement? Specific example of improved capacity/engagement for advocacy? Specific example of effective advocacy campaign? Were there any specific factors that may have limited this development, going forward? Office for Cooperation with Civil Society – what impact on CSO￾state relations? Advocacy on financial stability of CSOs? Were there any specific factors that may have limited the development of this engagement? 47 | Page RESULT 3: An enabling environment exists for CSOs to advocate effectively for reform issues Law on Associations, other legislative efforts Describe the NP legal environment in 2006? Describe the consultative and political processes leading up to the new 2009 LoA? To what extent did proposed drafts and the final legislation take into account ECNL’s expert recommendations? What has been the effect of the new LoA? Would the law have been revised without ECNL’s assistance? Specific examples of ECNL’s cooperation with other ISC & partner organizations, esp. awareness? Describe the consultative processes for LoV, LoEF, tax, etc.? Would these processes have taken place without ECNL? Could they now? Specific examples of how org participated in LoA process? Describe the political processes leading up to the new LoA? CSAIs role in encouraging cooperation? What has been the effect of the new LoA? Describe the re-registration process? Could leading CSOs have done this without ECNL? Other potential legislation? Describe the consultative processes for LoV, LoEF, tax, etc.? Would these processes have taken place without ECNL? Could they now? Specific examples of positive or negative role of CSOs/INGOs in consultative process for new LoA? What has been the effect of the new LoA? To what extent did information campaigns and materials help CSOs re-register? Describe the consultative processes for LoV, LoEF, tax, etc.? Describe the NP legal environment in 2006? Describe the consultative and political processes leading up to the new 2009 LoA? What has been the effect of the new LoA? Describe the re-registration process? Would the law have been revised without ECNL’s assistance? Describe the consultative processes for LoV, LoEF, tax, etc.? Would these processes have taken place without ECNL? Could they now? 48 | Page C. BCIF Grantee Mini-Survey Questions and Summary of Results, English Dear Sirs, Thank you for agreeing to participate in the evaluation of the Civil Society Advocacy Initiative project from 2006 to 2013, which was conducted by the Institute for Sustainable Communities (ISC) in cooperation with the Civic Initiatives, Smart Collective, BCIF and the European Centre for Non-Profit Law (ECNL). We’re contacting you because we found your organization to be a beneficiary of the grantee BCIF. Please take 10 minutes of time to answer these questions, which will help us in the evaluation. We assure you that your answers will be anonymous and will not be used individually but only as part of the aggregate data, and donors will not be able to connect your answers with your organization. Thank you for your heartfelt and prompt response. Please send the completed questionnaire as soon as possible to the address, evaluacijacsai@gmail.com. We value your opinion and thank you for your cooperation. Best regards from the Evaluation Team of Social Impact Number Question Results 1 Name of Organization Number of Respondents 16 (total) 2 How many grants did your organization receive from BCIF? Please, specify the number. On average 2.75 3 Did BCIF, during the interview and before signing the contract, recognize the needs of your organization for strengthening capacity? Number of Respondents % of Total Respondents 3a) Yes 14 87.5 % 3b) No 2 12.5 % 3c) Other, please, specify 0 0 % 4 If yes, in which areas did you receive the support (please, specify all answers that apply to you within BCIF project)? Number of Respondents % of Total Respondents 4a) Advocacy 12 75 % 4b) Finance management and reporting 6 37.5 % 4c) Communication and PR 6 37.5 % 4d) Project Management 6 37.5 % 4e) Other, please, specify 4 25 % 5 Please, specify a project, as an example (whichever donor financed it) in which you applied acquired knowledge or skills from BCIF training. Responses varied 6 Please, rate on a scale of 1-5 how easy/difficult reporting to BCIF was. Average Grade 4.38 7 Would you implement the project or projects supported by BCIF even if you did not have their support? Number of Respondents % of Total Respondents 7a) Yes 0 0 % 7b) No 14 87.5 % 7c) Other, please, specify 2 12.5 % 49 | Page D. Mini istraživanje dobitnika BCIF grantova i sažetak rezultata, srpski Poštovani, Hvala Vam što ste prihvatili da učestvujete u evaluaciji projekta Inicijativa civilnog društva za javno zastupanje koji je od 2006. do 2013. godine sprovodio Institut za održive zajednice (ISC) u saradnji sa Građanskim inicijativama, Smart kolektivom, BCIF-om i Evropskim centrom za neprofitno pravo (ECNL-om). Sa Vama smo stupili u kontakt zato što smo Vašu organizaciju našli u bazi dobitnika granta od BCIF-a. Molimo Vas da odvojite 10-ak minuta vremena i odgovorite na pitanja koja će nam pomoći u evaluaciji. Uveravamo Vas da će Vaši odgovori biti anonimni i da se neće koristiti pojedinačno već samo kao deo zbirnih podataka, bez mogućnosti da donator Vaš odgovor poveže sa Vašom organizacijom. Hvala Vam na iskrenom i blagovremenom odgovoru. Molimo Vas da popunjeni upitnik pošaljete što pre na adresu evaluacijacsai@gmail.com. Stalo nam je do Vašeg mišljenja i zahvaljujemo Vam se na saradnji. Srdačan pozdrav od Evaluacionog tima organizacije Social Impact Broj Pitanje Rezultati 1 Naziv organizacije Broj anketiranih 16 (ukupan) 2 Koliko je grantova Vaša organizacija dobila od BCIF-a? Molimo upišite broj. Prosečno 2.75 3 Da li je BCIF, tokom intervjua sa Vama, pre potpisivanja ugovora, prepoznao potrebe Vaše organizacije za jačanjem kapaciteta organizacije? Broj anketiranih % od ukupnog broja ispitanika 3a) Da 14 87.5 % 3b) Ne 2 12.5 % 3c) Drugo, molimo navedite 0 0 % 4 Ako da, u kojim oblastima ste dobili podršku (molimo navedite sve odgovore koji se odnose na Vas u okviru BCIF projekta)? Broj anketiranih % od ukupnog broja ispitanika 4a) Javno zastupanje 12 75 % 4b) Upravljanje finansijama i izveštavanje 6 37.5 % 4c) Komunikacija i PR 6 37.5 % 4d) Upravljanje projektom 6 37.5 % 4e) Drugo, molimo navedite 4 25 % 5 Molimo Vas da navedete neki primer drugog projekta (bez obzira na to koji je donator finansirao taj projekat) u kome ste primenili veštine i znanja stečena na BCIF treningu. Odgovori variraju 6 Molimo Vas da ocenite lakoću/težinu izveštavanja za BCIF ocenom od 1 do 5 kao u školi Prosečna ocena 4.38 7 Da li biste projekat ili projekte koji-e je BCIF podržao realizovali i bez njihove podrške? Broj anketiranih % od ukupnog broja ispitanika 7a) Da 0 0 % 7b) Ne 14 87.5 % 7c) Drugo, molimo navedite 2 12.5 % 50 | Page E. MAS Mini-Survey Questions and Summary of Results, English Dear Sirs, Thank you for agreeing to participate in the evaluation of the Civil Society Advocacy Initiative project from 2006 to 2013, which was conducted by the Institute for Sustainable Communities (ISC) in cooperation with the Civic Initiatives, Smart Collective, BCIF and the European Centre for Non-Profit Law (ECNL). We're with you in touch because we found your organization on the list of participants for the Mobile Advocacy School. Please take 10 minutes of time to answer these questions, which will help us in the evaluation. We assure you that your answers will be anonymous and will not be used individually but only as part of the aggregate data, and your answers will not be connected with your organization. Thank you for your heartfelt and prompt response. Please send the completed questionnaire as soon as possible to the address, evaluacijacsai@gmail.com. We value your opinion and thank you for your cooperation. Best regards from the evaluation team of Social Impact Number Question Results 1 Name of Organization Number of Respondents 53 (total) 2 How were you contacted for the Mobile Advocacy School (MAS)? Number of Respondents % of Total Respondents 2a) Directly by BCIF 12 22.6 % 2b) Mailing lists, notices, website, by other organizations 33 62.3 % 2c) Other, please, specify 8 15.1 % 3 Which topics have you dealt with at the training? Number of Respondents % of Total Respondents 3a) Advocacy 43 81.1 % 3b) Finance management and reporting 28 52.8 % 3c) Communication and work with media 31 58.5 % 3d) Project management 36 67.9 % 3e) Project writing, fundraising 42 79.3 % 3f) Other, please specify 9 17.0 % 4 Please, specify a project, as an example (whichever donor financed it) in which you applied acquired knowledge or skills from MAS training. Responses varied 5 Did your organization submit project proposal to BCIF? Number of Respondents % of Total Respondents 5a) Yes, it did and it was financed 17 32.1 % 5b) Yes, it did, but it was refused 9 17.0 % 5c) No, it did not 24 45.3 % 6 Would you implement the project or projects supported by BCIF even without their help? Number of Respondents % of Total Respondents 6a) Yes 19 35.9 % 6b) No 25 47.2 % 51 | Page F. Mini istraživanje MAS i sažetak rezultata, srpski Poštovani, Hvala Vam što ste prihvatili da učestvujete u evaluaciji projekta Inicijativa civilnog društva za javno zastupanje koji je od 2006. do 2013. godine sprovodio Institut za održive zajednice (ISC) u saradnji sa Građanskim inicijativama, Smart kolektivom, BCIF-om i Evropskim centrom za neprofitno pravo (ECNL-om). Sa Vama smo stupili u kontakt zato što smo Vašu organizaciju našli na spisku polaznika Mobilne škole za javno zastupanje. Molimo Vas da odvojite 10-ak minuta vremena i odgovorite na pitanja koja će nam pomoći u evaluaciji. Uveravamo Vas da će Vaši odgovori biti anonimni i da se neće koristiti pojedinačno već samo kao deo zbirnih podataka, bez mogućnosti da se odgovor poveže sa Vašom organizacijom. Hvala Vam na iskrenom i blagovremenom odgovoru. Molimo Vas da popunjeni upitnik pošaljete što pre na adresu evaluacijacsai@gmail.com. Stalo nam je do Vašeg mišljenja i zahvaljujemo Vam se na saradnji. Srdačan pozdrav od Evaluacionog tima organizacije Social Impact Broj Pitanje Rezultati 1 Naziv organizacije Broj anketiranih 53 (ukupan) 2 Kako su vas kontaktirali za Mobilnu školu za javno zastupanje (MAS)? Broj anketiranih % od ukupnog broja ispitanika 2a) Direktno od BCIF-a 12 22.6 % 2b) Mailing liste, obaveštenja, vebsajt, od drugih organizacija 33 62.3 % 2c) Drugo, molimo navedite 8 15.1 % 3 Koje ste teme obradili na treningu? Broj anketiranih % od ukupnog broja ispitanika 3a) Javno zastupanje 43 81.1 % 3b) Upravljanje finansijama i izveštavanje 28 52.8 % 3c) Komunikacije i rad sa medijima 31 58.5 % 3d) Upravljanje projektom 36 67.9 % 3e) Izrada projekta, prikupljanje sredtsava – fundraising 42 79.3 % 3f) Drugo, molimo navedite 9 17.0 % 4 Molimo Vas da navedete primer drugog projekta (bez obzora na to koji ga je donator finansirao) u kome ste primenili stečena znanja ili veštine sa MAS treninga. Odgovori variraju 5 Da li je Vaša organizacija podnela predlog projekta BCIF-u? Broj anketiranih % od ukupnog broja ispitanika 5a) Da i finansiran je 17 32.1 % 5b) Da i odbijen je 9 17.0 % 5c) Nije poslala 24 45.3 % 6 Da li biste projekat ili projekte koji-e je BCIF podržao realizovali i bez njihove podrške? Broj anketiranih % od ukupnog broja ispitanika 6a) Da 19 35.9 % 6b) Ne 25 47.2 % 52 | Page Annex IV: Sources of Information Persons Interviewed Name Organization/Affiliation Aleksandar Popov Center for Regionalism Aleksandra Veselinovic ISC Alisa Halak Center for Development of CS Ana Koeshall Foundation of Ana and Vlade Divac Biljana Stojanovic Kikinda Youth Initiative Biljana Vickovic Association Oktoopus( former Association of Citizens SFERA No) Bogdan Gavanski ISC Bojan Boskovic EXIT Bojana Ruzic Center for Democracy Foundation Boris Ilijevski Resource Center Majdanpek Budimir Ivanisevic Humanitarian Law Center, Belgrade Coordinator/activist Informal Group-Equal Opportunities in sport for women and girls on Vracar Danijela Stojicic Resource Center Bor Danko Nikolic Zajecarska Inicijativa Darko Radicanin Junior Achievement Darko Savic Center for Creative Development Knjazevac Dejan Matic The Third Square Deputy of Ana Sacipovic Roma Society OSVIT Djordje Milic The Scout Organization Dragana Stevanovic Kolakovic CSAI Agreement Officer’s Representative (AOR) Dragan Djordjevic Human Rights Committee Dragan Golubovic Former ECNL employee; Consultant Dragan Lukic Association for Promotion of Inclusion in Serbia-API Dragan Randjelovic Society of Young Researchers, Bor Dragan Sreckovic ISC Dragan Taskov Society for Environmental Protection Stara Planina Dubravka Velat Civic Initiatives Goran Mitrovic Peoples' Parliament Leskovac Gordan Paunovic B 92 Foundation Gordan Velev Group 484 53 | Page Igor Jezdimirovic Engineers of Environmental Protection Novi Sad Ivan Knezevic European Movement Ivan Svetozarevic Green Zone Knjazevac Ivana Cirkovic Office for Cooperation with CS Ivana Krstic Club for Synchronized Swimming Vracar Ivana Trkulja SOS Women's Center Jadranka Jelincic Fund for an Open Society Jasna Filipovic Center for Applied European Studies Jasna Isailovic ISC Jovana Vukovic Regional Center for Minorities Katarina Jelovac Association of Sports Trainers Ljiljana Majkic Informal Group Kikinda Forum Maja Dzelatovic Former Marketing and Communication Manager at Holcim; Currently freelance consultant for CSP Marija Cecen B 92 Foundation Marijana Tufonic-Pavlovic Association for Majdanpek Development Marina Dovljanski Academy of Women's Entrepreneurship Marko Blagojevic CESID Marko Saric Judges Association of Serbia Mia Vukojevic BCIF Milan Antonijevic YUCOM Milija Todorov Civic Library PIRGOS Mira Novakovic-Ilin Novi Sad Humanitarian Center Mladen Jovanovic National Coalition for Decentralization Nathan Koeshall Balkan Trust for Democracy Nebojsa Trailovic Informal group Natalino Nenad Stojanovic Proaktiv Neven Marinovic Smart Kolektiv Nikola Dojcinovic Center for Sustainable Future-Green Zone Knjazevac Nikola Petrovic Kosmopolis Nina Topic ISC Olivera Milosevic Green Key Radoslava Aralica Zrenjanin's Educational Center Rhonda Schlangen ISC Sandra Stanic ISC Sinisa Kockar Kikinda Mammoths Slavisa Jeremic Informal Group Neighbourhood Council 54 | Page Slobodanka Macanovic Autonomous Women's Center Snezana Pavkovic Timok Club Sonja Konakov-Svircev Erste bank Svetlana Djukic Delegation of EU Tanja Bjelanovic BCIF Tanja Petrovic Young Researchers of Serbia Violeta Jovanovic NALED Visnja Sijacic Citizens' Association kuda.org/ck 13 Vladan Joler Share Foundation Vukosava Crnjanski CRTA Walter Doetsch USAID Serbia, Democracy and Governance Office Zarko Sunderic SIPRU Zeljka Burgund Professional Association for Support of Children and Family FICE Serbia Zoltan Kovac Focus M Zorica Zenulovic Sunrise 55 | Page Documents Reviewed Program documents 1. Civil Society Advocacy Initiative (CSAI) Program Assessment. Prepared for U.S. Agency for International Development, Prepared by Brooks, Amber and Stern, Cara. July 15, 2010. 2. CSAI Quarterly Report: January-March 2013. Prepared for U.S. Agency for International Development (USAID), Prepared by Institute for Sustainable Communities (ISC). April 30, 2013. 3. CSAI Quarterly Report: October-December 2012. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. January31, 2013. 4. CSAI Quarterly Report: July-September 2012. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. October 31, 2012. 5. CSAI Quarterly Report: April-June 2012. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. August 1, 2012. 6. CSAI Quarterly Report: January-March 2012. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. May 3, 2012. 7. CSAI Quarterly Report: October-December 2011. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. February 1, 2012. 8. CSAI Quarterly Report: July-September 2011. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. October 17, 2011. 9. CSAI Quarterly Report: April – June 2011. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. July 31, 2011. 10. CSAI Quarterly Report: January-March 2011. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. April 29, 2011. 11. CSAI Quarterly Report: October-December 2010. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. February 1, 2011. 12. CSAI Quarterly Report: July-September 2010. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. November 1, 2010. 13. CSAI Quarterly Report: April – June 2010. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. July 30, 2010. 14. CSAI Quarterly Report: January-March 2010. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. April 30, 2010. 15. CSAI Quarterly Report: October-December 2009. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. February 3, 2010. 16. CSAI Quarterly Report: July-September 2009. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. October 30, 2009. 56 | Page 17. CSAI Quarterly Report: April-June 2009. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. July 31, 2009. 18. CSAI Quarterly Report: January-March 2009. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. April 30, 2009. 19. CSAI Quarterly Report: October-December 2008. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. January 31, 2009. 20. CSAI Quarterly Report: July-September 2008. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. October 31, 2008. 21. CSAI Quarterly Report: April-June 2008. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. August 15, 2008. 22. CSAI Quarterly Report: January-March 2008. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. May 15, 2008. 23. CSAI Quarterly Report: October-December 2007. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. January 31, 2008. 24. CSAI Quarterly Report: July-September 2007. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. November 9, 2007. 25. CSAI Quarterly Report: April-June 2007. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. July 31, 2007. 26. CSAI Quarterly Report: January-March 2007. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. April 30, 2007. 27. CSAI Quarterly Report: October-December 2006. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. January 31, 2007. 28. CSAI Quarterly Report: July-September 2006. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. October 31, 2006. 29. Grants Administration Policy and Procedure Manual (CSAI). Prepared by ISC. October 2008. 30. Advocacy in Practice Grants Recommended Proposals 2007, CSAI. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. 2007. 31. Advocacy in Practice Grants Recommended Proposals 2008, CSAI. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. January 24, 2008. 32. Advocacy in Practice Grants Recommended Proposals 2009, CSAI. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. February 2009. 33. Advocacy in Practice Grants Recommended Proposals 2009, CSAI. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. 2010. 57 | Page 34. Advocacy in Practice Grants Recommended Proposals 2009, CSAI. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. 2011. 35. Advocacy in Practice Grants Recommended Proposals 2009, CSAI. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. 2012. 36. Advocacy in Practice Grants Recommended Proposals 2009, CSAI. Prepared for U.S. Agency for International Development (USAID), Prepared by ISC. 2013. 37. Republic of Serbia Civil Society Assessment Report. Prepared for U.S. Agency for International Development (USAID) Serbia and Montenegro. Prepared by Barnes, Catherine. February 28, 2011. 38. Work Plan July 2006-2007, CSAI. Prepared for U.S. Agency for International Development Serbia, Prepared by The Institute for Sustainable Communities. November 20, 2006. 39. Work Plan Year 2, CSAI. Prepared for U.S. Agency for International Development Serbia, Prepared by The Institute for Sustainable Communities. June 30, 2007. 40. Work Plan Year 3, CASI. Prepared for U.S. Agency for International Development Serbia, Prepared by The Institute for Sustainable Communities. October 10, 2008 41. Work Plan Year 4, CSAI. Prepared for U.S. Agency for International Development Serbia, Prepared by The Institute for Sustainable Communities. October 13, 2009. 42. Work Plan Year 5, CSAI. Prepared for U.S. Agency for International Development Serbia, Prepared by The Institute for Sustainable Communities. September 30, 2010. 43. Work Plan Year 6, CSAI. Prepared for U.S. Agency for International Development Serbia, Prepared by The Institute for Sustainable Communities. August 12, 2011. 44. Work Plan Year 7, CSAI. Prepared for U.S. Agency for International Development Serbia, Prepared by The Institute for Sustainable Communities. May 3, 2012. 45. Modification of Assistance Number 7, Award Number 169-A-00-06-00104-00, CSAI. Administered by: U.S. Agency for International Development, Granted to: ISC. September 2008. 46. Modification of Assistance Number 7, Award Number 169-A-00-06-00104-00, CSAI. Administered by: U.S. Agency for International Development, Granted to: ISC. September 2008. 47. Modification of Assistance Number 10, Award Number 169-A-00-06-00104-00, CSAI. Administered by: U.S. Agency for International Development, Granted to: ISC. September 2009. 48. Award of CSAI. Prepared by: U.S. Agency of International Development (USAID), Prepared for: Institute for Sustainable Communities. May 31, 2006. 49. Proposal to USAID for Cost Extension to the: Serbia Civil Society Advocacy Initiative. Agreement No. 169-A-00-06-00104-00. Prepared for U.S. Agency for International Development, Prepared by: ISC. April 12, 2010. 58 | Page 50. Modification of Assistance Number 12, Award Number 169-A-00-06-00104-00, CSAI. Administered by: U.S. Agency for International Development, Granted to: ISC. September 2010. References Cited Antevski, Miroslav. EKONOMSKI HORIZONTI, 2010, 12, (1) p. 47. Barnes, Catherine (October 2011), ‘Civil Society Assessment Report,’ prepared for USAID Serbia and Montenegro. BCIF, ISC and USAID (December 2012), ‘Individual and Corporate Philanthropy in Serbia: Practices and Attitudes of Citizens and Company Representatives,’ Ipsos Strategic Marketing, http://www.tragfondacija.org/media/PDF/BCIF%20-%20Research%20on%20philanthropy.pdf. Civic Initiatives (2009), ‘Public Perception and Attitudes Towards NGO Sector in Serbia in 2009,’ ISC and USAID. Civic Initiatives and Office for Cooperation with Civil Society (September 2011), ‘Assessment of Conditions in Civil Society Organizations and the Civil Society Sector in Serbia, September 2011,’ ISC and USAID. CRNPS Directory (2005). Fiscal Council, Republic of Serbia http://fiskalnisavet.rs/en/. Freedom House (2001), ‘Annual Report 2001,’ http://www.freedomhouse.org/sites/default/files/inline_images/2001.pdf. INDIVIDUALNA I KORPORATIVNA FILANTROPIJA U SRBIJI, Praksa i stavovi gradjana i predstavnika kompanija, December 2012; http://www.bcif.org/site/media/PDF/BCIF%20- %20Istrazivanje%20o%20filantropiji.pdf. Institute for Sustainable Communities (2013), ‘Civil Society Advocacy Initiative: Who We Are,’ http://csai.iscserbia.org/who_we_are/. Jadranka Jelincic, Open Society Fund, key informant interview conducted on June 3, 2013. Republic of Serbia, Civil Society Assessment Report prepared by Catherine Barnes for USAID Serbia and Montenegro, October 2011 https://serbia.usaid.gov/upload/documents/CS_Assess_FINAL-revOct2011.pdf. Milivojević, Zdenka, (2007), ‘Civicus Civil Society Index - A Report for’ Serbia’ http://www.crnps.org.rs/wp-content/uploads/civicus_sr2007.pdf. Stachowiak, Sarah (2007), ‘Pathways for Change: 6 Theories about How Policy Change Happens,’ Organisational Research Services, http://www.organizationalresearch.com/publicationsandresources/pathways_for_change_6_theo ries_about_how_policy_change_happens.pdf. 59 | Page USAID. The Civil Society Advocacy Initiative, 2013 http://csai.iscserbia.org/who_we_are/. 60 | Page Annex V: Disclosure of Conflict of Interest Green, Andrew: Evaluation Team Leader 61 | Page Nikolin, Sanja: Evaluation Team Member 62 | Page Annex VI: Coverage Tables Table 1. Comparison of Total Grants to Evaluation Interview Grants, in dollars Total Grants Region IP AP SI IF SG EurInt APJI Central Serbia $3,502,099 $1,460,946 $4,212,212 $437,071 $186,183 $159,016 $186,000 Eastern Serbia $102,744 $192,241 $19,550 $19,000 Southern Serbia $547,503 $351,140 $13,815 $24,971 $70,000 Vojvodina $450,623 $680,147 $163,422 $56,737 $43,606 Intl $381,646 $159,043 Grand Total $3,883,745 $2,561,816 $5,594,784 $614,307 $262,470 $227,593 $275,000 Total Grants, cont. Region EJI SERJI BCIF GRAG BCIF GL BCIF ACAP BCIF PCAP BCIF SF Grand Total Central Serbia $223,725 $220,333 $53,234 $27,306 $117,480 $6,250 $10,791,855 Eastern Serbia $54,666 $28,071 $31,645 $50,845 $40,959 $539,721 Southern Serbia $52,410 $71,106 $90,285 $147,987 $27,756 $17,940 $1,414,913 Vojvodina $95,976 $20,504 $128,647 $26,696 $11,908 $1,678,267 Intl $540,689 Grand Total $276,135 $274,999 $248,387 $169,740 $444,960 $95,410 $36,098 $14,965,445 Evaluation Interview Grants Region IP AP SI IF SG EurInt APJI Central Serbia $3,502,099 $612,281 $2,319,945 $166,870 $70,630 $51,946 $145,000 Eastern Serbia $73,054 $144,685 $19,550 $19,000 Southern Serbia $194,513 $69,390 $70,000 Vojvodina $212,629 $425,878 $83,422 Intl $381,646 Grand Total $3,883,745 $1,092,477 $2,959,898 $250,292 $90,180 $51,946 $234,000 % Interviewed 100% 43% 53% 41% 34% 23% 85% Evaluation Interview Grants, cont. Region EJI SERJI BCIF GRAG BCIF GL BCIF ACAP BCIF PCAP BCIF SF Grand Total Central Serbia $163,892 $145,000 $21,601 $6,245 $33,489 $6,250 $ 7,245,247 Eastern Serbia $54,666 $11,654 $20,511 $35,788 $40,959 $419,866 Southern Serbia $11,449 $5,717 $44,524 $13,867 $409,459 Vojvodina $9,213 $7,491 $25,546 $26,696 $11,908 $802,783 Intl $381,646 Grand Total $163,892 $199,666 $53,916 $39,964 $139,346 $81,521 $18,158 $9,259,001 % Interviewed 59% 73% 22% 24% 31% 85% 52% 62% 63 | Page Table 2: Comparison of Total Grants to Evaluation Interview Grants, in #Grants Covered by Interviews (Not #Interviews) Total Grants Region IP AP SI IF SG EurInt APJI EJI SERJI BCIF GRAG BCIF GL BCIF ACAP BCIF PCAP BCIF SF Grand Total Central Serbia 3 33 71 15 8 6 3 4 4 13 7 10 1 178 Eastern Serbia 3 11 1 1 2 7 9 4 3 41 Southern Serbia 12 15 1 2 1 1 14 28 12 2 3 91 Vojvodina 12 16 4 3 2 23 6 11 2 3 82 Intl 1 3 4 Grand Total 4 60 116 20 12 10 5 5 6 57 50 37 7 7 396 Evaluation Interview Grants Region IP AP SI IF SG EurInt APJI EJI SERJI BCIF GRAG BCIF GL BCIF ACAP BCIF PCAP BCIF SF Grand Total Central Serbia 3 14 32 5 3 2 2 3 2 4 1 2 1 74 Eastern Serbia 2 9 1 1 2 3 6 3 3 30 Southern Serbia 4 3 1 2 2 3 1 16 Vojvodina 5 7 2 3 2 2 2 3 26 Intl 1 1 Grand Total 4 25 51 7 4 2 4 3 4 12 11 10 6 4 147 % Interviewed 100% 42% 44% 35% 33% 20% 80% 60% 67% 21% 22% 27% 86% 57% 37% 64 | Page Table 3: Comparison of Total Grants to Evaluation Interview Grants by Year, in dollars Total Grants Granter Type (short) 2006 2007 2008 2009 2010 2011 2012 2013 Grand Total ISC IP $3,883,745 $3,883,745 AP $718,157 $382,097 $604,266 $857,296 $2,561,816 SI $466,648 $303,140 $846,334 $929,565 $546,019 $652,068 $1,759,118 $91,892 $5,594,784 IF $168,557 $248,019 $49,367 $148,364 $614,307 SG $262,470 $262,470 EurInt $67,462 $160,131 $227,593 APJI $275,000 $275,000 EJI $276,135 $276,135 SERJI $274,999 $274,999 ISC Total $4,350,393 $1,021,297 $1,228,431 $1,769,850 $2,073,935 $977,570 $2,457,481 $91,892 $13,970,849 BCIF BCIF GRAG $23,946 $143,793 $80,648 $248,387 BCIF GL $25,065 $49,994 $70,190 $6,491 $18,000 $169,740 BCIF ACAP $58,818 $249,120 $137,022 $444,960 BCIF PCAP $95,410 $95,410 BCIF SF $36,098 $36,098 BCIF Total $23,946 $143,793 $139,466 $274,185 $187,016 $201,698 $6,491 $18,000 $994,595 Grand Total $4,374,339 $1,165,091 $1,367,897 $2,044,035 $2,260,952 $1,179,267 $2,463,973 $109,892 $14,965,445 Evaluation Interview Grants Granter Type (short) 2006 2007 2008 2009 2010 2011 2012 2013 Grand Total ISC IP $3,883,745 $3,883,745 AP $317,367 $180,141 $324,655 $270,314 $1,092,477 SI $360,751 $303,140 $210,036 $647,915 $195,724 $248,400 $952,764 $41,168 $2,959,898 IF $108,759 $91,774 $24,759 $25,000 $250,292 SG $90,180 $90,180 EurInt $38,846 $13,100 $51,946 APJI $234,000 $234,000 EJI $163,892 $163,892 SERJI $199,666 $199,666 ISC Total $4,244,496 $620,507 $390,177 $1,120,174 $661,092 $437,051 $1,411,430 $41,168 $8,926,095 ISC % Interviewed 98% 61% 32% 63% 32% 45% 57% 45% 64% BCIF BCIF GRAG $21,432 $32,484 $53,916 BCIF GL $4,136 $13,542 $12,794 $6,491 $3,000 $39,964 BCIF ACAP $71,245 $68,101 $139,346 BCIF PCAP $81,521 $81,521 BCIF SF $18,158 $18,158 BCIF Total $21,432 $32,484 $75,381 $81,643 $112,474 $6,491 $3,000 $332,906 BCIF % Interviewed 0% 15% 23% 27% 44% 56% 100% 17% 33% Grand Total $4,244,496 $641,939 $422,661 $1,195,556 $742,735 $549,525 $1,417,921 $44,168 $9,259,001 Total % Interviewed 97% 55% 31% 58% 33% 47% 58% 40% 62% U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523