APRIL 2017 This publication was produced for review by the United States Agency for International Development. It was prepared by Democracy International, Inc. SRI LANKA FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY FINAL REPORT Submitted to: USAID/Sri Lanka Prepared by: Melanie Reimer, Team Leader Shevon Gooneratne, Civil Society Expert Verni Vijayarajah, Democracy, Rights and Governance Expert Contractor: Democracy International, Inc. 7600 Wisconsin Avenue, Suite 1010 Bethesda, MD 20814 Tel: 301-961-1660 www.democracyinternational.com FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY FINAL REPORT APRIL 2017 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT TABLE OF CONTENTS EXECUTIVE SUMMARY................................................................................................................................................... I EVALUATION PURPOSE AND QUESTIONS..........................................................................................................1 PROJECT BACKGROUND............................................................................................................................................1 EVALUATION METHODOLOGY...............................................................................................................................2 FINDINGS AND CONCLUSIONS ..............................................................................................................................5 RECOMMENDATIONS.................................................................................................................................................25 ANNEX A: STATEMENT OF WORK.................................................................................................................... A-1 ANNEX B: EVALUATION TEAM, DESIGN AND METHODOLOGY...........................................................B-1 METHODOLOGY ........................................................................................................................................................B-6 ANNEX C: DATA COLLECTION TOOLS .......................................................................................................... C-1 ANNEX D: EVALUATION INFORMANTS BY CATEGORY..........................................................................D-1 ANNEX E: LIST OF DOCUMENTS REVIEWED.................................................................................................. E-1 ANNEX F. DESCRIPTION OF DEVELOPMENT GRANTS PROGRAM........................................................ F-1 ANNEX G: RESULTS FRAMEWORK AND SUMMARY OF SPICE................................................................G-1 ANNEX H: EXAMPLE OF FOG MILESTONES.................................................................................................... H-1 ANNEX I: SPICE CAPACITY DEVELOPMENT RECIPIENTS BY CATEGORY ............................................ I-1 ANNEX J: ROLE OF ORGANIZATIONAL DEVELOPMENT FACILITATORS............................................ J-1 ANNEX K: SUMMARY OF CORE VALUES TRAINING....................................................................................K-1 ANNEX L: CONFLICT OF INTEREST STATEMENTS....................................................................................... L-1 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT ACKNOWLEDGEMENTS This evaluation would not have been possible without the generous contributions of the current and former staff of USAID/Sri Lanka and the key implementers of SPICE, whether in Colombo, Washington DC, or on assignment elsewhere in the world. Although the project was in a hectic period of closing down during the evaluation field work, key personnel still made time for our team, in person and by email, sometimes repeatedly. In addition to patiently and candidly answering many questions and sharing numerous documents, they helped to link us with the right people to help us in gaining an overall perspective of the project and its context. Of course, many other informants across Sri Lanka from civil society and other sectors gave their time freely to participate in interviews, in some cases also responding to online surveys. We especially valued the contribution of beneficiaries and volunteers at the grassroots level, who shared their stories with our team. Although they may never perceive any direct benefit of this process, the participation of each and every individual added value to the evaluation, which it is hoped will ultimately contribute to an ever-stronger civil society sector in Sri Lanka. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT ACRONYMS AOR Agreement Officer’s Representative APS Annual Program Statement BMR Benchmark Review CBO Community-based Organization CBP Capacity Building Provider CSO Civil Society Organization CSOSI Civil Society Organizations Sustainability Index CSR Corporate Social Responsibility CTF Consultation Task Force on Reconciliation Mechanisms DGP Development Grants Program DGPE Development Grants Program Evaluation DRG Democracy, Human Rights and Governance FAA Fixed Amount Award FGD Focus Group Discussion FOG Fixed Obligation Grant FY Fiscal Year (refers to U.S. Government fiscal year) GBV Gender-based Violence KII Key Informant Interview ICNL International Center for Not-for-Profit Law IDF Institutional Development Framework IFES International Foundation for Electoral Systems KMP Knowledge Management Portal LTTE Liberation Tigers of Tamil Eelam LWA Global Civil Society Strengthening Leader with Associates (LWA) M&E Monitoring and Evaluation MSI Management Systems International NGO Nongovernmental Organization OCA Organizational Capacity Assessment ODF Organizational Development Facilitator OIP Organizational Improvement Plan PMP Performance Management Plan RBA Rights-based approaches FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT RFA Request for Application SOW Scope of Work SPICE Support for Professional and Institutional Capacity Enhancement UN United Nations USAID United States Assistance for International Development U.S. United States VSSO Voluntary Social Service Organizations Act FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT i EXECUTIVE SUMMARY Evaluation Purpose, Questions and Methodology The purpose of this evaluation is to document best practices and lessons learned in enhancing civil society space through USAID/Sri Lanka’s Support for Professional and Institutional Capacity Enhancement (SPICE) Activity, and develop recommendations to inform future civil society programming. The priority questions can be summarized as: 1) SPICE’s key successes, challenges and lessons learned; 2) relative strengths and weaknesses of CSO capacity development tools and approaches targeting Development Grants Program (DGP) and SPICE grantees; 2a) effectiveness of SPICE’s approach to capacity development; 2b) remaining civil society gaps; and 3) best practices and recommendations for future civil society programming in Sri Lanka. The Evaluation Team used four main data collection methods: document review, key informant interviews (57), focus group discussions (19), and an online survey of 23 DGP and SPICE grantees. Information from these methods and various informants has been triangulated to ensure reliable findings. The Team conducted the bulk of in-person data collection in Northern Province, Eastern Province, and the greater Colombo area during three weeks of field work in February and March 2017. Project Description USAID/Sri Lanka designed SPICE in mid-2012, to address the problem of alarmingly diminished space for effective civic engagement and the protection of citizen rights. The project began in December 2012 and is to close in June 2017; total approved funding was $15,182,000. SPICE aimed to expand space for independent collective action by citizens, protect citizens’ rights and support critical voices by focusing on the key issues of governance, inclusion and reconciliation. The project purpose includes an emphasis on conflict-affected regions, thus its original mandate was to focus on civil society in the north and east of the country, while working with strategically important national CSOs. In early 2014, SPICE was modified to provide capacity enhancement services to 14 DGP recipients, direct grantees of USAID. Starting in 2014, SPICE gradually extended its geographic scope to include the south and center of the country. Since 2015, Sri Lanka has undergone dramatic political change; there has been a democratic opening and increased space for civil society, though the transition is still ongoing. Findings and Conclusions Question 1: What have been the SPICE activity’s key successes, challenges, and lessons learned? Analysis of this question focused on SPICE funding support to its 67 CSO grantees. Question 1 Findings A. Policy advocacy and reform The space for civil society influence on policy prior to 2015 was extremely limited by the repressive environment, on both national and regional levels. In spite of the obstacles, in 2013-2014 Sri Lankan CSOs were able to maintain a certain level of political pressure on the government. SPICE provided both financial, technical and moral support to some prominent CSOs speaking out publicly on such issues during that period. Interviewees agreed that civil society made a major contribution to raising public awareness of the political situation during that period. Those involved included some SPICE-funded CSOs, alongside others. Following the transition, many SPICE grantees have engaged actively in various nationwide dialogues and reform processes, including in successful advocacy for the 19th amendment to the Constitution and for the Right to Information Act. However, concrete changes in government policy have been more difficult to achieve in relation to some other core issues on which SPICE grantees have advocated: much land is still occupied by the military in spite of government policy; the recent Consultation Task Force on Reconciliation Methods (CTF) report recommendations were largely not accepted by the government; and although some FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT ii positive steps have been taken, major decisions and actions are still pending in the complex and sensitive domain of transitional justice. SPICE also funded a wide range of “regional CSOs” based outside of Colombo, many of whom worked with local authorities to tackle local-level problems for their constituencies, rather than using confrontational advocacy methods. In addition to achievements on that level, some regional grantees contributed to the above-described national advocacy initiatives, for example by making submissions to consultative bodies and mobilizing citizens. B. Citizen awareness and mobilization SPICE has supported substantial efforts by grantees to inform and mobilize citizens across diverse topics. Examples include fostering groups of volunteer human rights defenders; mobilizing people to demand for basic needs to be met; and training village women’s groups to respond to gender-based violence incidents. One strategy used by CSOs was to organize or strengthen groups at community or division levels to discuss and try to address priority needs. SPICE reported that as of December 2016, 89 percent of civic associations targeted by the project had been “active in community development and governance.” On the other hand, in 2013-14, CSOs found it difficult to organize public meetings on any subject, especially in the north. While the situation has eased considerably, grantees in the north and east continue to report impromptu visits and other forms of intimidation by security forces. C. Human rights SPICE supported grantees to work on human rights issues from a variety of angles, ranging from political prisoners to land rights (among other issues), from national to local levels. Notable achievements include production of videos on torture and documentation of disappeared persons; and assisting thousands of people to defend their rights. For example, with grantees’ help, 45,552 people received legal aid or victims’ assistance. At the national level, SPICE grantees have produced research on a range of human rights issues and participated in CSO coalitions submitting reports for the Universal Periodic Review and other United Nations (UN) forums. SPICE adopted a cross-cutting strategy that promoted use of rights-based approaches (RBA), which was particularly important with organizations heavily focused on service delivery. As a result, various CSOs moved towards RBA and advocacy as core strategies. D. Coalition-building and networking SPICE staff described an “organic approach” to networking, intentionally focused on facilitating linkages (mostly informal) in response to opportunities rather than on actively promoting or establishing new formal networks. Overall, informants did not highlight this as an area where SPICE had made significant impact, but in general, most felt their existing networks were sufficient. SPICE did help strengthen links between CSOs at the regional level with new or existing community-based groups, both formal and informal. It also contributed to building bridges across ethnic divides. The challenges to networking were multiple, especially prior to the political transition when CSOs were reluctant to open up to new contacts who might not be trustworthy. On the other hand, some said the high-pressure situation served to unite many CSOs. E. Civil society space Expanding the space for collective citizen action was the overarching purpose of SPICE. There was ample evidence that the project played an important role in this area, primarily before but also after the political transition. The mere fact of funding key organizations sufficiently to keep them active and engaged on national and local issues was a major SPICE contribution, prior to 2015 , as other donors reduced or adjusted their support. MSI collaborated with ICNL as technical experts to work on the legal and regulatory framework for CSOs, once opportunities emerged in 2015. While CSOs have been appreciative of SPICE inputs and certain key government officials have shown some openness to reform during exchanges with SPICE, progress has been slow and informants expect plenty of resistance to any moves to facilitate CSO functioning with less government control. F. Grant-making in general FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT iii SPICE began by selecting 19 CSOs through non-competitive processes, to expedite project startup and keep a low profile for the project. Project designers felt that some key organizations needed immediate funds to remain operational, so both core and activity funding were provided. SPICE soon began to broaden its reach to offer project-focused grants to other organizations through competitive processes, initially using mailing lists. The SPICE focus on the north and east of the country gave rise to challenges: there was pressure to target Sinhala-dominated areas; activities aiming at reconciliation could not include those on both sides of the conflict other than through national grantees; and the operating environment in the north was extremely difficult. SPICE used Fixed Obligation Grants (FOG, now known as Fixed Amount Awards or FAA) for the majority of its grant making. This was intended to mitigate the risk of working with less experienced grantees, reduce the management burden of verifying every grantee expense, and to build grantee capacity. Most informants had misgivings about the use of FOGs, especially the heavy up-front work in finalizing the grant milestones and budget; the inflexibility of milestones in an unpredictable context; the time-consuming process of gathering evidence for the budget; and lack of available funds to cover shortfalls or delays. Almost all grants had to be modified at least once, and many had three or more modifications. That said, over time some grantees came to appreciate the way that FOGs required them to improve their planning and time management, and cut back on financial reporting. Diverse informants mentioned the short duration and gaps between rounds of grants as a constraint on completion of planned activities and achievement of results. Many grantees found SPICE grant reporting to be overly burdensome, especially given demands on their staff to deal with government reporting, the fluid environment, and other projects. SPICE grant making placed a major emphasis on gender, which was reinforced through capacity development activities. Project data shows that 20,000 out of 37,000 beneficiaries of grant activities were women, and that 36 percent of grants were in some way related to gender (43 grants). Question 1 Conclusions SPICE made a substantial contribution to the maintenance and later expansion of civil society space in Sri Lanka during its four years of activity. SPICE made a substantive impact in all five thematic areas described above, which is impressive given the constraints and risks of its early years. The most notable gains were in relation to human rights and citizen awareness and mobilization. Although the grants were highly disparate in nature—ranging from psychosocial counselling to preschool education, and from gender-based violence to voter education—SPICE promoted a cross-cutting theme of rights, and helped coalesce CSOs around certain thematic areas. There remains significant work to be done on the legal restrictions and operational control exerted by the government over civil society, especially in the north. With respect to the grants process, SPICE gave opportunities to a diverse set of organizations based on rigorous selection and proposal development processes. Use of the FOG mechanism for 85 percent of grants met a range of obstacles, including the short duration of grants, unpredictable situation, and inexperience with the milestone system. A huge up-front effort was needed to get FOGs successfully off the ground, and repeated modifications also proved time-consuming. A USAID informant observed that “SPICE made it work,” primarily due to the staff’s high level of commitment to the project. In retrospect, although CSOs with less financial management capacity would have needed support from SPICE to ensure compliance, simplified grants would have been a better choice for many SPICE grantees. Question 2: What conclusions and recommendations can be drawn about the relative strengths, weaknesses, and effectiveness of the various tools and approaches of building professional and institutional capacity of local organizations? The three main approaches studied for this question are: USAID direct capacity development support to DGP grantees, SPICE support to DGP grantees, and SPICE support to 26 targeted SPICE grantees. The findings and conclusions with respect to DGP support are based on a combination of primary data collected by this evaluation and comparative data gleaned from the DGP evaluation of 2016. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT iv Question 2 Findings USAID support to DGP grantees The first step by USAID was to facilitate an in-depth Organizational Capacity Assessment (OCA) with each grantee; they greatly appreciated its participatory nature and self-assessment approach. USAID staff then offered capacity support focused on financial management, internal administration and project monitoring and reporting. Grantees generally indicated that the quality and relevance of this support was high and that they valued this assistance, especially to enable them to handle USAID requirements. All 13 reported that standard operational procedures were strengthened, and that help with financial procedures helped enhance internal control systems and reporting. Familiarity of USAID staff with relevant regulations, templates and tools was a key advantage of this support. However, USAID staff had limited time and training for playing a capacity development role in addition to their other responsibilities. Engagement by USAID staff and grantees declined over time, especially after SPICE took on a significant role with these grantees. SPICE support to DGP grantees According to the DGP Evaluation, 12 grantees made progress in improving organizational sustainability, and those that undertook strategic planning (the most common intervention by SPICE, targeting eight grantees) said it helped them to proactively focus rather than simply move from project to project. Seven grantees were able to improve governance, and 12 grantees established detailed performance monitoring systems (with both SPICE and USAID support). The relevance of SPICE capacity support was considered high, and SPICE’s follow up of consultant work was effective in ensuring quality. However, there were no Organizational Development Facilitators (ODF) to play a constant supporting role with these CSOs (as described below for SPICE grantees). Moreover, the level of commitment by DGP grantees was highly variable; for some grantees, only one or two capacity development activities were conducted with SPICE support, while others received up to six interventions. Another challenge was posed by the limited pool of available skilled consultants, which had to be stretched by SPICE to meet the needs of its own grantees and the DGP grantees. There was some indication that the addition of DGP responsibilities had made it more difficult for SPICE to concentrate on its primary purpose. SPICE support to SPICE grantees (Question 2a) This sub-question focused on the 26 CSOs selected by SPICE for intensive capacity development. SPICE used a five-stage cycle to guide this intervention, moving from the initial meeting and needs assessment to development of a comprehensive Organizational Improvement Plan, then on to activities and monitoring, including periodic review of benchmarks. ODFs were a key aspect of the SPICE methodology; these part￾time consultants provided ongoing coaching, monitoring and liaison for grantees. The overall quality and relevance of capacity support to these grantees by SPICE was high. Key strengths included the customized approach to each CSO, reliance on coaching and mentoring, use of ODFs matched to each grantee, bundling of capacity support with grant funding, and close monitoring. Challenges included the shallow pool of talent available to be engaged as organizational development experts, some resistance to professionalization by activist organizations, and measuring change in capacity of targeted CSOs (outcomes as compared to outputs). The SPICE capacity development process paid little attention to technical capacity of these 26 CSOs to implement their “core business” as compared to their organizational or internal management capacity. This was a strategic decision based on various factors, including persistent government and media criticism of CSOs, including allegations of misuse of funds after the tsunami relief effort, and high levels of scrutiny by government authorities. Civil society capacity gaps (Question 2b) Informants and secondary sources consistently chose access to funding as the uppermost need of Sri Lankan civil society. Many highlighted the need for CSOs to generate their own income to reduce reliance on international donors, given that prospects for private sector support are still dim. Although there are signs of FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT v progress in the public image and communications of CSOs, most consider that significant work is required— not only with CSOs but with other sectors of society. Another dominant theme was the challenge faced by civil society in fully grasping the complex political situation, and defining their role—as individual organizations and collectively—in the new context. The need to strengthen internal governance was seldom mentioned by CSOs, but recognized by SPICE and donors as problematic. A full civil society assessment would shed additional light on these issues. Question 2 Conclusions Re DGP grantees. Notable increases in DGP grantee capacity were achieved due to USAID support, particularly in financial management and administration. However, it proved difficult for USAID staff to sustain the level of effort required to both manage and support a full range of capacity building of these grantees, as the context and grants evolved. The solution of contracting with SPICE was generally satisfactory for DGP, but resulted in added strain on key personnel of the project and slight reduction in support to SPICE grantees. SPICE’s support was highly valued and had positive effects on almost all DGP grantees in various areas of capacity. The evaluation observed limitations on the variable commitment by DGP grantee leadership, and the effects of staff turnover on sustainability of capacity changes. Re SPICE grantees. The project’s systems for ensuring quality of services and the relatively long intervention with each organization were key factors in its ability to achieve notable progress in grantee capacity. ODFs were an effective way of coaching and monitoring targeted CSOs, although SPICE had to make a major investment in training, coaching and monitoring of the coaches themselves. Comparatively greater advances were made in certain areas of governance (especially planning) and in management systems and procedures, including financial management. Fewer results were achieved in relation to external resources and financial resources, where the context posed substantial obstacles. Question 3: Best Practices and Lessons Learned The Team has identified various successful practices from the SPICE experience. They include:  The all-Sri Lankan in-country team and its awareness of the context and players  The low profile of the project  The wide diversity of grants, including thematic and geographic spread  Close supervision of capacity development and grants execution  Organic, responsive approach to network development  USAID willingness to take a risk on SPICE in a tense context for donors and civil society As for lessons learned, i.e. practices that should be carefully considered in future, they include:  Heavy reliance on Fixed Obligation Grants (Fixed Amount Awards)  Highly detailed monitoring requirements and reporting systems, including many indicators and multi￾layered milestones, with limited focus on overarching results Key Recommendations  Adopt a nationwide strategy for a potential future civil society support project  Continue financial and technical support to improve the legal enabling environment for civil society  Promote CSO synergies and coordination by facilitating the preservation and expansion of various types of interaction  Support CSOs to strengthen their constructive engagement (from collaboration, to coordination, to constructive critique and non-violent dissent) with government  Continue to integrate assistance with livelihoods of the most vulnerable as part of a broader strategy to strengthen civil society and citizen engagement  Continue with grants to CSOs, with a focus on the regions and on grassroots FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT vi  Continue to invest in capacity development of the sector as a complement to funding, including technical as well as management capacity  Prioritize CSO financial sustainability, including efforts to kindle corporate and individual philanthropy and a long-term strategy for enhancement of the visibility and credibility of civil society FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 1 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE The purpose of this performance evaluation is to document best practices and lessons learned in enhancing civil society space (programs and advocacy to protect citizens’ rights) through USAID/Sri Lanka’s Support for Professional and Institutional Capacity Enhancement (SPICE) Activity, and develop recommendations to inform future civil society programming. EVALUATION QUESTIONS The Evaluation Team (“the Team”) was asked by USAID to research a specific set of evaluation questions, within the framework of the overall purpose set out above. Those questions are: 1) What have been the SPICE activity’s key successes, challenges, and lessons learned? Responses shall focus on the following aspects of civil society support: (a) policy advocacy and reform; (b) citizen awareness and mobilization; (c) human rights prevention and response mechanisms; (d) human rights documentation and analysis; (e) enhancing civil society space; and (f) coalition-building and networking. 2) USAID/Sri Lanka has provided technical assistance for civil society capacity development using a variety of approaches. This includes capacity development of local organizations through the SPICE project, Development Grants Program (DGP), and directly led capacity development of local organizations by USAID Mission Staff. Using the DGP evaluation already conducted by USAID and findings from this evaluation, what conclusions and recommendations can be drawn about the relative strengths, weaknesses, and effectiveness of these tools and approaches of building professional and institutional capacity of local organizations? 2a) How effective has SPICE’s capacity development approach been in increasing civil society capacity in the following areas? Vision/ governance, Management resources, Human resources, Financial resources, External resources and Coalition-building and networking 2b) What are the remaining civil society gaps that could benefit from future support? 3) Based on the evaluation findings and current context, what best practices and recommendations should be considered for incorporation into future civil society programming in Sri Lanka? In order to facilitate and clarify the focus of the evaluation, the Team defined the above questions in further detail, as set out in the Evaluation Design and Methodology in Annex B. PROJECT BACKGROUND PROJECT DESCRIPTION1 USAID designed the SPICE Activity in mid-2012, aiming to address the problem of alarmingly diminished space for effective civic engagement and the protection of citizen rights. This was intended to support the major development objective that USAID identified in its Country Development and Cooperation Strategy (FY2011-2013): a strengthened relationship between the state and its citizens, which will allow for greater reconciliation and protection of citizen rights. The project began in December 2012 and is scheduled to close in March 2017; total approved funding was $15,182,000. Since the project was conceived, Sri Lanka has undergone considerable political change; there has been a democratic opening and increased space for civil society participation, though the transition is still ongoing and most of the change has not fully manifested in the country. While opening new opportunities, this FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 2 period has also introduced new challenges to civil society. A key obstacle is the gap between powerful governmental actors and groups that are able to represent citizen concerns and advocate on their behalf, such as civil society organizations. The overall purpose and outcomes, as described in the Scope of Work (SOW) for this evaluation, were as follows: Purpose: Expanded space for independent collective action by citizens, particularly in war affected areas, with regard to the key issues of governance, inclusion and reconciliation Outcome 1: Improved Protection of Citizens’ Rights by Sri Lankan Organizations Outcome 2: Enhanced civil society support for development, reconciliation, and governance Outcome 3: Enhanced management and technical capacity of Sri Lankan organizations supporting civic participation and inclusive development In addition, SPICE had a gender-related cross-cutting objective: Gender equality and women’s concerns, capacity and action better addressed by civil society. In February 2014, the budget was realigned, and the project description modified to provide institutional and management capacity enhancement services to 14 Development Grants Program (DGP) recipients, direct grantees of USAID/Sri Lanka (of which one also had a grant from SPICE). As described below, the evaluation questions also require consideration of the support provided to those organizations, in part relying on the DGP evaluation conducted in 2016. Originally slated as a three-year, $12,000,000 project to end in December 2015, SPICE has been extended three times: a three-month no-cost extension until March 2016, a one-year extension to March 2017 (which increased the budget to $15,182,000 and revised the project description to respond to the changed political situation), and a final no-cost extension to June 2017. However, these modifications did not alter the purpose and outcomes of SPICE. The SPICE cooperative agreement was implemented under the Global Civil Society Strengthening Leader with Associates (LWA) led by Counterpart International. Management Systems International (MSI) was the lead organization implementing this activity. LWA consortium members International Center for Non-Profit Law (ICNL) and International Foundation for Electoral Systems (IFES) also had minor roles in the project; ICNL to provide expert advice and support on legal regulatory frameworks, and IFES in collaboration with MSI to replicate a USAID initiative in India. EVALUATION METHODOLOGY The evaluation team consisted of three key experts: Team Leader, Ms. Melanie Reimer, an international expert with significant experience in civil society programming and in conducting similar evaluations; Mr. Shevon Gooneratne, an experienced lawyer and civil society specialist fluent in Sinhala and English; and Ms. Verni Vijayarajah, an expert in democracy, human rights and governance issues fluent in Tamil and English. In designing and implementing this evaluation, the Team has followed the guidelines outlined in USAID’s Task Order for this evaluation and accompanying Scope of Work (Annex A). The team used both quantitative and qualitative research methods to gather data to investigate the evaluation questions, as described in more detail in the Evaluation Design and Methodology (Annex B). The Team used four main data collection methods: document review, key informant interviews (KII), focus group discussions (FGD), and an online survey. Information from these methods has been triangulated to ensure reliable findings, while discrepant observations and data have been noted and used as well. The Team collected primary data over the course of three weeks of field work in Sri Lanka in February and March 2017. The Team conducted the bulk of in-person data collection in Northern Province, Eastern Province, and the greater Colombo area, and also held several remote interviews with key informants based in the USA or other countries. One team member briefly visited the southern part of the country to gather primary data about grantee work in those areas. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 3 DOCUMENT REVIEW The Team reviewed a comprehensive set of documents related to SPICE, including a variety of qualitative data such as quarterly progress reports, annual work plans and the Performance Management Plans (PMP), the SPICE mid-term review report, the DGP evaluation (DGPE) report, SPICE and DGP grantee descriptions, SPICE grant descriptions and reports, and capacity development assessments and plans. The document review also included a limited amount of quantitative data, notably those included in quarterly reports in relation to PMP indicator results. In addition, the Team reviewed a range of documents describing the rapidly evolving context of the country and situation of civil society, such as the 2014 and 2015 versions of the USAID Civil Society Organizational Sustainability Index for Asia. A full listing of documents reviewed is in Annex E. SELECTION OF INFORMANTS Considering the number of CSOs targeted by SPICE and the evaluation time constraints, the Team designed a robust method of selecting informants related to the CSO grants. From the total population of 67 SPICE grantees, the Team used purposive sampling to select 22 grantees as “cases” to be targeted for in￾person data collection. For this purpose, the Team relied primarily on a factsheet of information provided by USAID and the implementer, including: organization names and office locations; locations targeted by grant activities; type of capacity development assistance received; grant numbers, durations and sizes; and thematic area of activity. The Team divided SPICE grantees into two main categories: the 26 CSOs targeted with intensive capacity development support and the 41 that did not receive that support. Within each category, the Team chose the most diverse set of organizations possible, which formed the basis for targeting of both KIIs with the CSOs and FGDs with beneficiaries and volunteers, as described below. With respect to the DGP grantees, the focus was confined to informing Evaluation Question 2. The Team carried out KIIs with five of the 14 organizations included in this evaluation. Those five grantees were also chosen for their diversity, as they are based in various regions of the country and working in different thematic areas. KEY INFORMANT INTERVIEWS The Team conducted 57 in-depth interviews with a wide range of informants.2 In consultation with USAID, the Team selected interviewees who could provide substantive feedback on SPICE and its effects and who reflected diverse views. Stakeholders interviewed included: USAID and U.S. Embassy officials; current and former SPICE staff; SPICE subcontractors; other donor organizations; experts in civil society and governance issues; leaders of CSOs not funded by SPICE; and representatives of selected CSO grantees. The final category was by far the largest, accounting for 29 KIIs. In many cases, CSO leaders invited several staff or volunteers to participate in the interview, which meant that three or more persons often attended a single KII (though not all participated actively). Although this may have affected responses to some questions and required more time, the Team decided to respect the inclusive nature of these encounters and use this as an opportunity to canvass a variety of perspectives, going beyond the organization leadership. The key informants are described in more detail in Annex D. In all, 145 people participated in the KIIs: 81 males, 56 females, and eight unspecified. The Team conducted interviews in English, Tamil or Sinhala, using a semi-structured interview approach based on question guides designed to suit different categories of informants and respond to the core evaluation questions. A sample of those guides can be found in Annex C. MINI SURVEY The Team carried out a web-based survey consisting of 34 questions (mostly closed-ended), distributed among key staff of 13 DGP grantees and a sub-set of 10 SPICE grantees. The survey was conceived to supplement the information gathered through other methods, enriching the pool of data and allowing more grantees to have input into the evaluation. The Team selected the targeted CSOs and individuals in conjunction with USAID, considering three key criteria: the CSOs were well-established national level FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 4 organizations that received intensive capacity development support from SPICE; respondents were senior staff of those organizations, fluent enough in English to respond to closed-ended questions; and they had sufficiently advanced computer skills and connectivity to easily complete an online survey. The survey explored how the leaders and senior staff of these CSOs view the support received from USAID and/or SPICE to develop their organizational capacity, including assessments, training, coaching and technical assistance, with a specific focus on whether and how the capacity and work of those CSOs has been affected. As well, it sought the opinions of respondents on future civil society support priorities. The survey questions can be found in Annex C. The Team chose to use Survey Monkey™ to ensure a confidential, easily accessible, inexpensive, and rapid means of collecting and analyzing data. That tool was supplemented by telephone and email follow-ups, although it proved difficult to significantly boost response rates. The survey link was ultimately sent to 56 individuals identified by SPICE and USAID, representing 23 CSOs.3 Responses were obtained from 24 individuals for a 43 percent response rate, including partial responses.4 The respondents represented 17 different CSOs, including 10 DGP grantees. FOCUS GROUP DISCUSSIONS The Team conducted 19 FGDs to gather more in-depth qualitative information from selected categories of project stakeholder and verify data obtained through other methods. The main target population for these discussions was beneficiaries of or participants in SPICE-funded activities led by grantees (15 FGDs), with a focus on assessing their experience with the organizations and projects, including benefits and possible disadvantages of the activities. As well, the Team held FGDs with the “program team” of SPICE to probe the details of the grants component, with partner or member CSOs of grantee organizations, and with a sample of Organizational Development Facilitators (ODF) contracted to support the capacity development of SPICE grantees. The Team held FGDs in Tamil, Sinhala or English according to the preferences of informants, in the north, east and south of the country, as well as two in Colombo. In all, 175 people participated in FGDs: 101 males, 70 females and four unspecified. Questions were customized according to the specific engagement of the participants in the project. LIMITATIONS As with all evaluations, certain limitations have affected data collection and analysis. The key constraints in this case are outlined below. 1. Although very few informants declined to participate in the evaluation, there is a possibility of selection bias, i.e. those respondents who chose to participate might differ from those who did not in terms of their attitudes and perceptions, socio-demographic characteristics and experience, among other factors. In mitigation, the Team made informants aware of the independence of the process and the confidentiality of responses. 2. There is a known tendency among respondents to under-report socially undesirable answers and alter responses to approximate the social norm (social desirability bias). The extent to which respondents are prepared to reveal their true opinions may also vary for questions that call upon respondents to assess the performance of colleagues or people on whom they depend for services or funding. To mitigate this, the Team provided respondents with confidentiality guarantees, conducted interviews in settings where respondents felt comfortable, and aimed to establish rapport with respondents. 3. The Team worked with CSO grantees to identify beneficiaries and volunteers to participate in FGDs, which was necessary on a practical level but almost certainly led to increased levels of selection and response bias. In at least two FGDs, grantee staff were present. The Team sought to mitigate by clearly communicating the purpose and independent nature of the evaluation, seeking to establish rapport with participants, and highlighting the importance of honest responses to allow for identification of lessons learned. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 5 4. The scope of this evaluation did not include meeting with any parliamentarians or officials of the Government of Sri Lanka, nor any media representatives. This limited the ability to capture certain perspectives on both national and local levels. The Team relied on other informants as well as publications to get a general understanding of relations between CSOs, government and media. 5. Because SPICE has not yet closed, the final report on the activity was not available to the evaluation. Access to that report could have added depth to analysis of SPICE, especially considering that progress reporting is quarterly and tends to focus on short-term achievements. The Team mitigated this by consulting with key implementer staff in Colombo and Washington, as well as USAID/Sri Lanka staff. 6. The Performance Management Plan (PMP) was found to be of limited usefulness in terms of assessing results of SPICE. The Team found various overlaps among the 23 indicators (including three at purpose level), and at least 15 indicators measured outputs as compared to higher level outcomes or results. Targets were amended significantly over time, particularly after the political transition, and large variances between targets and progress were typical. Nevertheless, the Team has used indicator data to supplement other sources of information, whenever relevant. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 Question 1: What have been the SPICE activity’s key successes, challenges, and lessons learned? Responses shall focus on the following aspects of civil society support: (a) policy advocacy and reform; (b) citizen awareness and mobilization; (c) human rights prevention and response mechanisms, documentation and analysis; (d) enhancing civil society space; and (e) coalition-building and networking. The scope of question considers only the SPICE grantees, not the DGP grantees. The evaluation has focused on SPICE grant funding support to 67 CSOs, given that capacity development is covered in the subsequent question. FINDINGS A. Policy advocacy and reform The space for civil society influence on public policy prior to 2015 was extremely limited by the repressive environment and generally tense relations between government and CSOs, on both national and regional levels; that fact was underlined repeatedly by informants and documents. However, informants consistently observed that in 2013-2014 Sri Lankan CSOs were able to maintain a certain level of political pressure on the government, including through monitoring and reporting on fundamental rights issues. SPICE provided financial, technical and moral support to some prominent CSOs speaking out publicly on such issues during that period, according to USAID staff and grantees. In fact, all those interviewed by the Team agreed that civil society made a major contribution to raising public awareness of the political situation during that period. Following the transition, many SPICE grantees have engaged actively in various nationwide dialogues and reform processes, and also mobilized citizen participation in some cases. Success has been observed in some areas: several SPICE grantees engaged actively alongside others in advocacy for the 19th amendment to the Constitution passed in late 2015 (which imposed limits on presidential terms and powers, among other reforms); grantees were also active in pushing for adoption of the Right to Information Act. On the other hand, grantees’ advocacy in relation to certain other core national issues has met with less success, according to some informants: much land is still occupied by the military in spite of government policy to return it; the recent Consultation Task Force on Reconciliation Methods (CTF) report recommendations were largely not accepted by the government; and major decisions and actions by the government are still pending on how transitional justice will go forward in spite of considerable international pressure. It should be noted that positive steps in these areas—such as the fact that the CTF consultations took place and that FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 6 legislation to establish the Office of Missing Persons was passed—were also highlighted by informants. Nevertheless, focus group discussions with grantee beneficiaries in the north and east indicated that feelings of frustration and helplessness are becoming more widespread as the expectations of the populace (especially in relation to transitional justice) are not met by the government. In addition to the more prominent national CSOs, SPICE also funded a wide range of “regional CSOs” as they are commonly called. These CSOs are based outside of Colombo, and generally focus their activities in one province or part of a province. Given the initial geographic focus of SPICE, most are in northern or eastern Sri Lanka. These CSOs took significant policy-related actions at more local levels, as indicated by their reports and corroborated through interviews. Often these initiatives worked with local authorities to tackle problems, rather than using confrontational advocacy methods. Examples include two organizations serving disabled people achieving better access for disabled at public buildings, and two other CSOs fostering the formation of multi-stakeholder committees to address gender-based violence. Furthermore, some regional grantees contributed at local levels to the above-described national advocacy initiatives, for example by making submissions and mobilizing citizens to make inputs to the Constitutional Reform Committee and the CTF at zonal level. Several regional CSOs in the east told the Team how they had managed, primarily due to SPICE financial and technical support as well as linkages fostered by SPICE with other CSOs and decision-makers, to not only get responses from local authorities but also to elevate those issues to national level. Those issues run the gamut from women’s rights to preschool education policy and claims for return of lands to their rightful owners. Overall, SPICE reports that as of December 2016, it had worked with 241 “organizations representing marginalized constituencies trying to affect government policy or conducting government oversight” and 197 “CSOs engaged in advocacy interventions,” one of its PMP indicators. On the purpose level, the PMP tracked the percentage of “target civic groups that had an impact on development or governance decisions.” 5 This only considered the 67 SPICE grantees. While the target was 75 percent throughout implementation, achievement each year hovered close to 50 percent, with the latest data indicating 51 percent. The SPICE team anticipates the final percentage may be higher, once all grantee data has been analyzed. While the context since 2015 is much more conducive to advocacy and policy reform, significant challenges continued during the past two years. Informants mentioned that CSO leaders who now have closer ties with the government (through an appointed position, consulting work, etc.) can have some positive influence, but some fear that certain former activists are becoming too close to the government and thus losing their ability to criticize when needed. B. Citizen awareness and mobilization SPICE has supported substantial efforts by grantees to inform and mobilize citizens, across diverse topics. Examples include fostering groups of volunteer human rights defenders; mobilizing people to demand for their basic needs to be met; raising awareness of gender-based violence and training village women’s groups to respond to incidents; outreach to youth through sports clubs; launching a preschool teachers’ association to claim allowances for the first time; and helping plantation workers to learn and seek to secure their rights. One strategy often mentioned by informants was to organize or strengthen groups at community or division level to discuss and try to address priority needs, as seen in the examples in the previous paragraph. The grantees working with these groups are generally optimistic that they can continue to function after SPICE, although several admitted that without continued grant funding it may not be possible to keep the groups as active as before. Following the political transition, SPICE encouraged and funded at least 22 grantees in its extension period to focus on activities related to transitional justice. Those CSOs worked on awareness and mobilization of people around various processes and issues (mentioned in the previous section on Advocacy), including psychosocial support for affected individuals to facilitate their engagement. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 7 The SPICE PMP results as of December 2016 show that 89 percent of civic associations targeted by the project (direct grantees and those indirectly supported through grantees) had been “active in community development and governance.” This meant that 309 out of 349 supported organizations had met these criteria to SPICE’s satisfaction. By contrast, only 39 percent qualified under this indicator in FY2014 (due to the time lag between reporting and verification processes, according to SPICE). The PMP also captures the number of SPICE-supported “community projects completed with active citizen participation” (meaning that citizens and community groups must be involved in planning and implementing the initiative), reported as 214 as of the end of 2016. Challenges faced by SPICE and its grantees prior to 2015 were daunting; CSOs found it difficult to organize public meetings on any subject, especially in the north, where all such activities were tightly controlled by security forces. Prohibitions announced in 2014 on press releases and press conferences further limited the ability of CSOs to do public outreach and mobilization of any kind.6 While the situation has eased considerably for most of the country, grantees in the north and east continue to report impromptu visits and other forms of intimidation by security forces. CSOs in the north and east said that they are required to target beneficiaries identified by government with their activities. The selection can be politically influenced, based on outdated information, and not necessarily target the persons most in need, but as indicated by one grantee, contesting the selection could lead to significant delays. Finally, several CSOs working on transitional justice reported that many affected people in the north and east are increasingly reluctant to tell their stories; they feel hopeless and marginalized, and are tired of promises and commissions that never result in change. C. Human rights - prevention, response, documentation, analysis SPICE supported grantees to work on human rights issues from a variety of angles, ranging from disappearances and political prisoners to rights of sexual minorities and land rights (among many other issues), from national to local levels. The project reports that 56 of its grantees were “critical Sri Lankan organizations advocating pluralism, human rights and democracy” that remained active. Notable achievements by these CSOs include the production of videos on torture and documentation of disappeared persons, successful court rulings and highly publicized meetings with top government officials on land rights, and human rights defenders who report that police are now afraid to abuse detainees for fear of being reported. On a very practical level of service delivery, SPICE grantees assisted thousands of people to defend their rights: as of December 2016, the project reports that 45,552 people had received legal aid or victims’ assistance, including 4091 reached by gender-based violence services and 37,608 who had received key legal documents, thanks to the work of SPICE-supported CSOs. In terms of information dissemination and training, nearly 80,000 people were reached by human rights awareness-raising activities, and 1490 human rights defenders received training. At the national level, SPICE grantees have produced research on a range of human rights issues and participated in CSO coalitions submitting reports for the Universal Periodic Review and other United Nations (UN) forums. One grantee has taken a leading role on torture issues, and worked with other CSOs to submit the 2nd Joint Alternative Report on torture; several other national and regional CSOs have made representations in relation to creation of the Office of Missing Persons, presenting data on missing individuals and pressing for solutions. In general, SPICE adopted a cross-cutting programmatic strategy that promoted increased understanding and use of rights-based approaches (RBA) by targeted CSOs; this was particularly important for organizations heavily focused on service delivery. SPICE provided extensive training and coaching to grantees not already well versed in RBA, and worked closely with prospective grantees to identify ways that RBA and advocacy for rights could be integrated in their proposed activities—either instead of or in addition to activities focused on delivery of services. Various grantees, including but not limited to regional CSOs, commented that SPICE had helped them to make (or at least start) a transition towards RBA and advocacy as core strategies for achieving their organizations’ objectives. However, both SPICE staff and some grantees FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 8 commented on the challenges inherent in this shift of approach, for both large and small organizations—not all were open to embracing RBA. Other challenges facing SPICE’s support to human rights work in Sri Lanka are largely the same as those outlined above. In particular, surveillance by government security forces was and is still a factor, especially in the north, which not only limits the scope of action of CSOs but also can intimidate victims of human rights violations from coming forward. For the CSOs, working with SPICE on sensitive subjects posed considerable risks. D. Coalition building and networking SPICE staff described an “organic approach” to networking, which involved the intentional facilitation of linkages (mostly informal) among organizations with common interests and encouraging ongoing contacts, as suitable opportunities were identified, but did not aggressively promote new networks or insist on formalization of ties among CSOs. Informants did not highlight this as an area where SPICE had made significant impact, but they did not identify it as a weak point of the project. Most grantees in fact felt that their existing networks were sufficient, although at least six interviewed CSOs credited SPICE with having intensified and to some degree expanded their links with other CSOs. Regional grantees were more likely to highlight the importance of this support than national organizations. However, in the words of one grantee at the national level, “SPICE underpinned existing relations between CSOs, including between regional and national level.” One notable example emerged from the project component implemented by MSI and IFES; those activities served to intensify the existing linkages between like-minded CSOs in various parts of the country on issues related to women’s rights, which was described as very useful by two regional grantees. SPICE reported that as of December 2016, 84 percent of SPICE grantees were “participating in at least one network,” while the baseline in FY2013 was 60 percent and in FY2014 the tally was only 45 percent. The CORE Values training program (described in Annex L) was often mentioned by grantees and other participating CSOs as a key SPICE contribution towards networking, especially for the links forged among second-tier managers of national CSOs and key staff of regional CSOs, who were the primary targets. They consistently reported that the intensive interaction over multiple workshops helped to forge bonds among participants, including across thematic and ethnic lines. In addition, SPICE funding and encouragement helped forge and/or strengthen links between CSOs at regional level with new or existing community-based groups, both formal and informal, for implementation of various activities. Several examples are cited above in relation to citizen mobilization. SPICE also funded work that helped to build bridges across ethnic/linguistic divides: one grantee in the east is bringing Muslims and Tamils together to push on land rights issues that affect both groups; and an education-focused grantee is improving language skills to enable preschool teachers and committees to interact better with different communities. One formal network received a SPICE grant and capacity building support. With SPICE support, that network reported that it was able to grow from its base of members in 16 districts, primarily in the south and center of the country, to establish new consortia in five new districts—three in the north and east, as well as two in plantation areas. SPICE also assisted the network to further its work on a Code of Conduct for CSOs, including consultations with 350 CSOs nationwide. The revised Code (yet to be rolled out) is expected to assist CSOs to conform to accepted standards of good governance and credibility, integrity and cooperation. The challenges to progress in coalition building and networking during the project period were multiple, especially prior to the political transition when CSOs were under threat and understandably reluctant to open up to new contacts who may not be trustworthy. On the other hand, it was reported that the high￾pressure situation in some ways served to unite certain CSOs—on the national level, and between national and regional. Ethnic, linguistic and religious differences continue to act as barriers to unity within civil society. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 9 Finally, some informants noted that the lack of strong cross-sectoral national (or regional) umbrella bodies made it more difficult for CSOs to feel and act in a unified fashion, even on issues that are a threat to the entire sector. E. Civil society space Although most of the foregoing sub-sections are also relevant to maintaining or expanding civil society space, a few additional words are justified, considering that expanded space for collective citizen action was the overarching purpose of SPICE. The Team found ample evidence among informants and project documents to indicate that the project had played an important role in this area, primarily before but also after the political transition. Those interviewed, including other donors and civil society experts, agreed that the mere fact of funding key organizations sufficiently to keep them active and engaged on national and local issues was a major SPICE contribution. Although other donors and actors also supported these and other CSOs, there was a trend towards donors leaving Sri Lanka entirely or being unwilling to fund more politically sensitive activities or organizations. That made the USAID decision to invest in SPICE even more important. The convergent views of various informants on this subject are illustrated well by the following direct quotes captured by the Team in interviews:  USAID staff: “SPICE helped CSOs to respond to the shrinking political space and continue to serve the population.”  SPICE staff: “We (and USAID) were realistic at the outset, we set expectations low. For the first two years, basically what we achieved was to sustain existing organizations, or ‘hold the line.’”  Grantee CSO: “SPICE was like rain in a drought as other donors were phasing out.” A specific SPICE intervention in relation to civil society space was the work on the legal and regulatory framework for CSOs. MSI as lead implementer collaborated with ICNL as technical experts, once opportunities emerged in 2015 to make inroads in this area. Together, they have moved steadily but cautiously in the evolving context to consult with CSOs and government on the difficulties posed by the current systems and structures and on options for reform. MSI has engaged Sri Lankan legal experts to study the situation in depth and feed into these processes, while ICNL has shared examples and experiences from other countries, and provided technical advice, including on the CSO Code of Conduct mentioned above. While certain key government officials have shown some openness to reform during exchanges with SPICE, progress has been slow and informants expect plenty of resistance to any moves to facilitate CSO functioning with less government control. As pointed out by ICNL and others, many “regulatory” issues facing CSOs do not in fact have a basis in law, but rather are extra-legal or irregular measures imposed by security forces and other authorities to control and/or hamper civil society—especially in the north and east. The restrictive legal framework affects CSOs unevenly, and informants noted that a “critical mass” of committed CSOs would be needed to achieve genuine reform. F. General findings on grant-making In addition to the foregoing successes and challenges in defined thematic areas, important findings emerged in relation to the design and management of the grants component of SPICE. With respect to grant design and selection, SPICE began by selecting 19 CSOs through non-competitive processes, to expedite project startup and keep a low profile for the project. SPICE designers felt that some key organizations were in danger of closing or significantly scaling back their work and needed immediate funds to remain operational, so both core and activity funding were provided. SPICE soon began to broaden its reach to offer grants to other organizations through competitive processes, starting with an Annual Program Statement (APS) in October 2013, followed by six Requests for Applications (RFA) between 2014 and December 2015. SPICE staff said that they agreed with USAID to disseminate the initial APS and RFAs in 2013 and 2014 primarily via mailing lists of CSOs and those working with CSOs. Informants did not mention this as a constraint on SPICE’s reach or results. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 10 The solicitation documents were broadly worded to maintain flexibility in the selection process and avoid flagging politically sensitive issues, and in most cases SPICE required only concept papers at the first stage. SPICE staff described this as a strategy for reducing the initial time and effort required of CSOs, but more importantly as a way for the SPICE team to have considerable input on the focus and approach of grant projects, through an intensive collaborative process of development of full proposals. SPICE staff indicated that the quality of proposals and concept papers was often relatively low, making their support necessary to ensure well-structured projects were funded. Only one interviewed grantee mentioned that SPICE was too pushy with its own direction during this process; others expressed appreciation for inputs from the project team. “- The project purpose included an emphasis on conflict-affected regions, and on that basis, SPICE’s original mandate from USAID was to focus on civil society in the north and east of the country, while also working with strategically important national CSOs. Although some national level grantees were active in the south and center of the country, the ability of SPICE to work in those areas was very limited during the first two years. Informants highlighted several resulting challenges: the government was dissatisfied with the failure to target Sinhala-dominated areas; activities aiming at reconciliation were less effective since they could not include the south/central regions; and the operating environment for CSOs and SPICE in the north was extremely difficult due to tight controls by security forces and other authorities (among other factors). A policy shift by the U.S. Government in early 2014 opened the doors to so-called “people to people” initiatives and allowed slightly more room for SPICE to work directly in the south and Sinhalese majority areas, and the political transition in 2015 enabled even more targeting in those areas. However, the majority of SPICE grantees were based in the north, east and Colombo. USAID and MSI agreed that SPICE would use Fixed Obligation Grants (FOG, now known as Fixed Amount Awards or FAA) for the majority of awards; of 117 total grants by SPICE, 100 were FOG or FAAs. USAID’s “help document” on FOGs/FAAs for non-governmental organizations provides detail about how this grants mechanism works. 7 The document indicates as advantages that FOGs/FAAs focus on outputs and results, limit risk for both parties, and require only limited financial and management capacity. It also highlights that USAID does not need to verify that the recipient incurred the costs as estimated in making the award, and that less experienced grantees have the “opportunity to strengthen and improve their internal procedures, systems, and policies” through implementation of the FOG. While various reasons were mentioned by informants at USAID and MSI, it appears that FOGs were favored primarily to mitigate the risk of working with less experienced grantees while building capacity of those grantees. The remaining non-FOG or “simplified grants” were based on the more traditional mechanism of periodic Reimbursements and liquidation of all expenses in line with grant budgets, without the use of milestones. Informants of the evaluation had a wide range of experiences and opinions of the FOG mechanism. Almost every grantee and several SPICE staff expressed misgivings about the FOG, especially the heavy up-front work involved in finalizing the grant milestones and budget; the difficulty of identifying and reaching pre￾determined milestones in an unpredictable context; the time-consuming process of gathering quotations, curriculum vitae, salary records, etc.; and lack of available funds to cover shortfalls in estimates or delays in advances. In a few cases, grantee management mentioned the lack of advances for expenses and lack of allowance for overhead costs as a hindrance. SPICE staff concurred that the time needed to prepare milestones and budgets for grant agreements was substantial, and that “nobody knew what they were doing” at the start in terms of milestone design. As one outcome of these difficulties, almost all grants had to be modified at least once, and many had three or more modifications. This was time-consuming for all concerned, especially as most milestones had several deliverables that had to be completed for a milestone to be met.8 On the plus side, grantees noted that SPICE staff was understanding of their problems, helped grantees to get modifications whenever needed, and were better able to guide grantees on suitable milestones as everyone learned the process. Some grantees came to appreciate the FOG over time, citing improved time management and planning skills, and FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 11 the reduced burden of financial reporting. Only one grant was terminated early by SPICE, after irregularities were observed in financial reporting. Most grants were one year long, though some were considerably shorter. Diverse informants from grantees to donors mentioned the short duration as a limitation on effectiveness, presenting challenges to completion of planned activities and achievement of results, while also putting considerable stress on grantee staff at times. SPICE staff explained the decision to keep grants short on the basis of the uncertainty of the project context at the outset and the limited absorptive capacity of targeted CSOs, although that did not apply to more established national CSOs. The idea was to award follow-on funding to CSOs that proved effective in the first rounds, and indeed 35 of 67 grantees did receive more than one grant. However, at least six interviewed grantees mentioned that time lapses between grants were problematic in terms of retaining staff and continuity of activities. With few exceptions, interviewed grantees indicated that SPICE grant reporting was overly burdensome, especially given demands on their staff to deal with government reporting, the fluid environment, and other projects. One grantee described it as: “Time consuming and labor demanding reporting requirements, which divert us from the focus of the project.” In the FOG-based grant agreements reviewed by the team, many milestones were time-bound and most fell only one or two months apart, which necessitated frequent milestone reports. Moreover, SPICE was required as part of the LWA mechanism to contribute detailed data on every major grant activity to the Knowledge Management Portal (KMP) maintained by Counterpart International. This meant that grantees had to submit data on a multitude of events and activities with very tight deadlines. One of them remarked that “For KMP there were 86 activities. It is another layer on top of progress and financial reporting.” SPICE staff indicated that the KMP data was not useful for any project management purposes, referring to it as a “vacuum cleaner of numbers.” Finally, it should be underlined that SPICE grantmaking placed a major emphasis on gender, which was reinforced through capacity development activities, including CORE Values training. As one SPICE staff described it, “gender was more than cross-cutting, it was a core issue.” The informant cited that 20,000 out of 37,000 beneficiaries of grant activities were women, and that 36 percent of grants were in some way related to gender (43 grants). Those grants were implemented by 27 organizations, including 17 classified by SPICE as women’s organizations. Another 31 organizations integrated gender concerns in the project design and implementation in their respective focus areas, such as governance and reconciliation, transitional justice, livelihood support, legal aid and human rights. SPICE staff indicated that their grant making aimed to address gender in two primary ways: by ensuring that all grants integrated gender equality concepts, concerns and practices, from project planning through to reporting; and by assisting certain grantees to implement projects or activities specifically designed to protect and promote women rights. The latter approach included (among other strategies) formation and strengthening of District/Divisional level structures and networks (including civil society and government representatives) addressing GBV in the North and East. A new PMP indicator added in 2015 shows that SPICE has supported 34 forums “bringing together duty bearers and rights holders to address gender-based violence.” As well, SPICE worked to build the capacity of grantees to work more effectively on GBV issues, for example by providing guidelines to inform their work with GBV victims. Beyond GBV-focused activities, SPICE also worked with a range of grantees towards enhancing the rights of women more generally, LGBTI community members, female garment workers and female-headed households. With respect to beneficiaries of grant activities, SPICE PMP data indicates that women or girls were:  52 percent of those who received legal aid or victim’s assistance,  54 percent of those assisted to obtain government identification and other vital legal documents  63 percent of those targeted by human rights awareness raising Moreover, SPICE data reports that women’s organizations constituted 47 percent of project-supported FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 12 organizations representing marginalized constituencies trying to affect government policy or conducting government oversight. CONCLUSIONS SPICE made a substantial contribution, primarily through strategic grantmaking and related support, to the maintenance (at first) and later expansion of civil society space in Sri Lanka, during its four years of activity. SPICE made an important contribution by funding and otherwise facilitating the work of CSOs engaged in activism, advocacy and awareness-raising during 2013 and 2014. Subsequently, SPICE has continued to provide important support to CSOs and citizens seeking to engage in key national and local decision-making and consultation processes. SPICE made a substantive impact in all five thematic areas described above, which is especially impressive given the constraints and risks of the early years. The most notable gains were reported in relation to human rights and citizen awareness and mobilization, although grantees contributed to significant achievements in advocacy, as highlighted in the preceding paragraph. However, the PMP suggests that many CSOs (at least half of the 67 grantees) faced an uphill battle in achieving verifiable impact on decision making by government bodies. Although the grants were highly disparate in nature—ranging from psychosocial counselling to preschool education, and from gender-based violence to voter education— SPICE promoted a cross-cutting theme of rights, and also helped coalesce CSOs around certain thematic areas that emerged over time, such as legal aid for human rights cases, support to people with disabilities and systems for dealing with gender-based violence. Support to CSOs working at the grassroots level with war-affected communities and some of the most vulnerable populations in the country was able to start to heal the deep wounds of war and give people hope for the future. Mobilization of groups at community level around issues of concern can be an important stepping stone towards empowerment. However, it is clear that the road towards reconciliation and resolution of the many grievances will be long and strewn with obstacles; CSOs can play an important role in ensuring fair outcomes and restoring national unity in a context where government is struggling to maintain public confidence. CSOs are working together in small clusters on many issues, but trust takes time to build and solid networks need time to develop; civil society’s freedom of speech and action continues to be hampered by negative attitudes and restrictions from various levels of government. In terms of civil society space, a core SPICE achievement was simply keeping key organizations active on national and local issues during a very dark period for civil society in Sri Lanka. As noted by one informant, “SPICE was an intervention in an area that few others would touch” at the time, and this earned huge appreciation from the civil society sector. On the other hand, there remains significant work to be done on the legal framework and operational control being exerted by government over civil society, which combine to limit its independence and sustainability, especially in the north. With respect to the grants process, SPICE was as inclusive as possible given the constraints, and gave opportunities to a diverse set of organizations based on rigorous selection and proposal development processes. The FOG mechanism was found to have certain advantages over simplified grants, especially as it reduced the time and effort involved for all parties when detailed accounting for every expense is required. However, it appears that the huge up-front effort to get FOGs successfully off the ground (not to mention the many subsequent modifications) may have negated those benefits for this project. Factors working against the FOG included the short duration of grants, unpredictable situation, inexperience with the milestone system, and the multiple other stressors affecting both the SPICE team and CSOs. Although in theory the FOG requires only “limited financial and management capacity” of the grantee (according to the USAID help document), in reality it appears that the process of budgeting accurately, justifying all budget amounts and reporting against complex milestones was very demanding for many grantees. A USAID informant summed up the experience by saying that “SPICE made it work,” but that was primarily due to the staff’s high level of commitment to the project, as they also faced a steep learning curve on FOGs. In retrospect, simplified grants would have been a better choice for many SPICE grantees, FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 13 allowing for faster startup and easier mid-course corrections by using a system that was more familiar to CSOs and project staff and less tied to highly specific and time-bound outputs. Although organizations with less financial management capacity and experience would have needed some assistance to comply with accounting and reporting requirements, the process would have served to build capacity in those areas over the entire period of implementation. A significant focus on gender was evident in SPICE’s grant making activity throughout the implementation period, both in terms of the grants and grantees that integrated gender considerations to some extent in their grant-funded work, and in terms of the proportion of women beneficiaries of grantee activities. Given the major challenges to gender equity in Sri Lanka, this emphasis was appropriate and highly valued by stakeholders. The increased engagement of government and CSOs in collaborative mechanisms to combat GBV (including replication in new areas) was a particularly important outcome. Finally, it is important to note that although this question focused on the grants activity of SPICE, the results outlined above have been supported in many cases by capacity development investments by the project, which are the focus of the next section. EVALUATION QUESTION 2 Question 2: What conclusions and recommendations can be drawn about the relative strengths, weaknesses, and effectiveness of the various tools and approaches of building professional and institutional capacity of local organizations? The three main approaches studied by the Team for this question are: USAID direct capacity development support to DGP grantees, SPICE support to DGP grantees, and SPICE support to 26 targeted SPICE grantees. The first two approaches are addressed here, while the third is covered under Question 2a below. The specific types of support provided by SPICE to DGP and its own grantees are listed in AnnexI, indicating how many CSOs were targeted in each area of capacity. The objectives and approaches of the DGP activity in Sri Lanka are described in AnnexF. A total of 13 DGP awards were implemented from 2012 to 2016, with a budget of 11.3 million U.S. Dollars.9 Initially, USAID staff took the lead on capacity development support to these grantees, who were receiving direct USAID funding for the first time. After approximately six months of implementation, SPICE was launched, and in early 2014 USAID amended its cooperative agreement to add a significant role for SPICE in capacity development of DGP grantees. At that point, USAID drafted so-called “tripartite action plans” for each grantee, allocating responsibilities for capacity development to USAID, SPICE and the grantee itself. In addition to specific support activities, USAID retained a management and oversight role of capacity development, including liaison with the grantees, while SPICE played a more technical role, largely consisting of verification of needs and priorities of grantees, contracting of service providers to help meet certain needs, and monitoring /quality control. USAID contracted Social Impact, Inc. to conduct a performance evaluation of the DGP in mid-2016. Many findings and conclusions of that evaluation did not distinguish between the effects of the capacity development support provided by USAID and by SPICE, which limited their utility in analyzing the pros and cons of each approach. Therefore, the findings and conclusions below are based on a combination of primary data collected by this evaluation and comparative data gleaned from the DGP evaluation. FINDINGS USAID support to DGP grantees The first step taken by USAID with DGP grantees was to conduct an in-depth Organizational Capacity Assessment (OCA).10 The grantees greatly appreciated the OCA process, especially its participatory nature and self-assessment approach. As noted by the DGPE, “12 of the 13 IPs 11 had never systematically analyzed their organizational capacity before. All IPs agreed that the seven OCA categories were a useful tool to guide action planning for capacity building.” The OCAs were done before award of grants, which several FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 14 informants said may have inhibited organizations from openly sharing weak points with their prospective donor. In fact, when SPICE staff met with grantees about nine months later, in most cases the grantees identified different priorities from those in the OCA. As one informant pointed out, however, doing the OCAs beforehand allowed USAID to include capacity development lines in some grantee budgets. The support provided by USAID staff was reported to have focused on financial management, internal administration and project monitoring and reporting.12 Interviewed and surveyed informants generally indicated that the quality and relevance of capacity support was high; seven of eight survey respondents from DGP grantees rated it as “highly relevant” while only one chose “somewhat relevant.” Four said the quality was “good,” four described it as “excellent.” Most grantees valued this assistance, especially to enable them to handle USAID requirements, which several described as rather overwhelming. As one grantee explained, “USAID went to great lengths to make sure we could handle the grant.” In terms of internal policies and procedures, the DGPE reported that all but one grantee lacked functional policies or administrative manuals at the outset. Each organization expressed that developing the policies and manuals was a lot of work, but that USAID gave them a strong head start by providing samples and feedback. However, all 13 reported that standard operational procedures were strengthened, and that USAID’s help with financial procedures helped enhance internal control systems and reporting. Survey results show that all DGP respondents thought USAID support had a positive effect on the capacity of their organization—five very positive, three somewhat positive. The key strengths of this support were:  Customized nature of support based on in-depth capacity assessment  USAID working hands-on with each grantee’s staff  Familiarity of USAID staff with relevant regulations, templates and tools  Funds for capacity development were included in budgets for some grantees; some informants said this made grantees more motivated to work on their own capacity development, beyond inputs provided by USAID or SPICE The main challenges were:  Support was largely confined to compliance-related capacities, including in some cases provision of templates to be adapted by grantees.  USAID staff had limited time and training for playing a capacity development role in addition to their other responsibilities, especially considering the geographic distribution of grantee organizations. As first time grantees, they needed a lot of handholding, which not all assigned staff were willing or able to provide. According to a senior SPICE staff, “USAID did not realize how much work it would be to have direct grantees.”  Per the DGPE: USAID staff played a dual role as decision-makers and advisors for grantees; they had to decide where to draw the line between assistance with capacity and action on poor performance, while grantees had to decide how much of their struggles to disclose.  Engagement by USAID staff and grantees declined over time. “Once SPICE was on board, the motivation of USAID and grantees went down,” according to one USAID staff.  USAID did not require grantees to report on either capacity development activities or results, which may have affected their level of attention or commitment to those activities. No records of the provided support were kept by USAID either, which made it challenging to assess this support beyond a general level (as noted by the DGPE). SPICE support to DGP grantees The initial intervention by SPICE was to meet with leaders of each grantee to review OCA results. Both grantees and SPICE staff reported this to be very useful, especially as it narrowed down priorities and clarified areas of weakness. As noted above, there is evidence that grantees were more open with SPICE FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 15 and its consultants than with USAID staff. However, project records indicate that for some DGP grantees, only one or two capacity development activities were conducted with SPICE support, while in other cases there were up to six interventions of various kinds. For at least two grantees, there appears to have been no outcomes at all, for example due to difficulties encountered in completing strategic planning processes. According to the DGPE, 12 grantees made progress in improving organizational sustainability. This included strengthening second-tier leaders, steps towards succession planning, and/or longer term planning for funding. Grantees that undertook strategic planning (the most common intervention by SPICE, targeting eight grantees) said it helped them to proactively focus rather than simply move from project to project. Seven grantees were able to improve governance by clearly defining roles and responsibilities. One grantee said they had moved from a “one person show to an organization,” thanks to this support. Twelve grantees established detailed performance monitoring systems. One grantee commented that “The most change for us was definitely in M&E. We realized the need for organizational level results-based management, not just project based.” In general, informants indicated that the quality and relevance of SPICE capacity support was high. Twelve respondents from DGP grantees responded to most survey questions on SPICE capacity development; nine rated SPICE support as highly relevant, two as somewhat relevant. Results on quality were more variable: six gave an excellent rating, three said it was good, while three said it was fair quality. Respondents were satisfied with how SPICE had affected capacity of their organization, with an even split (6-6) between very positively and somewhat positively. Asked if they had any major capacity needs that USAID and/or SPICE were so far unable to address, the respondents diverged, with roughly half saying yes, and the other half no. The key strengths of this support were:  Customized nature of support based on updated capacity assessment  Wide range of support made available  Quality of consultants engaged, and follow up/monitoring by SPICE  SPICE able to continue support even after some grants completed in 2016 The main challenges were:  Management of funding by USAID was far removed from the capacity development stream, so neither SPICE nor USAID had much leverage to pressure grantees to engage in capacity development.  The level of commitment by DGP grantee leaders was highly variable. According to one senior SPICE staff: “Some grantees from DGP really took the ownership of the capacity building and made a success. Others looked at it as a burden.” Various other informants agreed with this view.  The main difference in the SPICE approach with these grantees was the absence of Organizational Development Facilitators (ODF), whose function is described under Question 2a below and in AnnexJ. Several SPICE staff felt that this hampered the ability of SPICE to secure full buy-in by DGP grantees to capacity development.  The pool of available skilled consultants in Sri Lanka with requisite language skills in Sinhala and Tamil is limited, and had to be stretched by SPICE to meet the needs of not only its own grantees but also the DGP grantees. This was reported to have caused some delays in provision of capacity services.  Senior informants within USAID and SPICE expressed some concern that the addition of DGP responsibilities had distracted SPICE from its primary purpose, and the effort required to manage and execute the modified scope of work was a significant drain on the time of SPICE management and the limited pool of consultants. SPICE expected to be able to target 31 SPICE grantees with intensive capacity development under the project, but that number was reduced to 26, largely due to DGP. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 16 CONCLUSIONS This evaluation has confirmed a core conclusion of the DGPE: “To build the capacity of small and medium size direct grantees required a significant investment of time by USAID staff.” During a period of diminished activity in the mission due to political and financial constraints, providing intensive support to all grantees in key areas of compliance was feasible and by all accounts effective. Grantees and USAID staff agree that substantive increases in capacity were achieved due to that support, particularly in financial management and administration. Being direct grantees of USAID for the first time was grueling, but a beneficial learning experience, especially for those that embraced the process of capacity development. However, it soon became apparent to USAID that additional resources were required to offer more comprehensive and meaningful capacity development assistance to meet the diverse needs of DGP grantees, as the available USAID staff did not have all the required skills and experience. Although it might have been possible to engage Sri Lankan consultants to support USAID in this respect, the solution chosen was to contract with SPICE, taking advantage of its resources and expertise for capacity building of CSOs. That decision had several ramifications for both DGP and SPICE, both intended and unintended. The evidence of this evaluation and the DGPE suggests that SPICE’s support was highly valued and had positive effects on almost all DGP grantees in various areas of capacity. Strategic planning emerged as the most highly appreciated intervention. Nevertheless, there were notable limitations on what could be achieved. As pointed out by a SPICE team member, “Institutionalization of change needs hand-holding, but that was not possible with DGP.” That was primarily due to the other commitments of USAID and SPICE staff, and the lack of ODFs to provide more continuous support and monitoring. As well, commitment by DGP grantee leadership was variable, and their staff turnover posed challenges to sustainability of capacity changes. Although by all accounts, SPICE succeeded in balancing DGP support with its core activities, and of course had extra resources to do so, there was definitely added strain on key personnel of the project, and some reduction in support to SPICE grantees. The DGPE concluded that the three-pronged method of capacity development devised by USAID, which divided responsibilities among the grantees, USAID, and SPICE, “worked well” with all DGP grantees. However, the Team could not find evidence in the DGPE report nor elsewhere to support that statement. In fact, there was minimal evidence of capacity development action by USAID and the grantees, after SPICE came on board—although this may be partly due to lack of reporting.13 With respect to positive change in DGP grantees, the Team agrees with the observation of one USAID staff that “SPICE was the main factor.” Based on recommendations of the DGPE, USAID is planning to support closing OCAs with all DGP grantees, which may shed more light on areas of more and less improvement with both USAID and SPICE support, as well as changes based on the grantees’ own initiatives. Question 2a. How effective has SPICE’s capacity development approach been in increasing civil society capacity in the following areas? Vision/ governance, Management resources, Human resources, Financial resources, External resources, Coalition-building and networking The focus of this sub-question was the 26 CSOs selected by SPICE for intensive capacity development. At the outset of the project, the first 14 grantees were selected for this intensive treatment, and SPICE chose another 12 as more grants were made. When SPICE was extended in 2015, the budget contained relatively fewer resources for capacity development, which meant that only 12 of the original 26 could be targeted during the shorter second phase of capacity development.14 SPICE used a five-stage process for its intervention with these selected CSOs, as depicted in the figure below. They used a self-assessment tool called the Institutional Development Framework (IDF) rather than the OCA used by USAID, and customized it for the context and needs of SPICE grantees. Figure 1. SPICE capacity development cycle FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 17 The assessment led to individualized Organizational Improvement Plans (OIP), which described in detail the priority needs and planned actions for meeting them. The OIPs defined benchmarks to indicate completion of activities in each area. Benchmark reviews (BMR) were carried out by SPICE staff and consultants with each organization for each phase of capacity development work. As mentioned above, ODFs were a key aspect of the SPICE methodology; these part-time consultants provided ongoing coaching, monitoring and liaison for grantees. 15 Each ODF was allocated one to four CSOs, spending approximately two to three days per month with each. In the first phase, SPICE entered into subcontracts with several Sri Lankan consulting companies to provide up to two years of capacity development services (including ODF services); these contracts with “Capacity Building Providers” (CBPs) were not renewed when the project was extended. SPICE opted instead to engage ODFs and other consultants directly, and make more use of SPICE staff for capacity support services. FINDINGS The overall quality and relevance of capacity support to these grantees by SPICE were reported to be high. Nine respondents from SPICE grantees participated in the survey, although for most questions, eight responses were recorded. Six rated the support as highly relevant, and two as somewhat relevant. In terms of quality, seven considered it to be good, and one as excellent.16 Although some informants referred to the limited availability of organizational development experts in Sri Lanka, they generally agreed that SPICE had used the best consultants available and worked with them to fill any gaps. Survey respondents were generally satisfied with how SPICE had affected the capacity of their organizations, with five indicating very positively, and three somewhat positively. However, six of nine said that they had major capacity gaps that SPICE had not been able to address.17 In general, the key strengths of the intensive capacity development support were:  SPICE tailored its support to each organization’s needs and priorities.  SPICE relied heavily on coaching and mentoring, using an in-house approach that included multiple personnel and board members when relevant.  Use of ODFs was positively viewed by all interviewed grantees and SPICE staff. Informants said that ODFs had added significant value to the process, and were well matched to their organizations.  SPICE oversaw both grants and capacity development, but the frontline role played by CBPs and ODFs helped to ensure grantees would share problems openly, without fear of affecting the funding flow/donor relationship.  The IDF tool used to assess CSO capacity was comprehensive and generally viewed positively by grantees and consultants. However, adapting it to the local context was time-consuming.  The engagement of CBPs as a channel for delivery of services was seen positively by most informants. Several reported that working with SPICE had strengthened the expertise of CBPs in certain areas.  SPICE took an extremely hands-on approach to design and monitoring of capacity development, by all accounts. Grantees and ODFs appreciated the close follow-up by project staff; in almost every interview where a grantee mentioned a shortcoming in a capacity development activity, it emerged that SPICE had later taken action to ensure the need was met through supplementary support.  SPICE achieved a high level of gender balance in its organizational capacity development work overall: 51 percent of those trained in specific areas designed to improve institutional capacity were female, according to the PMP data (which includes personnel from both DGP and SPICE grantees). FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 18  Gender was included in the IDF tool, and five grantees requested and received capacity support to develop their gender policies. The SPICE team’s gender expert provided considerable direct assistance to grantees to analyze gender aspects of their operations and activities and SPICE trained five grantees on incorporating gender into their programming. This led several grantees to report major shifts in their integration of gender into programming.  The “bundling” of intensive capacity development with 26 SPICE grants was generally considered as suitable and effective by USAID, SPICE and external informants. This made it easier for SPICE to support less experienced organizations to meet grant implementation and reporting requirements. In the words of one grantee that received international donor funds for the first time: “SPICE gave us an opportunity to grow from a CBO or women’s group to a real organization.”  Given the tense relations between government and civil society, especially prior to 2015, several informants felt that having good internal systems could help CSOs to defend themselves against criticism or harsh regulatory action by the government. SPICE was useful because, in the words of one grantee, it “forced us to think about those aspects in addition to our usual focus on core business.” In addition, the CORE Values workshop series launched by SPICE for a wider audience of CSOs (grantees and others) was much appreciated by informants, who repeatedly highlighted the networking benefits of this initiative. Informants remarked that this training was an appropriate strategy to target mid-level managers, and that the multilingual translation was superb. Several observed that CORE Values “opened people’s eyes on the subject of transitional justice,” which was featured in one of the six core components.18 The main challenges to the effectiveness of SPICE capacity development for its grantees were:  The CBPs and ODFs were recognized experts in certain areas of organizational development when selected, but skill levels varied and weak areas emerged. SPICE had to devote significant time and effort in the early stages to ensure sufficient capacity across the board.  Management of up to five CBPs and 10-15 part-time ODFs was also tricky, according to SPICE staff, especially to ensure clear lines of reporting and feedback.  Due to their activist nature, some CSOs did not embrace the “professionalization” of their organizations that SPICE promoted, according to several informants. Buy-in appeared to be notably higher for SPICE than for DGP grantees.  Most CSOs in Sri Lanka are plagued by high staff turnover, which some grantees and project documents indicated as a significant constraint on the sustainability of capacity development effects.  Many grantees learned about Rights-Based Approaches (RBA) for the first time through SPICE, which adopted RBA as a key strategy for encouraging service delivery-focused CSOs to think about how democracy and governance issues were relevant to their work and beneficiaries. This was a difficult shift according to some DGP and SPICE grantees, especially for large organizations accustomed to needs-based programming; the leader of one commented that: “We found the turning point for rights based approaches, but we had difficulty finding the turn.”  Benchmark review reports were generated by SPICE staff after meeting with each targeted CSO at the end of each phase of capacity support.19 This allowed for close tracking of outputs, and progress has been generally positive with most grantees meeting benchmarks consistently. SPICE monitoring data from the PMP shows that for FY2016, 75 percent of grantees were “meeting improvement benchmarks” in their OIPs on time (at most one month after the agreed deadline for completion of those tasks). However, as observed by the Team and confirmed with several SPICE staff, the OIP/BMR tool is less than ideal for monitoring outcomes of capacity building, as it does not really examine change (or lack thereof) in the organization’s operations and effectiveness. As well, the Team noted variable approaches to the BMR; while some reports pointed out both shortcomings and achievements, in others the conclusions appeared to be excessively positive compared to feedback heard from those same CSOs in interviews. In discussions with SPICE staff, the nature of FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 19 the BMR was described as an “appreciation of the process” or “validation” exercise, and not necessarily evaluative. Full end line assessments of capacity were not done by SPICE. In terms of the six specific areas of SPICE capacity development intervention given specific attention by this evaluation, the Team found the following: 1. Governance/vision/strategic planning. CSO informants and project reports pointed to notable successes on governance structures and succession planning in five SPICE grantees (including significant revamping of structures and responsibilities). Strategic planning processes and resulting plans were highly valued by almost all informants; SPICE even helped some grantees to revisit the original plans after the political transition. This comment by one grantee sums up the views repeatedly heard by the Team: “Strategic planning was our first priority, it was very useful and helped to generate many new proposals, we were able to get new funding.” Generally, it was difficult for SPICE to work on board and second-tier management development where dominant founders or charismatic leaders are playing multiple roles in the organization—as is common for Sri Lankan CSOs. 2. Management resources (including internal systems and financial management). The effects of SPICE support were highlighted by many grantees, who noted that internal procedures had become stronger and more accountable. Several commented that management of their grants with the support of SPICE staff was useful in this respect, and at least five interviewed grantees indicated that implementing Fixed Obligation Grants (FOG) for the first time had strengthened internal procedures, by forcing them to plan activities and budget well. 3. Human resources. SPICE worked with most grantees to help revamp or fine-tune (as needed) their human resources management skills and procedures. In several cases, the project supported in￾depth skills audits and revisions of job descriptions that were highly valued by the grantees. Gender policies were developed or improved for five grantees that identified this as a priority. 4. Financial resources/Sustainability. SPICE training on proposal writing, which eventually reached almost all 26 grantees, was given high marks by participants. Several mentioned progress in creative fundraising and income generation options, thanks to SPICE coaching and analysis. One grantee observed: “We finally understood the difference between fundraising and income generation.” However, almost all grantees interviewed and surveyed highlighted financial sustainability as their most important current need, especially in view of declining donor interest in Sri Lanka and the low level of private sector support.20 5. External resources. This category focused on external communications of CSOs. Although SPICE supported some organizations substantively in this area—for example with new websites, social media training or equipment and training for video production—there were constraints due to the context before 2015, which may have led CSOs not to prioritize communications. 6. Coalition building and networking. This subject is more closely linked to the grants activity of SPICE, and therefore has been covered under Evaluation Question 1. Although not necessarily a strength or weakness, the Team observed that SPICE capacity assessments and subsequent training and mentoring of the 26 targeted CSOs paid little attention to development of technical capacity (i.e. implementation of the “core business” of those CSOs, such as how to best support livelihoods of displaced people, how to develop a library for the blind, etc.) as compared to organizational or internal management capacity. This was described by USAID and SPICE as a strategic decision based on various factors, including the very public criticism of CSOs for misuse of funds that has persisted after the tsunami relief effort, and the many demands by government authorities for detailed information on CSO activities and finances. A senior SPICE staff member indicated that the need for organizational development support was more acute, and that technical support formed an important part of the grants development and management process. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 20 CONCLUSIONS One capacity-building consultant described SPICE as a “unique experience,” citing the project’s systems for ensuring quality of services by subcontractors, and the relatively long intervention with each organization, which allowed for better absorption and follow-up of activities and outcomes—especially for grantees able to participate in the second phase. Although “unusual” is perhaps a better word choice than “unique,” the Team considers this an accurate assessment, based on its experience with civil society programming in Sri Lanka and other contexts. Both elements mentioned by the consultant were key factors in the results achieved in capacity development, to which the Team would add the tailoring of support to each organization and emphasis on in-house coaching, which helped to firmly internalize changes and sustain improvements. The methods used by SPICE came with certain challenges and costs, in order to ensure their effectiveness. ODFs proved to be an effective way of coaching and monitoring targeted CSOs, and providing a buffer between SPICE’s role as grantmaker and SPICE as provider of capacity support. However, SPICE had to make a major investment in training, coaching and monitoring of the coaches themselves, due to the scarcity of available skilled consultants with the required language skills, and the fact that ODFs were not accustomed to playing the role that SPICE had in mind. That investment by the SPICE team, including staff based in Washington DC and Sri Lanka, and the strategy of assigning ODFs in consultation with the CSOs themselves, were the most important factors in making the ODF system work. The “bundling” of intensive capacity development with grant funding was an effective strategy for a civil society support project, which enabled SPICE to shore up weak systems in well-established CSOs while also funding some relatively small and undeveloped organizations. While SPICE and its consultants provided regular careful follow-up of quality and outputs of capacity development activities, monitoring of outcomes and actual changes in organizational effectiveness proved more difficult using the project tools. Although objectivity is always challenging in a participatory process, carrying out an endline assessment of capacity would have been informative for purposes of seeing the bigger picture for each organization and for the project as a whole—i.e. which weak areas had been adequately addressed, which were still untouched, and which had been worked on but with insufficient results. Based on the information available, including the BMR reports and primary data collected in person from a sample of 14 these 26 CSOs, the Team found that comparatively greater advances were made in certain areas of governance (especially planning) and in management systems and procedures, including financial management. Less significant results were achieved in relation to external resources and financial resources, where SPICE and the grantees faced substantial obstacles due to the context and other factors beyond their control. Financial sustainability remains very problematic for almost all those interviewed, though developing or updating strategic plans combined with proposal writing support had helped multiple grantees to secure new donor funding. It should be noted that although this analysis has focused on the SPICE activities devoted to capacity development, the results described in this section have been supported in many cases by the grants design and management process, especially in terms of planning capacity and management skills. Question 2b: What are the remaining civil society gaps that could benefit from future support? The Team posed this question to almost all KII participants, and reviewed secondary data such as the 2015 CSO Sustainability Index for Asia (CSOSI)21. In addition, the Team considered the views of 25 CSO staff and leaders who responded to the survey. However, the level of attention to this sub-question was much less than would be possible in a civil society assessment. Therefore, only findings are presented for this section. FINDINGS The following is a brief description of the areas of civil society organizational capacity highlighted most often by primary and secondary sources. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 21 1. Financial sustainability Informants consistently mentioned access to funding as the uppermost need of Sri Lankan civil society. Among surveyed CSOs, fundraising and income generation was the most frequently selected “unmet capacity need” of their own organizations following SPICE and USAID support, and by far the most mentioned “key area of organizational capacity in which Sri Lankan civil society in general remains weak or undeveloped.” Figure 1 below summarizes the responses from DGP and SPICE grantees on weak areas of capacity, which varied little between the two categories of grantees. This was the single category of the 2015 CSOSI where the situation in Sri Lanka was in the range of “sustainability impeded,” and the worst of the seven areas analyzed by that index. That report cited the United Nations in Sri Lanka as saying that “donor funding to service-providing CSOs continued to decrease in 2015, mainly due to Sri Lanka’s transition to lower middle-income status in 2010.”22 On the other hand, the CSOSI indicated that donor funding for advocacy oriented CSOs increased in 2015 due to the strengthening of Sri Lanka’s relations with the international community and the government’s willingness to engage with international and domestic CSOs on rights-based issues. Many informants highlighted the need for CSOs to generate their own income, to reduce reliance on international donors that have been the mainstay of the sector for so long. Another option mentioned was seeking funds from the private sector; however, most informants took a dim view of the short-term prospects for Corporate Social Responsibility (CSR). The following quotes are illustrative:  “CSR is not feasible in Sri Lanka for anything related to DRG. This is why CSOs need ongoing international support.” (SPICE staff)  “Generally, philanthropy is very weak. Private sector – it was hopeless before, might be some hope now.” (Civil society expert)  “Sometimes it is possible to get in-kind donations, but they do not want to give funds.” (CSO leader) This is largely consistent with the State of Civil Society 2015 report, 23 which found that CSR appears to be gaining ground in Sri Lanka as a concept, but “corporate foundations are reluctant to support initiatives that are viewed as controversial, which in many contexts includes human rights work or anything that is perceived as a challenge to the status quo.”24 Figure 2. CSO survey responses identifying key areas of organizational capacity in which Sri Lankan civil society in general remains weak or undeveloped.25 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 22 2. Communications and image – with public, government, and media (including social media) The 2015 CSOSI reported that the public image of CSOs had improved in 2015 compared to 2014 when attacks in the media were frequent, and that linkages between CSOs and the media also improved in 2015. The report found that the government view of CSOs had also improved. However, the public continued to be suspicious of CSOs, especially in the South. It mentioned that only larger and more visible CSOs publish annual reports, which enhance their credibility with the public. Notwithstanding these apparently positive steps, most interviewed informants still consider this to be a weak area for most CSOs, where significant work is required— not only with CSOs but with the other sectors of society. Communications was among the most commonly identified needs by survey respondents, as seen in Figure 1. 3. Coalition building and networking Although the CSOSI reports that activity by informal coalitions was on the upswing in 2015 due to the changed political environment, it points out there are very few established CSO networks in Sri Lanka. Although advances have been made and SPICE grantees tended to be satisfied with their linkages (as highlighted under Evaluation Question 1), informants including donors and some CSOs (both interviewed and surveyed) mentioned this as an area that needs to be strengthened. Many felt that divisions among CSOs were worsening since the political transition, and seven survey respondents selected this as a capacity gap. A donor representative described the sector as very polarized, and in need of a national umbrella organization. 4. Understanding of the role of civil society Another theme that emerged from diverse informants was the challenge faced by civil society in fully grasping the complex political situation of Sri Lanka, and defining what role they should and could play—as individual organizations and collectively—in the new context. One SPICE consultant explained that “the context has changed dramatically, there is no need for opposition role, we need to engage more with authorities and official processes.” A CSO leader underlined the importance of civil society learning to “adjust and avoid backsliding. CSOs must prevent angels from becoming devils (referring to government leaders in the new regime).” 5. Governance FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 23 The need to strengthen internal governance was seldom mentioned by CSOs in interviews, but widely recognized by SPICE and donor interviewees as problematic. It also emerges as a key area of weakness in the survey, with nine respondents highlighting this issue (see Figure 1). The 2015 CSOSI stated: “Many CSOs are personality-driven and heavily reliant on the strategic vision of their founding members. Hence, frequently, an organization’s founding members make key decisions. Moreover, there is poor succession planning, making it challenging for CSOs to sustain their operations upon the departure of their founding members.” 6. Legal framework and government controls As described under Evaluation Question 1, the legal framework for civil society requires major reforms in order to foster (or at least not impede) the healthy development of the sector. Although mentioned by only one or two CSOs, several USAID, donor and SPICE staff highlighted this as an overarching issue affecting CSO capacity. 7. Other Other capacity gaps mentioned by multiple informants were information management (including archiving and information security), constituency building, and a variety of thematic areas that related more to societal needs that to specific needs of civil society. Monitoring and evaluation was also ranked as an important gap by survey respondents (nine selections), although it was seldom referred to in interviews. Interestingly, almost none of those interviewed mentioned capacity gaps in financial management and other internal management functions, perhaps because interview informants were largely drawn from people who had worked to build those capacities in conjunction with SPICE (either as providers or recipients). EVALUATION QUESTION 3 - BEST PRACTICES AND LESSONS LEARNED The Team has identified various successful practices and lessons learned from the SPICE experience, which may be relevant to future programming in Sri Lanka and beyond. The most important practices are described below. 1. The all-Sri Lankan in-country team with a wealth of relevant experience was a defining feature and distinct advantage for SPICE. This ensured that senior management were intimately aware of the civil society context and players, as well as the political and security situation. These factors, as well as the strong sense of commitment by the SPICE team, were critical to the project’s success. SPICE leveraged pre-existing relationships of staff (in Sri Lanka and Washington), and importantly, senior managers could speak in Tamil with people in the north and east. Although the Chief of Party and Deputy lacked experience with USAID programming, SPICE showed that any deficits of this kind can be compensated by other resources – in the local team and/or from head office of the implementer. 2. Keeping a low profile for the project, its team and grantees was essential, especially in the tense political environment in 2013 and 2014. SPICE and USAID adopted effective approaches, including: flexible application of branding requirements, using a well-respected international CSO focused on livelihoods to get a first foothold in the north, and generally proceeding with caution—which meant a slow “burn rate” of funding at the start, but was proven worthwhile. 3. The SPICE team and approach were very consultative and participatory with CSOs and consultants, who felt that the staff were their allies despite the heavy demands and monitoring. As one grantee succinctly described it, “SPICE was bossy but friendly, and gave good advice.” SPICE maintained a generous amount of flexibility—to modify grants, accommodate changing priorities in capacity development, and respond to new opportunities and threats. SPICE recognized that solutions had to be tailored to each situation, and avoided one-size-fits-all tools and approaches. 4. The wide diversity of grants awarded by SPICE was effective in several respects. It enabled SPICE to fund and build capacity of both: a) core partners already working with USAID on DRG issues, who FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 24 were at risk politically, and b) a significant number of CSOs focused on service delivery, aiming to raise their awareness and where appropriate, nudge them towards rights-based approaches and advocacy. Funding social service delivery by grantees to vulnerable populations helped those organizations to gain and maintain trust of targeted communities through concrete forms of assistance, which later enabled them to mobilize those people to engage in governance-related activities. The mixed portfolio also served a very practical purpose, as less controversial activities and results could be publicized more widely. 5. The Fixed Obligation Grant (now called Fixed Amount Award) or FOG was not the best grant mechanism for the highly unpredictable environment that existed in Sri Lanka in 2013 and 2014, and even thereafter. The mechanism imposed a steep learning curve for everyone from grantees to staff to consultants, and consumed much valuable time at the front end of grants. It proved extremely difficult for most grantees to predict outputs, and outcomes were even harder (which led SPICE to stop using outcome-based milestones). Attaching temporal deadlines to milestones added another layer of difficulty. A SPICE staff member summed up the challenge as follows: “You have to take the original idea for a project and craft it into milestones that respond to the design AND the indicators AND the cash flow – and still ensure the project is achievable in a technical sense. It is extremely challenging to get this right.” The SPICE team was in the end able to make the FOG work for most grantees, but only because they had both skills and numbers of staff to support CSOs through this process. 6. Rigorous monitoring of grants, including activities and spending, was effective in ensuring successful execution by relatively less experienced CSOs, and in allowing SPICE to minimize misuse of funds or early termination of grants. Rigorous monitoring was particularly important to promote accountability and protect CSOs from allegations of misconduct. The establishment of sub-offices in the regions facilitated this level of support and helped with tracking of the local context. 7. Comprehensive organizational capacity assessment at the outset of support to selected grantees was a valuable exercise; both the process and the information generated were very useful to grantees, regardless of the tool used. The benchmark review method was not sufficient to measure the effects on each organization’s capacity and functioning, given the focus on outputs. It would have been beneficial to repeat the assessment process with grantees at the end of funding, to analyze progress in more depth. The endline process would be less quantitative and more to gauge if the work on capacity is creating results leading to lasting change. 8. Close supervision of capacity development conducted by SPICE staff and ODFs included frequent interventions to develop and refine training approaches and tools, build capacity of trainers and other service providers, support implementation of new learning and procedures, identify remaining or new needs, etc. This practice enhanced the impact of capacity development investments, and generally the use of ODFs added value. However, a deep talent pool of strong ODFs is required to meet the needs of a large target group of CSOs; where not readily available, as in Sri Lanka, significant time and resources must be invested to develop capacity of the capacity providers themselves. 9. Continuity of support to CSOs was generally a positive aspect of SPICE, especially with respect to the intensive capacity development of 26 selected grantees, who benefited from an ongoing cycle of support. Overall, the duration of capacity development support was longer and more inclusive than is often the case. On the grants side, continuity was hampered by short grant durations and gaps between rounds of funding, which often meant a break of three or more months for grantees without resources to fall back on to continue activities. 10. Monitoring and reporting was taken very seriously by SPICE, as evidenced by the team’s intensive efforts at verification of grantee activities and results, multiple levels of reporting by grantees, and SPICE’s detailed quarterly progress reports. Close attention was paid to quantitative outputs, as well as the capture of many success stories to illustrate the changes taking place, which was doubly important in a situation where numerical PMP indicators and FOG milestones posed difficulties for measuring impact. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 25 While impressive in terms of reliability of data and detail, the sheer number of indicators, reports and related verification processes took up significant time of both staff and CSOs. Focusing on fewer strategically chosen indicators would have been just as informative and less labor-intensive. Having reviewed a portion of the voluminous monitoring data, the Team considers that some efforts would have been better spent focusing on outcomes and impacts—of both grants and the project as a whole—as analysis of higher level results is much less visible. For example, SPICE quarterly reports were very long and detailed (the last one was 70 pages, without executive summary or annexes), but did not discuss the overall progress towards outcomes and purpose of the project, nor critically examine the progress on indicators. 11. Networking was promoted by SPICE using an “organic approach,” as described under Evaluation Question 1, based on looking for new opportunities to foster synergy among CSOs, while supporting and strengthening existing linkages related to project objectives. This relatively hands-off and non￾directive strategy was valued by CSOs that were so engaged, and achieved some notable outcomes, even though SPICE had limited scope and resources to work on these linkages. As one grantee commented: “[Donor-funded projects] should not put together any coalition; that will fail and be attacked.” SPICE recognized that fact, and realized that networks and alliances of a wide variety can be effective; in fact, informal and/or temporary collaborations can be the best option in politically tense contexts. The CORE values training program appeared to have a positive impact on mutual understanding and interaction among staff of participating CSOs; further study of that intervention would be appropriate to identify the results in this area and how they could be sustained and replicated. 12. A certain amount of risk was involved in funding and implementing DRG-related programming in the Sri Lankan environment between 2012 and early 2015. Some donors were reacting to the increasing constraints by leaving the country, ceasing to fund DRG activities, or avoiding CSOs viewed negatively by the government. USAID decided to take another path with SPICE, in the hope that it would help sustain a core group of CSOs to speak out against escalating repression and authoritarian governance. This decision was consistent with a viewpoint captured in the State of Civil Society Report 2015: “A retreat into supporting safer work, as a way of sustaining civil society in difficult periods, can be a valid strategy, but it will not challenge government restrictions.” 26 The success of SPICE is a testament to the determination of USAID, the implementer team, and civil society to carry on in the face of significant risks. The supportive relationships and flexible approaches among the key personnel at USAID and MSI were critical to navigating the rough waters prior to 2015, and almost as important in charting a path during and after the political transition, which has introduced myriad new challenges. RECOMMENDATIONS Overall, the Team wishes to underline that, despite the political transition and subsequent progress towards a more open environment for civil society and citizen participation in Sri Lanka, and notwithstanding the positive effects achieved by SPICE and other donor investments during recent years, many obstacles remain to have a vibrant and effective civil society sector in the country. USAID and other supporters of democracy and governance in Sri Lanka should not shift their focus away, as CSOs are still vulnerable and much needs to be done to shore up their important role in a society recovering from decades of conflict and repression. 1. USAID should adopt a nationwide strategy for a potential future civil society support project, including all regions. Research should be undertaken (potentially a civil society assessment) to identify specific geographic areas of need and opportunity, not just for civil society but for the population in general. The plantation sector should be given special consideration in this regard. However, different regions FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 26 may need to be approached in distinct ways, considering religious and ethnic issues, post-conflict fallout, and the relationships between CSOs and the rest of society, including government. 2. USAID should continue to provide financial and technical support to improve the legal enabling environment for civil society through various complementary and coordinated channels. Adoption and implementation of the National Policy on Non-Governmental Organizations is of paramount importance. External advisors should be drawn upon as needed to shore up Sri Lankan expertise with examples and lessons learned from other countries, but local actors from civil society and government should play a leading role (as they have to date). These initiatives should consider not only formal laws and policies but also de facto controls applied to CSOs by authorities and security forces. 3. USAID should promote CSO synergies by working with civil society actors to preserve and expand demand-driven forums where CSOs can discuss relevant research, advocacy and watchdog initiatives and explore areas for collaboration. SPICE took steps in this direction by facilitating linkages and cooperation among CSOs with common interests, especially as shared thematic areas emerged from the grantmaking activity. This should become a more intentional strategy going forward, though USAID and the implementer should continue to play a minor supporting role that facilitates rather than drives the process. One possibility is to support the development of regional or national level advocacy groupings of CSOs and activists around key issues—such as land rights or transitional justice—and assist participants to come up with shared strategic plans (including advocacy plans) to tackle the problems they have prioritized, and to allocate roles among different actors depending on the strengths and resources of each player.. 4. USAID should prioritize financial and technical support to CSOs on strengthening their engagement with government, to achieve needed changes for their beneficiaries and society at large while maintaining their independence. Engagement strategies may run the gamut from collaboration and coordination, through to constructive critique and non-violent confrontation. This may include replication of certain modules of the CORE Values training, and capacity development in skills such as negotiation, research and policy analysis. In parallel, it is important to work on the “supply side” of the equation, raising awareness at all levels of government of the role of civil society and examples of successful government-CSO collaboration. 5. USAID should develop an exit strategy well in advance of any anticipated significant reduction of US government support to civil society in Sri Lanka; the long-term sustainability and independence of organizations and of the sector as a whole should be prioritized in all programming, including all strategic planning exercises. 6. In spite of overall advances in the economic sphere, many Sri Lankans continue to struggle daily to meet their most basic needs. As part of civil society support initiatives, USAID should continue to prioritize assistance with livelihoods of the most vulnerable, both because it is desperately needed, especially in the north and east, and because empowering people at the grassroots to assert their rights and engage in governance will not be possible while their basic needs remain unmet. In the words of one informant: “There is no point in DRG if there is no economic development.” Future program design should assess the experience of SPICE grants in which immediate livelihoods support was combined with rights-based awareness raising and work on longer term solutions, to see how those approaches can be enhanced and replicated. 7. Donor coordination in relation to programming focused on civil society (and DRG more broadly) should be intensified on the working level, now that the environment is more conducive to open sharing of information and strategies. USAID should seek to coordinate and ideally collaborate with other donors, international and otherwise, as much as possible in supporting activities and organizations of fundamental importance for democratic development of the country. 8. USAID should continue providing financial support via granting to CSOs, with the following provisos: FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 27  Continue to be flexible about the definition of CSO, and open to supporting various forms of organization, ideally including more membership-based organizations (to promote greater engagement of the populace in civil society) and networks  Aim at least 50 percent of grantmaking at CSOs based in the regions (while taking into account investments by other donors)  Fund national CSOs primarily to conduct research, analysis and high-level advocacy, or to lead joint initiatives that involve regional CSOs. More generally, encourage consortia or partnerships of CSOs to apply for grants, especially those that bring together national and regional CSOs and community￾based organizations  Continue to support the establishment and growth of grassroots organizations, including through mid-range regional CSOs as intermediaries, and consider the use of regional resource centers to provide information and advice to local organizations  Reduce reliance on the FOG mechanism, especially when grants are short and the situation and/or results unpredictable, and avoid over-burdening CSOs with reporting requirements  Allow longer periods of time to implement grants, regardless of mechanism and even if the funds are relatively small  Pay more attention to monitoring and evaluation of higher level outcomes of grant projects, going beyond short-term outputs/milestones, including periodic check-ins after grants are completed  Whenever possible, couple funding with support for developing strategies for sustainability – considering the survival of the organization itself as well as the mechanisms, groups and other effects it may be fostering or creating at community level 9. USAID should continue to invest in capacity development of the civil society sector as a complement to funding, including consideration of technical as well as management capacity. Both forms of capacity need to be rebuilt, and support is important at all levels from grassroots community organizing, to membership groups such as cooperatives and farmers’ associations, to district and national level CSOs and forums. 27To ensure sustainable effects, support should focus on strengthening systems and institutions over at least two years, and continue to target multiple personnel and mid-level managers. 10. USAID should prioritize CSO financial sustainability, including through renewed efforts to kindle philanthropy at both corporate and individual levels, stimulate social entrepreneurship and income generation initiatives, and promote volunteering, among others. Research to pinpoint the various challenges and opportunities in these areas would help to guide and inform future programming. However, it is important to have realistic expectations in the short to medium term with respect to sustainability in the Sri Lankan context, given the poor (though improving) relations with government, media and the private sector, and the limited culture of philanthropy. Some organizations may continue to need core funding for the foreseeable future, due to the nature of their activities. 11. To help lay the foundation for future programming, USAID (ideally in conjunction with other donors) should develop a long-term strategy for enhancement of the visibility and credibility of civil society as a crucial step towards long-term sustainability. This could include supporting efforts towards enhanced transparency and accountability by CSOs, including dissemination of information through annual reports, compliance with the new Right to Information Act, and adoption of the draft Code of Conduct for CSOs developed with SPICE support. 12. USAID should consider the following as potential thematic areas for future civil society and other programming, based on recommendations of informants of this evaluation:  Transitional justice and reconciliation initiatives, including psycho-social services  Land rights and resolution of land-related conflicts  Civic education, including role of civil society and new legislation such as the Right to Information Act FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 28  Ongoing investment in legal aid provision to most vulnerable, and development of sustainable legal aid systems  Youth unemployment and lack of opportunities, especially as a potential source of unrest  Gender equity, including women’s political participation, gender-based violence, LGBTI rights, and reform of discriminatory laws 1 This section is largely based on the Scope of Work for this evaluation, which in turn drew upon the original SPICE cooperative agreement. Additional details on SPICE outcomes and approaches are set out in AnnexG. 2 Throughout this document, when the word “interviewed” is used, it refers to informants that participated in KIIs. Those questioned via surveys or focus groups are indicated with different wording according to the method used. 3 Some of the same individuals and CSOs participated in key informant interviews. 4 When reviewing the survey data on questions related to USAID direct capacity development support, it emerged that three persons who responded to those questions were not from DGP grantees. This led the Team to conclude that they were mistakenly identifying SPICE support as direct USAID support, so to avoid skewing the results, the Team removed those three respondents from the data set for those particular questions. 5 Target civic groups are defined by the PMP as the SPICE grantees. The indicator is met if the civic group (a) believes their participation made a difference, (b) can document at least one concrete example of making a difference, and the SPICE team has verified/assessed the reported impact. The term “development or governance decisions” is not defined. 6 SPICE contracted an expert to study the operating environment for CSOs in early 2015, just prior to the presidential elections, which describes these obstacles in detail. The March 2015 report by Ambika Satkunanathan is entitled “We are afraid of them. They are afraid of us: A Study of the Context in which CSOs and CBOs in Sri Lanka function and its impact on Freedom of Expression, Association and Assembly.” 7 USAID guidance on FOGs was described in “Fixed Obligation Grants to Non-Governmental Organizations, an Additional Help document for ADS Chapter 303,” Partial Revision Dated 12/24/2014, now updated to refer to FAAs and available at https://www.usaid.gov/ads/policy/300/303saj 8 An illustrative example of the milestones and related deliverables contained in the FOG of a SPICE grantee is included in Annex H. 9 One additional grant was terminated early by USAID, and does not form part of this evaluation. 10 The current OCA tool is available at https://usaidlearninglab.org/library/organizational-capacity-assessment 11 The DGP evaluation referred to DGP grantees as implementing partners or “IPs,” which appears in some quotes. 12 Several grantees were also trained in use of geographic information systems (GIS), primarily due to the presence in the mission of an expert in that field. 13 Information on what the DGP grantees themselves had undertaken in terms of capacity development did not emerge clearly from the DGPE, and it was beyond the scope of this evaluation to explore that in any depth. 14 Although periodic training workshops and other events included other SPICE grantees (and sometimes other CSOs), those activities were not closely analyzed by this evaluation. SPICE data as of December 2016 shows that 90 CSOs had received some form of capacity building support (including SPICE grantees, DGP grantees and other CSOs), and 589 individuals had been trained in areas related to organizational capacity. 15 The ODF mechanism is described in detail in AnnexJ, a document prepared by SPICE as a guideline for its Capacity Building Partners and ODFs. 16 This was one of the few areas where responses diverged notably between SPICE and DGP grantees, with the latter more inclined to give an excellent rating. 17 These responses were answering the question “Does your organization have major capacity needs that USAID and SPICE were so far unable to address?” Only two options were given, “yes” or “no,” which may have inclined respondents to choose “yes” even if they were generally satisfied with the support received. 18 See Annex Kfor a brief outline of the CORE Values training program. 19 SPICE did BMRs with all 26 targeted grantees in early 2016 in preparation for the anticipated end of the project, after which SPICE offered a second phase of capacity development to 12 of those grantees. That phase also concluded with BMRs. 20 One component of the original SPICE design was to promote Corporate Social Responsibility (CSR), but after the Sri Lankan government withheld approval for that component, it was cancelled. 21 Available at https://www.usaid.gov/asia-civil-society 22 Op. cit. page 52 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – FINAL DRAFT REPORT 29 23 Available at http://www.civicus.org/images/StateOfCivilSocietyFullReport2015.pdf 24 Ibid, at page 356 25 Respondents could select up to five areas. 26 The report attributes this comment on page 141 to Adam Pickering of the Charities Aid Foundation and Ambika Satkunanathan, of the Neelan Tiruchelvam Trust in Sri Lanka. Full report is available at http://www.civicus.org/images/StateOfCivilSocietyFullReport2015.pdf 27 Op. cit. page 44 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT A-1 ANNEX A: STATEMENT OF WORK FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT A-2 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT A-3 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT A-4 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT A-5 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT A-6 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT A-7 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT A-8 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-1 ANNEX B: EVALUATION TEAM, DESIGN AND METHODOLOGY EVALUATION APPROACH In conducting this evaluation, DI used an approach it has refined in evaluating many projects related to civil society support and otherwise. DI’s evaluation design and approach reflects principles outlined in USAID’s 2011 Evaluation Policy. According to that document, “importance and relevance will be achieved by explicitly linking evaluation questions to specific future decisions to be made by USAID leadership, partner governments, and/or other key stakeholders.” By measuring project effectiveness, DI aims to provide USAID with objective information to enable evidence-based decision-making on future programming and inform strategic planning activities. The evaluation will generate evidence and learning related to the performance of SPICE in achieving sustainable results, with a focus on the specific evaluation questions posed by USAID. To conduct this evaluation, DI proposes a mixed-methods, multi-phase evaluation approach to address these questions as rigorously as possible. This combination of tools enables a robust understanding of SPICE activity’s strengths, weaknesses, and overall effectiveness; mitigates bias in any single methodology; and provides sophisticated and defensible analysis to inform actionable recommendations. A mixed-methods approach is important because each evaluation question requires a unique methodology for sound analysis. Question 1 requires identification, critical analysis, and utilization-focused summaries of SPICE activity’s processes for, and success in, achieving outputs and outcomes on a broad spectrum of systematic civil society support. Question 2 will require a structured comparative analysis of selected beneficiary organizations in order to draw accurate conclusions about the relative effectiveness of the major approaches to capacity development. It will also call for examination of both primary and secondary data that points towards the current priority needs of civil society. Question 3 requires synthesis-based identification of best practices and formulation of recommendations, which DI will apply to serve USAID/Sri Lanka’s specific learning needs. EVALUATION TEAM Melanie Reimer (Team Leader) is a civil society and evaluation expert with 20 years of experience working across the democracy and governance sector. Her numerous experiences in democracy and governance include previous evaluation and implementation work on civil society and organizational development programs. Ms. Reimer’s work has allowed her to work in Eurasia, Latin America, the Middle East, Africa, and Eastern Europe. She has worked specifically on civil society evaluations and projects in Morocco, Moldova, Ecuador, Serbia and Chad, and has experience serving in senior-level leadership roles on implementation programs and evaluation teams. During her career in international development, Ms. Reimer has developed a strong familiarity with USAID regulations and also become a versatile and effective manager. Ms. Reimer’s evaluation experience includes more than 10 previous evaluations of USAID-funded programs. She has served as Team Leader on at least 12 evaluations and her extensive background in USAID evaluations has allowed her to develop an in-depth understanding of USAID evaluation design and quantitative tools and approaches. Throughout her career, Ms. Reimer has developed performance monitoring plans; guided development of reporting systems, indicators, and logical frameworks; designed advocacy indices; and conducted and managed surveys and survey teams. She holds a law degree from the University of Manitoba. Shevon Gooneratne (Civil Society Specialist) is an accomplished attorney and civil society expert with more than 18 years of experience working with Sri Lankan civil society, marginalized communities, and victims of armed conflict. He was a training consultant for the Law and Society Trust, providing training to organization FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-2 staff on Sri Lankan land rights and land law. He served for 14 years as Chief Program Officer, Advisor, and CEO of the Sarvodaya Legal Services Movement, a leading Sri Lankan NGO working with grass roots communities across Sri Lanka. In this role, he was responsible for the management and implementation of a variety of projects, including an EU funded civil society capacity building program, an EU funded community empowerment program, and a SIDA funded grass roots legal empowerment program. He was responsible for establishing and supporting the sustainable development of village based community groups and legal communities, and establishing positive working relationships with Sri Lankan NGOs and government departments. He holds a Master’s degree in Human Rights and Democratization from the University of Colombo. He is a native Sinhala speaker, who is fluent in English. Verni Vijayarajah (Democracy, Human Rights and Governance Specialist) is a consultant with extensive experience in human rights, peacebuilding, and organizational capacity development programming in Sri Lanka and abroad. She has supported women’s groups in Afghanistan and the Tamil diaspora, human and children’s rights groups in Sri Lanka, and evaluated donor funded projects in Sri Lanka. She supported GIZ programming in Indonesia, conducting capacity building exercises with local NGOs involved in the FORCLIME program. She worked with the ILO in Sri Lanka supporting human rights and children’s rights amongst conflict affected and marginalized communities. In this process, she conducted a sectoral and situational assessment, formulated an intervention strategy, and conducted a series of capacity development trainings for local implementers. She participated in the evaluations of CARE International’s Community Action for Development Project and ILO’s Skills Development for Economic Empowerment program in Sri Lanka. She holds an MBA from the University of Sri Jayewardenepura. She is fluent in English and Tamil, and conversant in Sinhala. EVALUATION MATRIX DI’s approach is to apply specific data sources, methodology, and data analysis to each of the core Evaluation Questions, to ensure a rigorous and streamlined process. The Evaluation Matrix is an effective vehicle for capturing that information in a concise and logical manner, and it will serve as a key guiding document for the evaluation. The Matrix below reflects DI’s learning from the document review phase and discussions among the Team and with USAID/Sri Lanka on best approaches for the evaluation. The team has further elaborated and defined each Evaluation Question posed by USAID to indicate clearly how the Team will approach each question. Table 1. Evaluation Matrix Interpretation and approach to question Methods Key Data Sources Evaluation Question 1. What have been the SPICE activity’s key successes, challenges, and lessons learned? Responses shall focus on the following aspects of civil society support:  Human rights prevention and response mechanisms  Human rights documentation and analysis  Policy advocacy and reform  Citizen awareness and mobilization  Coalition-building and networking  Enhancing civil society space Focus on Outcomes 1 and 2. The first 2 aspects relate to Outcome 1, the next 3 are most relevant to Outcome 2, and the final aspect is closely linked to the overall purpose of SPICE. These aspects will be analyzed within the framework of those outcomes. Successes = notable positive changes in the six areas listed, which can be reasonably seen as linked to direct support to CSOs and/or other activities supported by SPICE Key informant interviews, group interviews (especially with beneficiaries), document review Documents: SPICE contract and modifications, SPICE M&E data, work plans and progress reports, SPICE-issued RFAs and ensuing grant agreements, MSC stories and other reports from SPICE grantees, SPICE mid-term review, DRG assessment and other FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-3 Interpretation and approach to question Methods Key Data Sources Challenges = internal (to project or CSOs) or external (political, economic, etc.) factors that impeded or delayed positive change in the six areas during the project period Lessons learned = SPICE strategies and methods that affected the level of success and/or mitigated the challenges outlined above Enhancing civil society space = increased role for CSOs and citizens in development, reconciliation and governance-related activities N.B. This question does not include consideration of activities related to DGP grantees. relevant context documents from pre and post-transition Informants: USAID AOR and GVP Officer, SPICE staff, grantees, beneficiaries of grantee activities, CSOs working in related field, CSO coalitions or networks, independent experts and donors in related sectors, or journalists Evaluation Question 2. USAID/Sri Lanka has provided technical assistance for civil society capacity development using a variety of approaches. This includes capacity development of local organizations through the SPICE project, Development Grants Program (DGP), and directly led capacity development of local organizations by USAID Mission Staff. Using the DGP evaluation already conducted by USAID and findings from this evaluation, what conclusions and recommendations can be drawn about the relative strengths, weaknesses, and effectiveness of these tools and approaches of building professional and institutional capacity of local organizations? Focus on Outcome 3. The three main approaches to be studied are: SPICE capacity building of 26 targeted SPICE grantees, SPICE capacity building of DGP grantees, and USAID direct capacity building of DGP grantees. Strengths = factors that contributed to the intervention’s positive effect on capacity of targeted CSOs Weaknesses = factors that limited the intervention’s positive effect on capacity of targeted CSOs, or had a negative effect Effectiveness = notable positive change for CSOs in management capacity, that can be reasonably seen as linked to SPICE or USAID support Key informant interviews, group interviews, document review, online survey with selected CSOs SPICE documents: contract and modifications, M&E data, work plans and progress reports, reports from grantees and capacity building subcontractors, mid-term review, Institutional Development Framework (as adapted), Organizational Improvement Plans (and related/ predecessor capacity assessment docs) and benchmark reviews. DGP documents: DGP mid-term evaluation report, grantee project descriptions, initial and any subsequent Organizational Capacity Assessments (OCA), capacity building plans and reports, USAID staff reports on capacity building support and results, and related documents. Other documents: Civil Society Organization Sustainability Index, other available indices and publications Informants: USAID AOR and other USAID staff supporting DGP grantees, SPICE staff, SPICE and DGP grantees, capacity building subcontractors, Organizational Development FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-4 Interpretation and approach to question Methods Key Data Sources Facilitators, independent experts and donors in related sectors Sub-question 2a. How effective has SPICE’s capacity development approach been in increasing civil society capacity in the following areas? • Vision/governance • Management resources • Human resources • Financial resources • External resources • Coalition-building and networking The analysis of this sub-question will flow from the previous question, and will focus on the effects of the SPICE activity (only) on management capacity of its 26 targeted CSOs, in the six listed areas. *Vision/governance = clarity of mission/vision, organizational structure, strategic plan, roles in decision￾making *Management resources = documented internal systems, plans and administrative tools for management of operations *Human resources = sufficiency of competent personnel to support operations and activities and human resources management systems *Financial resources = access to diversified funding sources, plans and skills for fundraising and income generation that would support sustainability of the organization and activities *External resources = tools/plans for communications and outreach, links with clearly defined constituency *Coalition-building and networking = number and strength of links with other civil society actors to coordinate and work towards shared objectives N.B. This sub-question does not include consideration of capacity development targeting DGP grantees. Key informant interviews, group interviews, document review SPICE documents: contract and modifications, M&E data, work plans and progress reports, reports from grantees and capacity building subcontractors, mid-term review, Organizational Improvement Plans (and related/ predecessor capacity assessment docs) and benchmark reviews Other documents: Civil Society Organization Sustainability Index, DRG assessment, other available indices and publications Informants: USAID AOR and other USAID staff, SPICE staff, SPICE grantees, Organizational Development Facilitators, capacity building subcontractors, independent experts and donors in related sectors Sub-question 2b. What are the remaining civil society capacity gaps that could benefit from future support? Civil society capacity gaps = key areas of organizational development in which Sri Lankan civil society continues to have weak capacity, and which could reasonably be ameliorated by future USAID funding or other support Future support = will be focused on the types of support that could reasonably be funded or implemented by USAID or another international donor Key informant interviews, group interviews, document review, online survey with selected CSOs Documents: DRG assessment, SPICE mid-term review, DGP mid￾term evaluation, USAID Civil Society Organization Sustainability Index, other available indices and studies (published or otherwise), CORE VALUES and group training documents Informants: USAID AOR and other USAID staff, SPICE staff, SPICE and DGP grantees, CORE VALUES and group training participants, FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-5 Interpretation and approach to question Methods Key Data Sources Organizational Development Facilitators, capacity building subcontractors, civil society forums, independent experts and donors in related sectors Evaluation Question 3. Based on the evaluation findings and the current context, what best practices and recommendations should be considered for incorporation into future civil society programming in Sri Lanka? Best practices = strategies and methods that are found to be effective in contributing to SPICE (and DGP) outcomes. These will primarily be drawn from the lessons learned under EQ1 and the programming strengths identified under EQ2. Recommendations will be drawn from the key conclusions under both EQ1 and EQ2, considering the current context. No additional data collection, analysis of existing data No additional sources, but analysis will be informed by recommendations from all informants above, and by best practices gleaned from responses by informants and document review FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-6 METHODOLOGY OVERVIEW DI conducted its evaluation in discrete phases. Each phase built on its predecessor(s) to streamline data collection, informant selection, and final analysis across the evaluation questions. DI employed specific tools in each stage. These included: a desk review of available documents that describe the context and various aspects of the Project; preliminary collection and analysis of beneficiary organization data from USAID and the implementing partner to guide selection of CSOs for in-depth analysis; in-person data collection in the field; analysis of the primary and secondary data; and preparation of the draft report. By using multiple evaluation methods and triangulating results (described in more detail below), the Team strengthened the validity of its findings and answered the evaluation questions specified in the Scope of Work. PHASE I: INITIAL DESK REVIEW AND PREPARATION FOR FIELD WORK DESK REVIEW The information from the document review enhanced the Team’s understanding of SPICE, including activities carried out thus far, along with the range of programming approaches, opportunities, and constraints. The review helped the Team to refine its evaluation design and work plan based on a more thorough understanding of the SPICE and DGP activities. The team used a standard document review approach to conduct the literature review, which employs content analysis to review and analyze relevant SPICE and USAID records. DI analyzed a variety of qualitative data related specifically to these activities, including M&E plans, quarterly reports, the SPICE mid-term review report, the DGP mid-term evaluation report, reports from the Counterpart Knowledge Management Portal, SPICE and DGP sub-grantee descriptions, SPICE sub-award descriptions, and the SPICE program description. DI also analyzed quantitative data, such as those included in the project’s Performance Management Plans, indicators, and results, and other datasets collected by USAID or the implementer. In addition, the Team reviewed a range of documents that shed light on the rapidly evolving context of the country and situation of civil society, such as the 2014 and 2015 USAID Civil Society Organizational Sustainability Indices for Asia. This canvassing exercise allowed DI to get a preliminary view of the variety of successes, challenges, and lessons learned that SPICE program beneficiaries experienced (Question 1). The document review will also enabled DI to gather initial information about the relative effectiveness of both SPICE and USAID Mission￾led capacity building efforts with various CSOs and about remaining capacity gaps (Question 2), and extract ideas about potential best practices that could lead to actionable recommendations (Question 3). REMOTE DATA COLLECTION DI supplemented desk review findings with key facts about SPICE and DGP beneficiary organizations and their grant activities. USAID/Sri Lanka and the SPICE implementer provided this data in the form of a factsheet to enable purposeful fieldwork and data-driven case selection. DI provided a spreadsheet for completion within the first two weeks of award, and received the data just prior to preparing this work plan. The factsheet contains essential information to inform the Team’s selection of CSOs for in-person visits (described in more detail below), including: organization names and office locations; locations targeted by grant activities; whether the organization received capacity development assistance via SPICE, direct efforts led by USAID staff, or some combination thereof; grant numbers, durations and sizes; and thematic area of activity. The factsheet also sought the mission’s views on the priority of considering certain CSOs, as another factor that will help guide the process of selection. Post-award meetings with USAID/Sri Lanka and the SPICE implementer helped to finalize this tool to ensure that the team captured all relevant dimensions. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-7 In addition, the Team conducting conference calls with USAID/Sri Lanka key staff; USAID/DCHA and USAID/ASIA staff identified by the mission; and MSI staff and Counterpart International staff in Washington. Through these calls, the team gained a solid understanding of the objectives of USAID for this evaluation, the precise meaning of the evaluation questions, the factors underlying design of the project and subsequent adjustments, and the relationships among the various key stakeholders of SPICE. SELECTION OF INFORMANTS DI developed a list of potential informants based on suggestions from USAID, desk research, and Team members’ local knowledge. The selection of informants considered factors including location, relationship to the project being evaluated, type of organization/position, and affiliation of informant (USAID, implementer, CSO, donor, expert, etc.). The Team aimed for a reasonable balance of all those factors among interview informants to ensure that the Team heard a wide range of voices. Due to political sensitivities, the Team was advised not to plan on conducting data collection directly with government officials or media representatives. Given the number of CSOs targeted by SPICE grants and capacity development and the time constraints of the evaluation, the Team paid special attention to designing a robust method of selecting informants related to that category of project beneficiary. Based on desk review and preliminary data analysis, DI selected certain SPICE CSO grantees as “cases”; those CSOs were the primary focus for in-person data collection among the SPICE grantees. Those cases were not only be prioritized for KIIs with CSO leaders/key staff, but they were also be the ones for which the Team specifically sought out beneficiaries and other stakeholders who have an interest in or knowledge of the work of that CSO under its grant(s). This enabled the team to compare and triangulate information from different sources about the grant-funded work and capacity of the selected CSOs. (Note that other SPICE grantees were not excluded from the evaluation, but the Team will not visit those grantees and their beneficiaries.) From the total population of 67 SPICE grantees, the team selected an estimated 22 CSOs as cases for study using a purposive or judgmental method of sampling. The selection was “data-driven” and purposeful in order to provide information necessary to answer each evaluation question with rigor. The Team considered CSOs in all regions where support was provided or activities carried out, and aimed to include as much diversity as possible in the sample. In the case of Colombo-based CSOs that implemented their grant projects in other provinces, the Team carried out interviews with the CSO both in Colombo and in the field location, in order to ensure that clear information was gathered to inform the evaluation questions. It was important to analyze the SPICE grantees in at least two categories: the 26 CSOs that were targeted with intensive capacity development support (and who can thus help to inform analysis of Evaluation Question 2 in its entirety) and the 41 CSOs that did not receive that support (although they may have participated in occasional group training activities organized by SPICE). On that basis, the Team proposed the following allocation of cases between those two categories, together with the estimated number of KIIs and FGDs related to those grantees. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-8 Table 3. Breakdown of SPICE CSO grantees Type of support received from SPICE Relevant Evaluation Questions Number of CSO cases to be selected Number of KII with CSO staff Number of KII/FGD with beneficiaries or other stakeholders A. Grant(s) and intensive capacity development 1, 2, 2a, 2b and 3 13 (of 26) 16 (allows for 2 KII/CSO in a few cases) 8 B. Grant(s) without intensive capacity development 1, 2b and 3 9 (of 41) 9 7 TOTAL 22 25 15 The team opted to dedicate slightly more data collection time to the so-called Category “A” grantees, because of their greater interaction with SPICE and their concomitant ability to make a greater contribution to the analysis of all evaluation questions and sub-questions, as indicated in the table. Based on this projection, the Team selected CSOs from these two categories, taking into account criteria already discussed with USAID/Sri Lanka. First of all, the Team weighed geographic factors, aiming for a diverse mix of: 1) CSOs based and primarily implementing in Colombo area (including national scope projects); 2) CSOs based in Colombo but implementing grants in other provinces; 3) CSOs based and implementing grants in the northern provinces (Northern, North-Central and North-Western); and 4) CSOs based and implementing grants in the eastern part of the country (especially Eastern and Uva provinces). One team member also briefly traveled the southern part of the country to gather primary data about grantee work in that area. Although it does not constitute complete coverage of the provinces where CSOs are located and grant activities were implemented, this sample enabled the Team to obtain a diverse representation among the grantees in the areas where the project has invested most heavily. After the potential CSO cases were defined geographically, the Team balanced a number of other factors in making its final selection, including: thematic focus of each CSO’s grant activity (ensuring coverage of CSOs working on a diversity of issues, including youth empowerment and gender-related issues), the number and total value of grants received, and whether the emphasis of the CSO is on service delivery or awareness raising (less sensitive or controversial activities), or on advocacy and related types of engagement with decision-makers. The Team also considered whether each organization was supported by SPICE prior to 2015 when space for civil society was more restricted, during/after 2015, or during both periods. With respect to the DGP grantees, of which there are 14 included in this evaluation (one is also a SPICE grantee), the Team proposed to carry out KIIs with approximately five of these CSOs. Those five KIIs were divided among various regions of the country, and included both organizations that received direct capacity development support by USAID staff, and those that did not. PHASE II: FIELD WORK AND DATA COLLECTION KEY INFORMANT INTERVIEWS The Team conducted 57 in-depth interviews with a wide range of informants. In consultation with USAID, the Team selected interviewees who could provide substantive feedback on SPICE and its effects and who reflected diverse views. Stakeholders interviewed included: USAID and U.S. Embassy officials; current and former SPICE staff; SPICE subcontractors; other donor organizations; experts in civil society and governance issues; leaders of CSOs not funded by SPICE; and representatives of selected CSO grantees. The final FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-9 category was by far the largest, accounting for 25 KIIs. In many cases, CSO leaders invited several staff or volunteers to participate in the interview, which meant that three or more persons often attended a single KII (though not all participated actively). Although this may have affected responses to some questions and required more time, the Team decided to respect the inclusive nature of these encounters and use this as an opportunity to canvass a variety of perspectives, going beyond the organization leadership. In all, 145 people participated in the KIIs: 81 males, 56 females, and eight unspecified. The Team conducted interviews in English, Tamil or Sinhala, using a semi-structured interview approach based on question guides designed to suit different categories of informants and respond to the core evaluation questions. A sample of those guides can be found in Annex C. In the event that the Team was not able to interview a particular constituency or key actor, it identified alternate sources as proxies who were able to provide similar information. Those proxies were selected from the potential informant list developed as part of DI’s work plan. These interviews were conducted in person whenever possible, although in a few cases the Team conducted remote telephone interviews. We used a Key Informant Interview (KII) Protocol or questionnaire to ensure that each interview investigated the most pertinent evaluation question or sub-question, regardless of which team member was leading the conversation. To address the differing experiences of the identified individuals, the DI team developed questionnaires that catered to different categories of informant, with certain questions being cross-cutting for all or most informants, and others being tailored to elicit specific information within the purview of that type of informant. Final data collection tools are included in Annex C; once finalized in English, all questionnaires were translated to Tamil and Sinhala. These tools were piloted during the first few days of interviews, after which the Team reviewed their effectiveness and make amendments as needed (to all language versions) to optimize the data collection process. DI’s team employed a semi-structured interview approach, allowing some degree of deviation from the protocol. This allowed the interlocutor to have a partial role in determining the path of the conversation, which facilitated a natural flow of conversation and allowed the flexibility to focus on particularly useful topics that emerged during the interview, through the use of follow up questions. Whenever possible, two Team members attended each interview, with one taking the lead in questioning and the other responsible for timekeeping and notetaking. In all cases, informants had the choice of being interviewed in English, Tamil, or Sinhala; DI engaged interpreters as needed to facilitate smooth communications, working closely with those individuals to ensure they had a nuanced understanding of the purpose of the interview and confidentiality guarantees. To contribute to the collection of honest viewpoints and reliable information, the Team assured informants of their anonymity. FOCUS GROUP DISCUSSIONS The Team conducted 19 FGDs to gather more in-depth qualitative information from selected categories of project stakeholder and verify data obtained through other methods. The main target population for these discussions was beneficiaries of or participants in SPICE-funded activities led by grantees (15 FGDs), with a focus on assessing their experience with the organizations and projects, including benefits and possible disadvantages of the activities. As well, the Team held FGDs with the “program team” of SPICE to probe the details of the grants component, with partner or member CSOs of grantee organizations, and with a sample of Organizational Development Facilitators (ODF) contracted to support the capacity development of SPICE grantees. The Team held FGDs in Tamil, Sinhala or English according to the preferences of informants, in the north, east and south of the country, as well as two in Colombo. In all, 175 people participated in FGDs: 101 males, 70 females and four unspecified. Questions were customized according to the specific engagement of the participants in the project. These facilitated discussions with 5-10 allowed for exchange and discussion of experiences and viewpoints. Whenever possible, all three Team members attended each FGD, with one taking the lead in questioning FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-10 and the others responsible for monitoring participation, supporting follow-up questions, timekeeping, and notetaking. The team used FGD questionnaires or guides containing open-ended questions designed to prompt detailed and candid conversations. Depending on the nature of the group and specific purpose of the discussion, questionnaires were designed to respond to specific evaluation questions that were of most relevance to that particular group. MINI-SURVEY The Team carried out a web-based survey consisting of 34 questions (mostly closed-ended), distributed among key staff of 13 DGP grantees and a sub-set of 10 SPICE grantees. The survey was conceived to supplement the information gathered through other methods, enriching the pool of data and allowing more grantees to have input into the evaluation. The Team selected the targeted CSOs and individuals in conjunction with USAID, considering three key criteria: the CSOs were well-established national level organizations that received intensive capacity development support from SPICE; respondents were senior staff of those organizations, fluent enough in English to respond to closed-ended questions; they had sufficiently advanced computer skills and connectivity to easily complete an online survey. The survey explored how the leaders and senior staff of these CSOs view the support received from USAID and/or SPICE to develop their organizational capacity, including assessments, training, coaching and technical assistance, with a specific focus on whether and how the capacity and work of those CSOs has been affected. As well, it sought the opinions of respondents on future civil society support priorities. The survey questions can be found in Annex C. The Team chose to use Survey Monkey™ to ensure a confidential, easily accessible, inexpensive, and rapid means of collecting and analyzing data. That tool was supplemented by telephone and email follow-ups, although it proved difficult to significantly boost response rates. The survey link was ultimately sent to 56 individuals identified by SPICE and USAID, representing 23 CSOs. Responses were obtained from 24 individuals for a 43 percent response rate, including partial responses. The respondents represented 17 different CSOs, including 10 DGP grantees. DI relied on in-house survey methodologists with experience in operationalizing evaluation questions into specialized survey instruments. The DI support team included the Survey Director, who carefully reviewed questions’ internal logic, how well they link to evaluation questions, and their contribution to the overall evaluation design. After data collection closed, DI’s Survey Director produced data cross-tabulations and visualizations that assisted the evaluation team in identifying any patterns and drawing conclusions from the data. PHASE III: DATA ANALYSIS AND REPORTS DI transcribed data from interviews and focus groups and analyzed this content rigorously. A key tool for this process was a Data Synthesis Table, which helped to ensure that all members of the evaluation team recorded their primary data in a way that is comparable across interlocutor, and was easy to review and code, as appropriate. Using a spreadsheet format, the key information emerging from each data collection event and informant was summarized and classified by the evaluation question or sub-question to which that piece of information was most relevant. This tool allowed for an ongoing “at-a-glance” view of the data that was coming in each day, how it varied from one informant to the next, and how that data fed into the evaluation questions. The team conducted two main types of data analysis with respect to the SPICE CSO grantees that were selected for interviews. We analyzed each case individually to understand the key outcomes and challenges related to SPICE support. We then conducted cross-case analysis, comparing and contrasting the results of CSOs that received capacity development support with those who did not, those that received significantly different amounts of funding, those that worked in different regions and different thematic areas, etc. Various methods helped the team arrive at conclusions and recommendations, including content and trend FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT B-11 analysis to identify themes emerging from survey data, field work, and document review; and response convergence/divergence analysis to understand similar or differing responses. The team integrated data from this variety of approaches and sources into coherent conclusions and recommendations by aggregating and synthesizing findings. Data triangulation further ensured the validity and reliability of evaluation findings. Quantitative and qualitative-based inferences from a variety of sources using a range of data-collection instruments provided the team the fullest possible understanding of program performance. Findings from one data type or source were either be corroborated by findings from other sources (thereby strengthening the reliability of the conclusions) or were qualified by divergent findings from other sources (thereby increasing the validity of conclusions). More specifically, data from survey respondents was used to verify the findings related to capacity development from document review and interviews. During the team’s final days in country, the team reviewed the data collected through all methods, drew out highlights from that data, developed preliminary conclusions and recommendations, and prepared an out￾brief presentation for USAID. At that meeting, the Team presented an overview of the evaluation methodology; summarized the major findings and conclusions on each evaluation question; and outlined some potential recommendations. This formed the basis for the analysis included in the Evaluation Report. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT C-1 ANNEX C: DATA COLLECTION TOOLS The following general interview guides were used by the evaluation team. Questions were tailored to particular types of informants/organizations, and to the expertise, responsibilities, and characteristics of the specific person being interviewed. The interview guides are followed by the CSO survey questions. SPICE GRANTEE QUESTIONNAIRE Questions to be adapted and prioritized as needed to the individual’s role in the organization and in the grant project. Introductory 1) What is your role in this organization? How long have you held this role? a. How did you participate in SPICE activities to support capacity development? b. What was your role in relation to the activities funded by the SPICE grant(s)? Grant-funded Activities (EQ1 focus) Interview must take into account that some grantees received 2 or 3 grants. 2) What were the most important results you were able to achieve with the grant(s) from SPICE? (probe as appropriate about human rights, citizen awareness and mobilization, advocacy, and networking) 3) What were the key challenges that you faced in managing and implementing the grant(s)? (probe about internal related to the CSO itself, grant procedures or SPICE management, and external such as government monitoring, limited public support, etc.) 4) What is your view of the methods used by the Project to monitor the grant activities and results? (probe re reporting requirements, whether their info needs as an organization were met, etc.) 5) What could have been done differently to increase the impact of the grant(s) on your organization and on your beneficiaries? Capacity Development (EQ2) These questions are only for the 26 CSOs that received intensive support from SPICE. As appropriate, additional questions will be posed with respect to CORE VALUES training. 6) How relevant was the capacity development support to the needs of your organization? (probe about how priority needs were identified, the way of delivering support, duration, etc.) 7) What is your view of the quality of the capacity development support provided (by SPICE staff or subcontractors)? 8) What (if any) effect did SPICE support have on the capacity of your organization? (Probe as needed for the six areas of priority: Vision/ governance, Management resources, Human resources, Financial resources, External resources, Coalition-building and networking) a. In which area of capacity did SPICE have the most impact? And the least? b. Does your organization have major capacity needs that SPICE was unable to address? What are those unmet needs? 9) How were you and SPICE able to measure changes in capacity over time? (probe on utility of the Organizational Improvement Plans and benchmarks for monitoring progress) a. Were these tools and review activities useful? 10) As a result of the capacity support from SPICE, was there some change in the way your organization works? (probe for internal operations and external activities/delivery of services/etc.) FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT C-2 a. Have you observed any change in the effectiveness of your organization? (probe for examples, note mid-term review comments) General about SPICE 11) What are the key achievements of the SPICE Project? (refer to different outcomes and components, as needed, and probe for specific evidence of change) 12) What were the main challenges for the Project? (probe for internal, external, etc.) a. How did they affect the Project? b. How were those challenges handled by the SPICE team (and others involved)? 13) Was the Project appropriate for the situation in Sri Lanka at the time it was designed in 2012? a. How has the Project changed over time to respond to changing conditions? 14) What effect (if any) did the Project have on civil society linkages and networks? a. If linkages were enhanced, what results have flowed from that change? (Probe for specific examples.) b. If not, what have been the challenges? Closing 15) Has the public perception of civil society changed since the Project began? If so, how and why? 16) What were the most successful approaches and methods used by SPICE, which could be replicated in future programming in Sri Lanka or elsewhere? a. Were there specific approaches that worked well in the relatively closed space, prior to the 2015 political transition? 17) What are the key areas of organizational capacity in which Sri Lankan civil society in general remains weak or undeveloped? How could those be addressed? 18) What should be done now to expand the space for civil society and citizen participation in Sri Lanka (by USAID and/or by other actors)? a. What should be the priorities for USAID in particular? What part of civil society/what type of CSOs should be the priority? What type of support is most needed? 19) What other recommendations do you have for support of civil society, considering the current situation? 20) Do you have any suggestions of other people with whom we should speak, to further explore the kinds of questions we talked about today? DGP GRANTEE QUESTIONNAIRE Questions to be adapted and prioritized as needed to the individual’s role in the organization and the source of capacity development services. Focus is on EQ 2 and 3. Introductory 1) What is your role in this organization? How long have you held this role? a. How did you participate in capacity development activities provided by SPICE and/or directly by USAID staff? During which period of time? Capacity Development (EQ2) A. USAID direct support 2) How relevant was USAID direct capacity development support to the needs of your organization? (probe about how priority needs were identified (OCA), the way of delivering support, duration, etc.) 3) What is your view of the quality of that support? 4) What (if any) effect has that direct support had on the capacity of your organization? FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT C-3 a. In which area of capacity do you see the most impact? And the least? 5) How were you and USAID able to measure changes in capacity over time? a. Are the tools used by USAID to assess and monitor capacity useful? 6) As a result of the direct support from USAID, was there some change in the way your organization works? (probe for internal operations and external activities/delivery of services/etc.) a. Have you observed any change in effectiveness of your organization, which could be related to the USAID support? B. SPICE support (2014 onwards) As appropriate, additional questions will be posed with respect to CORE VALUES training. 7) How relevant has SPICE intensive capacity development support been to the needs of your organization? (probe about how priority needs were identified, the way of delivering support, duration, etc.) 8) What is your view of the quality of that support? 9) What (if any) effect has SPICE support had on the capacity of your organization? a. In which area of capacity do you see the most impact? And the least? 10) How have you and SPICE assessed changes in capacity over time? a. Are the tools used by SPICE to assess and monitor capacity useful? 11) As a result of the support from SPICE, has there been some change in the way your organization works? (probe for internal operations and external activities/delivery of services/etc.) a. Have you observed any change in effectiveness of your organization, which could be related to the SPICE support? C. Other 12) Does your organization have major capacity needs that USAID and SPICE were so far unable to address? (probe for types of support that USAID and SPICE could not provide) What are those unmet needs? 13) Do you think that having a direct grant of funds from USAID had an effect (positive or negative) on your organization’s capacity development? (explain that this is aside from the specific benefits of capacity development activities, looking at more general effect of being a direct grantee rather than a sub-grantee through SPICE) Why or why not? General/Closing (EQ3 and other) 14) Was the SPICE Project appropriate for the situation in Sri Lanka at the time it was designed in 2012? a. How has the SPICE Project changed over time to respond to changing conditions? 15) Have you observed that the SPICE Project had an effect on civil society linkages and networks? a. If yes, what results have flowed from that change? (Probe for specific examples.) 16) What have been the most successful approaches and methods used by SPICE, which could be replicated in future programming in Sri Lanka or elsewhere? a. Were there specific approaches that worked well in the relatively closed space, prior to 2015 political transition? 17) What are the key areas of organizational capacity in which Sri Lankan civil society in general remains weak or undeveloped? How could those be addressed? 18) What should be done now to expand the space for civil society and citizen participation in Sri Lanka (by USAID and/or by other actors)? a. What should be the priorities for USAID in particular? What part of civil society/what type of CSOs should be the priority? What type of support is most needed? FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT C-4 19) What other recommendations do you have for support of civil society, considering the current situation? 20) Do you have any suggestions of other people with whom we should speak, to further explore the kinds of questions we talked about today? MSI/SPICE STAFF QUESTIONNAIRE Questions to be adapted and prioritized as needed to the individual’s role in the project. The initial questions are focused on core SPICE programming, while later questions will address DGP elements. General 1) What has been your role in the Project? During what period of time? a. Were you involved in the grant-making aspects of the project, or in capacity development, or both? 2) What do you see as the key achievements of the Project? (refer to different outcomes and components, as needed, and probe for specific evidence of change) 3) What were the main challenges to achieving the objectives of the Project? (probe for internal, external, etc.) a. How did they affect the Project? b. How were those challenges handled by the SPICE team (and others involved)? 4) Was the scope and design of the Project appropriate for the situation in Sri Lanka at the time it was designed? a. How has the Project changed over time? b. How effective were those changes in responding to changing conditions? 5) What is your view of the methods used by the Project to monitor and measure the results of SPICE activities and the work of its grantees? What methods or tools were most useful? (probe on some issues raised in mid-term review, verification of grantee reports, etc.) a. Were there particular challenges in monitoring results? (probe regarding the use of subcontracted service providers, many sub-grants, multiple thematic and geographic areas, many indicators – some of them complex) SPICE Grantees – Grant-funded Activities (EQ1 focus) 6) What is your view of the grant-making activities of the Project? Were the grants awarded and managed in an effective way? 7) To what extent were SPICE grantees able to implement their activities and achieve their objectives? a. What were the challenges to grantees achieving their objectives? (probe for internal and external obstacles) b. How did results differ among the different types of SPICE grantees in this regard? Were some more successful than others? (by type of work, pre and post-transition, geographic area, size of organization, those targeted by capacity building and others, etc.) 8) What have been the most significant results of the grantee activities? a. Under which Outcome were results more notable? Under which Outcome were good results more widespread among grantees?  Outcome 1 – protect citizens’ rights, promote respect for human rights  Outcome 2 – mobilize citizens, advocate with authorities, promote reconciliation 9) What effect (if any) did the Project have on civil society linkages and networks? a. If linkages were enhanced, what results have flowed from that change? (Probe for specific examples.) FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT C-5 b. If not, what have been the challenges? SPICE Capacity Development (EQ2) Focus on the 26 CSOs that received intensive support 10) How did SPICE select CSO grantees for capacity development? Why was it decided to focus attention on this smaller group (rather than all grantees)? 11) How relevant was the capacity development support to the needs of the CSOs? (probe about how priority needs were identified, the way of delivering support, duration, etc.) 12) What is your view of the quality of the capacity development support provided (by SPICE staff or subcontractors)? Was it consistent, or was there lots of variation? 13) What effect did SPICE support have on the capacity of those CSOs? (Probe as needed for the six areas of priority: Vision/ governance, Management resources, Human resources, Financial resources, External resources, Coalition-building and networking) a. In which area of management capacity was SPICE able to have the most impact on capacity? Which types of CSOs were able to show the most improvement? 14) Was SPICE able to measure and clearly establish changes in capacity? (probe on utility of the Organizational Improvement Plans and benchmarks for monitoring progress) Were these tools effective in prioritizing and monitoring the capacity development activities and results? 15) Was it possible to observe the effects of improved capacity in the way that the CSOs operated and implemented their activities? How? (probe for examples, note mid-term review comments) Development Grants Program Focus on the DGP grantees that received capacity development from USAID staff or through SPICE. 16) How relevant was the SPICE capacity development support to the needs of DGP grantees? (probe about type of support, the way of delivering support, duration, etc.) 17) What is your view of the quality of the support provided to DGP grantees (by SPICE staff or subcontractors)? Was it similar to that provided to SPICE grantees? 18) What is your view of the relevance and quality of the support provided directly by USAID staff to certain DGP grantees? Was it more relevant or less? Better quality or worse? 19) What effect did SPICE support have on the capacity of the DGP grantees? Do you think SPICE had more impact on the capacity of DGP grantees than USAID staff support, or less? 20) What are the main strengths and weaknesses of SPICE support and USAID support in terms of achieving impact on DGP grantee capacity? 21) Was SPICE able to measure and clearly establish changes in capacity? (probe on the use of OCA, check what other methods were used by SPICE or USAID) Were these tools effective in prioritizing and monitoring the capacity development activities and results? General/Closing 22) What were the most successful approaches and methods used by SPICE, which could be replicated in future programming in Sri Lanka or elsewhere? a. Were there specific approaches that worked well in the relatively closed space, prior to 2015 political transition? 23) What are the key areas of organizational capacity in which Sri Lankan civil society in general remains weak or undeveloped? How could those best be addressed? 24) What should be done now to expand the space for civil society and citizen participation in Sri Lanka (by USAID and/or by other actors)? a. What should be the priorities for USAID in particular? What part of civil society/what type of CSOs should be the priority? What type of support is most needed? 25) What other recommendations do you have for support of civil society, considering the current situation? FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT C-6 26) Do you have any suggestions of other people with whom we should speak, to further explore the kinds of questions we talked about today? SURVEY QUESTIONS Introductory Questions 1) Name of your organization: ______________________________________________ 2) Which of the following best describes the main areas of work of your organization at present? Please, mark all that apply.  Delivery of social services  Local governance and civic participation  Advocacy  Policy analysis and research  Protection of human rights  Other> please specify 3) How long has your organization been active?  One to five years  Five to ten years  More than ten years 4) What is the geographical coverage of your organization at present?  National (whole country)  Sub-national (two or more provinces)  District (two or more districts in a single province)  Local (within a single district) 5) How many grants has your organization received from SPICE since 2012?  Zero  One  Two  Three 6) How many grants has your organization received directly from USAID under the Development Grants Program?  Zero  One 7) How long have you worked in this organization?  0-3 years  4-8 years  More than 8 years 8) What is your current role in your organization?  Director/head of organization  Coordinator/project manager  Finance or administration staff  Team member  Other (specify) 9) What is your gender?  Male FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT C-7  Female  Other Capacity Development (EQ2) 10) Did your organization receive capacity development support directly from USAID staff?  Yes or no – if no, skip section A 11) Did your organization receive capacity development support managed or delivered by SPICE?  Yes or no – if no, skip section B A. USAID direct support 12) Did you personally participate in capacity development activities provided to your organization by USAID staff?  Yes or no 13) For how long did your organization receive capacity development support directly from USAID staff?  1 year, 2 years, more than 2 years 14) To what extent was capacity development support by USAID staff relevant to the needs of your organization and its staff?  Scale of 1 to 4 - very irrelevant, somewhat irrelevant, somewhat relevant, very relevant 15) What is your view of the quality of capacity development support by USAID staff?  Scale of 1 to 4 - Poor, fair, good, excellent 16) How did capacity development support by USAID staff affect the capacity of your organization?  Scale of 1 to 4 –no effect, very negatively, somewhat negatively, somewhat positively, very positively 17) How did capacity development support by USAID staff affect the capacity of the staff of your organization?  Scale of 1 to 4 –no effect, very negatively, somewhat negatively, somewhat positively, very positively 18) In which area of capacity did you see the most impact?  Open-ended question 19) Following the USAID direct capacity development support, did you see changes in the way your organization works? (including internal operations and external activities and relations)  Yes or no (if no, skip to next relevant question) 20) What kind of change did you see in the way your organization works?  very positive, somewhat positive, somewhat negative, very negative, no change B. SPICE support Note: The following questions refer to intensive capacity development delivered or managed by SPICE, which included a series of support activities based on a comprehensive assessment of your organization (either by USAID or by SPICE). They are not intended to cover other capacity-related activities of SPICE such as occasional group trainings or CORE VALUES training. 21) Did you personally participate in this type of capacity development activities provided to your organization through SPICE and its subcontractors?  Yes or no (if not, can suggest they refer the survey to another person in the organization) 22) For how long did your organization receive intensive capacity development support through SPICE?  1 year, 2 years, more than 2 years 23) To what extent was capacity development support by SPICE relevant to the needs of your organization and its staff?  Scale of 1 to 4 - very irrelevant, somewhat irrelevant, somewhat relevant, very relevant FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT C-8 24) What is your view of the quality of SPICE capacity development support?  Scale of 1 to 4 - Poor, fair, good, excellent 25) How did capacity development support by SPICE affect the capacity of your organization?  Scale of 1 to 4 –no effect, very negatively, somewhat negatively, somewhat positively, very positively 26) How did capacity development support by SPICE affect the capacity of staff of your organization?  Scale of 1 to 4 –no effect, very negatively, somewhat negatively, somewhat positively, very positively 27) In which three areas of capacity did you see the most impact on your organization?  Open-ended question 28) Following the support from SPICE, did you see changes in the way your organization works? (including internal operations and external activities and relations)  Yes or no (if no, skip the next question) 29) What kind of change did you see in the way your organization works?  Scale very positive, somewhat positive, very negative, somewhat negative, no change C. Other 30) Does your organization have major capacity needs that USAID and SPICE were so far unable to address?  Yes or no (if no, skip the next question) 31) Which of the following areas best describes those unmet needs? (select up to 5)  Strategy/mission/vision  Internal governance  Financial management  Fundraising and income generation  Human resources management  Technical skills of staff  Advocacy  Communications and public image  Negotiation  Monitoring and evaluation  Knowledge of best practices in relevant thematic area  Research and analysis  Information management  Needs assessment and project design  Project planning and management  Teaching and training skills  Networking  other – can provide narrative answer General/Closing (EQ3 and other) 32) What are the key areas of organizational capacity in which Sri Lankan civil society in general remains weak or undeveloped? (select up to 5)  Strategy/mission/vision  Internal governance  Financial management  Fundraising and income generation  Human resources management  Technical skills of staff  Advocacy FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT C-9  Communications and public image  Negotiation  Monitoring and evaluation  Research and analysis  Knowledge of best practices in relevant thematic area  Information management  Needs assessment and project design  Project planning and management  Teaching and training skills  Networking  other – can provide narrative answer 33) What kind of support would be the most useful to Sri Lankan civil society in the next 5-10 years?  Open ended 34) What other recommendations do you have for support of civil society, considering the current situation?  Open ended FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT D-1 ANNEX D: EVALUATION INFORMANTS BY CATEGORY The Team held interviews and focus group discussions with individuals belonging to the categories defined below. Type of Informant Number of KIIs Number of FGDs USAID and U.S. embassy staff 6 0 Implementer staff (including IFES, ICNL and MSI) 11 1 SPICE subcontractors and consultants 2 1 SPICE grantees 24 1 DGP grantees 5 0 Donors and experts 4 0 CSOs and networks (non-grantees) 2 0 Beneficiaries and participants in grant activities 0 15 Other 3 1 Total KIIs and FGDs 57 19 Total participation 145 175 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT E-1 ANNEX E: LIST OF DOCUMENTS REVIEWED SPICE Project Documents  Cooperative Agreement and all modifications  Original and revised program descriptions  Performance Management Plan, original and modifications  Quarterly Progress Reports  Work Plans for each year  APS and RFAs issued by SPICE  Report on Mid-term Review, final version 2015  Sub-award contracts and modifications, including IFES and ICNL  Criteria for grant selection, various dates  Sample of grantee reports  Sample of grantee OIPs and BMRs (DGP and SPICE)  Sample of training workshop reports and evaluations  Several IFES quarterly reports and final report, November 2015  IDF assessment tool and sample of self-assessments by grantees  Network assessment tool  Security and the Capacity Building Process, PowerPoint presentation, June 2014  Discussion Paper on Strategy Planning by Mirak Raheem and Jeanne Samuel, unspecified date USAID reports and information:  Project Appraisal Document, 2012  USAID/Sri Lanka Country Development Cooperation Strategy 2011-13  USAID CSO Sustainability Index for Asia 2015  One-page summaries of DGP grant projects  Mid-term Performance Evaluation of Development Grants Program Portfolio, 2016  Fixed Amount Awards to Non-Governmental Organizations, An Additional Help Document for ADS Chapter 303, December 2014  USAID/Sudan: DG fixed obligation grants (FOGs), Evaluation report by Partners in Development Services, January 2013 Other documents and reports:  State of Civil Society Report, Civicus, 2013 and 2015  U.S. State Department Human Rights Report, 2015  “We are afraid of them. They are afraid of us: A Study of the Context in which CSOs and CBOs in Sri Lanka function and its impact on Freedom of Expression, Association and Assembly.” Report by Ambika Satkunanathan, March 2015 (commissioned by SPICE)  Assessment of Sri Lanka’s Legal and Regulatory Framework relating to CSOs, report by Kishali Pinto-Jayawardena and B. Skanthakumar, June 2015 (commissioned by SPICE)  Executive Summary of Final Report of Consultation Task Force on Reconciliation Mechanisms, November 2016 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT F-1 ANNEX F. DESCRIPTION OF DEVELOPMENT GRANTS PROGRAM Note: The information below is an extract from the DGP evaluation Scope of Work from mid-2016. USAID/Sri Lanka manages the Agency’s largest Development Grants Program (DGP) portfolio in the world in terms of number of awards, rather than total value. In support of President Obama’s 2013 “Stand with Civil Society Initiative” to strengthen civil society organizations, the Sri Lanka DGP portfolio builds on past successes and supports economic regeneration and community recovery activities in economically-lagging and former-conflict areas. More specifically, the awards in the DGP portfolio: 1) Strengthen the capacity of service providers to provide both access to quality services for people with disabilities and better livelihood opportunities; 2) Support the acceleration of sustainable economic recovery and growth opportunities in economically-lagging and former conflict areas; and 3) Support the strengthening of selected Sri Lankan non-governmental organizations (NGOs). The DGP portfolio advocates for issues of common concern and establishes alliances with local government and/or private sector partners to deliver more effective services. Consistent with USAID/Sri Lanka’s overall strategy, these awards maintain a focus on interventions/services that respond to the needs of youth, women-headed households, ex-combatants, Internally Displaced Persons (IDPs), and other vulnerable populations. The Mission used a three-prong approach to provide capacity building support for the recipient organizations: 1) The organization itself was required to execute a plan to improve their capacity. 2) USAID staff was required to provide capacity building/training/mentoring efforts. The weaknesses identified in the OCA process were to be addressed by hands-on training from the USAID Office of Financial Management (OFM), the AORs, the Program and Policy Support Office (PPS), and the Office of Acquisition and Assistance (OAA). 3) SPICE was contracted to provide capacity building training and consultations. SPICE’s role is limited to contracting for the provision of training services. A total of 13 DGP awards were finalized for implementation between 2012 and 2016. Total funding was $11,303,570. DGP awards are managed by two Development Objective (DO) teams in the Mission: the Governance and Vulnerable Populations team (GVP) and the Economic Growth (EG) team. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT G-1 ANNEX G: RESULTS FRAMEWORK AND SUMMARY OF SPICE Note: This summary is drawn from the revised project description contained in the Year 4 Work plan of SPICE (October 1, 2015 to March 17, 2017), in order to reflect the final stage of the project. The Project has a countrywide focus and works with CSOs as defined in the original proposal. It has a balance between regional and national interventions. With cognizance of the less-constrained space and to further expand and consolidate, SPICE activity supports a fuller spectrum of interventions. These include, (a) bringing issues to the attention of decision-makers; (b) influencing prioritization and agenda setting; (c) formulating solutions, providing alternatives and constructive suggestions; (d) helping in implementation; (e) collecting and channeling feedback, particularly with regard to the impact on affected populations; and (f) mobilizing the affected people and building the capacity of stakeholders to be able to do the above. Results Framework of SPICE Outcome 1: Improved protection of citizens’ rights by Sri Lankan organizations. To achieve this outcome, SPICE activity will continue to support CSOs working at national and regional levels to protect and advance rights. This includes working on and advocating for critical and pressing human rights issues (e.g. torture, disappearance, arbitrary detention, women’s rights, protection of lesbian, gay, bisexual and transgender and minority rights, land rights, farmers’, workers’ and fishermen’s rights, the right to information and dissent, etc.). In addition, given the opening of space, SPICE activity will focus on building on its work on accountability and addressing the culture of impunity. In doing so, SPICE activity will further develop its national and local interventions and partnerships, taking into consideration the evolving transitional justice (truth, justice, reparations, non-recurrence) discussions at the country level. Outcome 2: Enhanced civil society support for development, reconciliation, and governance. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT G-2 To achieve this outcome, SPICE activity will continue to support initiatives that foster active citizenship (e.g. civic documentation, engagement with local government), promote inter-religious and inter-ethnic understanding (including people-to-people interventions), support inclusivity of vulnerable communities (differently abled, ex-combatants, women-headed households, etc.). Taking into consideration recent developments, SPICE will also support initiatives to consolidate democratic gains and improve governance. This will take the form of supporting civil society engagement in constitutional reform processes and improving government’s accountability and performance on issues like corruption, freedom of expression, right to information, rule of law, independence of the judiciary, and human rights (e.g. by engaging the independent commissions and related institutions). Taking advantage of the space afforded by transition, SPICE activity will also support civil society initiatives that advance work on addressing nation-wide concerns and participating in reforms (e.g., through policy initiatives on power sharing, devolution and capacity￾building stakeholders). Strengthening civil society’s technical capacities to engage in these processes will also be crucial. SPICE has evolved a strong gender-equality focus and this will continue. It will build on SPICE’s work on addressing sexual and gender based violence, improving rights, wellbeing, and inclusivity of vulnerable women (female-headed households in North East, women workers’ rights). The interventions will support civil society’s engagement on these issues at operational and policy levels as well as with relevant government stakeholders. Outcome 3: Enhanced management and technical capacity of Sri Lankan organizations supporting civic participation and inclusive development. Under this outcome, SPICE will continue to provide systematic capacity development support to selected SPICE grantees. It would also provide support to DGs as agreed with the Mission. In addition, SPICE will support sector-wide capacity-building initiatives through analysis and training (e.g. civil society round tables and Core Values training to a new batch of participants.) FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT H-1 ANNEX H: EXAMPLE OF FOG MILESTONES The following is an extract from a FOG agreement for a SPICE grantee, which illustrates the level of detail and multiple reports and other deliverables that often were included in a single milestone. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT H-2 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT H-3 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT I-1 ANNEX I: SPICE CAPACITY DEVELOPMENT RECIPIENTS BY CATEGORY This table presents a summary of the five main areas in which SPICE provided or contracted capacity development services, and the number of SPICE grantees and DGP grantees targeted with each category of support. Some grantees benefited from multiple types of support in a particular category, while others (most often DGP) may have received only one service in a category. Category Most common types of support No. of SPICE Grantees No. of DGP Grantees Governance Strategic planning / strengthening governance systems / registration / constitution review 20 9 Management Resources Gender / reporting / volunteer development / M&E / proposal writing / project management 20 0 Human Resources Leadership / skills audits / HR manuals / salary structures / management skills training 11 5 Financial Management Finance and admin manuals / accounting systems / resource mobilization 16 4 External Resources Media strategies and training / website / library 6 0 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT J-1 ANNEX J: ROLE OF ORGANIZATIONAL DEVELOPMENT FACILITATORS Note: The information below was prepared by SPICE to describe and serve as a guide to their approach to capacity development, especially the use of Organizational Development Facilitators. Support for Professional and Institutional Capacity Enhancement (SPICE) Project Capacity Building of Civil Society Organizations Principles of Engagement with Civil Society Organizations The SPICE Project’s Partner Portfolio The SPICE project works with a range of grantees; including grassroots Community Based Organizations and Non-Governmental Organizations operating at the district and national level. These grantee Civil Society Organizations (CSOs) seek to protect rights, improve governance, promote reconciliation and inclusion, increase active and informed citizen participation, address gender-based violence, gender equity and the challenges faced by female-headed households in war-affected areas. Organizational Development The capacity building component of the project is to enhance the management and technical capacity of the above grantee organizations. The two Capacity Building Partners (CBPs) Strategic Inspirations (Pvt) Ltd. (SIPL) and PALTRA together with the designated Organization Development Facilitators (ODFs) facilitate organizational change processes in each of the grantee CSOs, maintain ownership of the capacity building objectives, activities and results. The ODF facilitates the organization to review its capacity, prioritize areas for improvement and dedicate the internal resources to making the changes that strengthen its capacity. The ODF uses her/his expertise to understand the organization and to help identify options for the organization to consider. Cross cutting themes of gender equality and equity, conflict sensitivity, and the Rights Based Approach are integrated in all interventions. The Organization Development (OD) Relationship The Organization Development Facilitators (ODFs) develop an OD relationship with the CSOs in which mutual respect, understanding and trust is built to provide a supportive environment for organizational learning and capacity development. The fundamental elements of this type of external involvement include; a) Creating a safe environment for the organization to be honest in assessing its own capacities, setting priorities for and making its desired organizational changes. b) Conveying respect for and acceptance of the organization by gaining deep knowledge of the organization through sensitive interaction. c) Building trust and confidence in the ODF through maintaining confidentiality of information about the organization, fulfilling one’s obligations to it, and knowing one’s own strengths and weaknesses so that additional resource persons can be brought into the relationship if needed. d) Facilitating a holistic organizational development process that is participatory and leads to improved and valued organizational capacity. Principles of Engagement with CSOs FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT J-2 CBPs and ODFs have experience working in diverse civil society contexts in various capacities and familiar with principles of engagement with CSOs. As ODFs with the selected grantee organizations they establish a professional relationship which is primarily focused on organizational development, rather than program implementation. The following three key principles of engagement outline this professional relationship between the ODFs and the CSOs. Each principle is stated as a core value or belief, and is supported by several examples of behaviour and/or actions that demonstrate how the value is put into practice by the ODFs. 1. Respect the local organization, its values, culture, systems and procedures - Adapt to the organization as a whole, including its culture, history and context. - Acknowledge the ideology of the organization, autonomy, core values, programmatic approach, areas of work, established structure, capacity, and existing skills. - Appreciate their contribution to society, building their confidence. - Facilitate the CSO’s ownership of its development process and align any external support with its priorities and systems. 2. Maintain a professional relationship throughout the organization development process - Honour confidentiality. All information about the organization that is acquired during the course of this engagement is subject to strict confidentiality requirements. No information, documents, photos should be disclosed to external parties outside of the SPICE capacity building team, unless the grantee has given its express consent. - Practice professional values of integrity, transparency, and accountability in one’s conduct and commitments to the organization. Discuss any potential conflict of interest with the SPICE team. - Appreciate diversity of gender, culture, religion, age and all other differences. - Practice conflict sensitivity, e.g. ‘do no harm.’ Avoid creating conflict within and between individuals. - Establish a good rapport with all organization members, ensuring that all perspectives (Board, management, program, finance, admin, females and males, ethnic minorities) are taken into account by the organization leaders during decision-making. - Do not represent the organization in outside forums, except as requested by the organization. - Maintain protocol of established SPICE project structures and channels for communication. Practice open, clear, and accessible communication. Document communications between the ODF and the CSO in writing as per the SPICE guidelines. (The ODF is the primary contact between the Capacity Building Partner and the organization.) 3. Adopt a participatory, holistic approach to the design, implementation and follow up of capacity building interventions - The OD approach addresses the whole organization, not only the SPICE funded project component. - Facilitate the organization to assess, plan and carry out changes, rather than prescribing solutions to its limitations. Empower everyone by allowing for diverse opinions and discussions, and give due consideration to inputs by the CSO itself. - Consider a range of different methods for strengthening capacity, including peer learning as well as training or onsite assistance, and bring in other consultants as needed. - Be conscious of the organization’s workload and timelines in establishing schedules for capacity building events and activities so as to set realistic expectations, and only commence change processes which can be completed. - Follow the Organization Improvement Plan (OIP). - Use objective measures along with participatory discussions to assess results. - Credit/success of tasks accomplished to be attributed to the grantee. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT K-1 ANNEX K: SUMMARY OF CORE VALUES TRAINING Note: This summary is drawn from the Year Four Work Plan of SPICE, and describes plans for the second round of CORE Values training, which was slightly modified from the first round but contained the same basic content. SPICE will repeat the Core Values training program for a fresh set of second level CSO leaders. The CORE VALUES capacity building initiative was conceived by SPICE in consultation with key stakeholders to address gaps and under-served awareness needs of civil society organizations in Sri Lanka, in relation to their diverse mandates and goals. As the title indicates, the focus of this eight-month training program is on facilitating core understanding and providing support to a critical mass of leaders of organizations across an agreed set of concerns and issues that are crucial to their work. The selected areas are not comprehensive, nor self￾contained, but together they constitute a syllabus which will prove invaluable to leaders and their organizations in the current context. It is believed that they collectively represent the core values and perspectives that should characterize the ethos of the sector today, and which the pressure of day-to-day activities of these organizations may tend to de-prioritize. It will include the following six modules:  The Roles and Functions of CSOs in Sri Lanka today  Issues and insights relating to CSO Accountability, including locating this within a broader Governance framework  Understanding the challenges and necessities of Networking that is not top-down, in order to pursue an Advocacy agenda that is generated at the community level  Unpacking “Identity” (including Ethnicity) in the Lankan context  Understanding and engagement with Nationalism(s) at multiple levels and contexts  Addressing Gender concerns, especially in relation to culturalist discourse Anticipated Results  A network of 2nd tier leaders identified and strengthened through capacity building on fundamentals of civil society roles and functions, accountability and governance, identity – inclusive of ethnicity, nationalism and gender and a capstone dealing with all of the themes.  The trained leaders are able to engage in bringing about constructive changes in their organizations and/or in their programmatic work. FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT L-1 ANNEX L: CONFLICT OF INTEREST STATEMENTS FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT L-2 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT L-3 FINAL PERFORMANCE EVALUATION OF SPICE ACTIVITY – DRAFT REPORT L-4 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov