This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the Monitoring, Evaluation and Learning program, DevTech Systems, Inc. STRENGTHENING ADVOCACY AND CIVIC ENGAGEMENT (SACE) PROGRAM: MID-TERM PERFORMANCE EVALUATION Cover photos, clockwise from top left: DRAC, CENSOJ and HERFON organize a "1,000 Person March" to demand increased transparency of spending on healthcare; Citizens meet Ward Development Committee members in Batagarawa Ward, Katsina State; Renovated primary healthcare center in Sabon Garin Chori, Kaduna State; Same primary healthcare center prior to renovations. Photo credit: SACE for USAID MEL PROGRAM FOR USAID USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | II This page is intentionally left blank. III | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV STRENGTHENING ADVOCACY AND CIVIC ENGAGEMENT (SACE): MID-TERM PERFORMANCE EVALUATION The Strengthening Advocacy and Civic Engagement (SACE) Program seeks to increase civil society capacity and engagement with public institutions to influence the development and implementation of key democratic reforms at national, state, and local levels. Submitted November 21, 2017 Contract No: AID-OAA-I-15-00018 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | IV This page is intentionally left blank. 1 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV CONTENTS ACRONYMS.......................................................................................................................................................................2 EXECUTIVE SUMMARY..................................................................................................................................................4 INTRODUCTION............................................................................................................................................................9 BACKGROUND .............................................................................................................................................................10 METHODOLOGY ..........................................................................................................................................................12 FINDINGS/CONCLUSIONS........................................................................................................................................13 ISSUES AND CHALLENGES ........................................................................................................................................39 RECOMMENDATIONS/FUTURE DIRECTIONS ...................................................................................................40 REFERENCES....................................................................................................................................................................46 ANNEXES .........................................................................................................................................................................48 ANNEX I: Evaluation STATEMENT OF WORK (SOW)......................................................................................49 ANNEX 2: Evaluation Team .........................................................................................................................................65 Annex 3: EVALUATION WORK PLAN....................................................................................................................67 ANNEX 4: Data Collection Instruments...................................................................................................................76 ANNEX 5: List of INDIVIDUALS CONTACTED ..................................................................................................90 ANNEX 6: KEY INFORMANT and FOCUS GROUP DISCUSSION SUMMARIES (AS AN ATTACHMENT)..............................................................................................................................................................92 USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 2 ACRONYMS ALS Annual Learning Summit ANEEJ Africa Network for Environment and Economic Justice BACIPEM Bauchi Coalition for Improvement of Public Expenditure Management BMO Business Membership Organization CAI Community in Action Initiative CCIDESOR Citizens Centre for Integrated Development and Social Rights CENSOJ Centre for Social Justice CISLAC Civil Society Legislative Advocacy Centre CITAD Centre for Information Technology and Development COR Contracting Officer’s Representative CRC Citizen’s Report Card CSO Civil Society Organization DQA Data Quality Assessment DFID Department for International Development EDOPADEC Edo State Oil and Gas Producing Areas Development Commission EiE Enough is Enough ESCFAL Edo State Co-Operative Farmers Agency Limited ET Evaluation Team FCT Federal Capital Territory FG Focus Group FGD Focus Group Discussion FY Fiscal Year GON Government of Nigeria HDI Human Development Initiatives HERFON Health Reform Foundation of Nigeria ISWF Innovation and Spread the Word Fund JONAPWD Joint National Association of Persons with Disabilities KIIs Key Informant Interviews LAPDO Life and Peace Development Organization LGA Local Government Area LITE-Africa Leadership Initiative for Transformation and Empowerment M&E Monitoring and Evaluation MAISE Model for Accessible and Inclusive School Environment MNCH Maternal, Newborn, and Child Health NASS National Assembly NCDF Nembe City Development Foundation NDCEF Niger Delta Civic Engagement Forum NDDC Niger Delta Development Commission NDI Niger Delta Institution NEITI Nigerian Extractive Industries Transparency Initiative NGO Non-Governmental Organization NI Network Improvement NNEW Nigeria Employers Consultative Association Network of Entrepreneurial Women NNPC Nigerian National Petroleum Corporation PEA Political Economic Analysis PHCUOR Primary Health Care Under One Roof PIND Foundation for Partnership Initiatives in the Niger Delta PMP Performance Management Plan 3 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV PMU Project Management Unit PPDC Public Private Development Centre PWD Person with Disabilities SACE Strengthening Advocacy and Civic Engagement SAVI State Accountability and Voice Initiative SDIC Social Development Integrity Centre SMW Social Media Week STAR System for Transformation and Results STARNET System for Transformation and Results Network SUBEB State Universal Basic Education Board UBE Universal Basic Education UBEC Universal Basic Education Commission UPFFA Ugboroke Progressive Fish Farmers Association USAID United States Agency for International Development USG United States Government VFM Value for Money WARDC Women Advocates Research and Documentation Centre WDC Ward Development Committee WEWE Widows and Orphans Empowerment Organization WWD Women with Disabilities YAF Youth Alive Foundation USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 4 EXECUTIVE SUMMARY DevTech Systems, Inc. is pleased to submit to the Peace and Democratic Governance (PDG) Office of the United States Agency for International Development (USAID) Mission in Nigeria the mid-term performance evaluation report for the Strengthening Advocacy and Civic Engagement (SACE) program. Mid-term evaluation results are intended for review by the PDG Office and other Mission staff, other United States Government (USG) officials and representatives of non-USG donor organizations, the Government of Nigeria (GON), and civil society organizations (CSOs). PROGRAM BACKGROUND The SACE program is a U.S.$19.3 million, five-year initiative that was awarded to Chemonics in 2014, with Root Change as a strategic partner. SACE’s funding includes support provided by the Chevron-affiliated Foundation for Partnership Initiatives in the Niger Delta (PIND) for activities in the Niger Delta region. The SACE program follows earlier USAID assessments and program interventions, which culminated with the decision to fund a new program to continue enhancing CSOs’ capacity to influence public institutions whose functions are to serve citizens’ interests. The SACE program’s development hypothesis is “that increased civil society capacity and engagement with public institutions will influence the development and implementation of key democratic reforms at national, state, and local levels if the security situation does not deteriorate.” These goals were to be achieved through the rolling out of an innovative anchor/cluster CSO collaborative model focusing on the development of higher-order advocacy capacity building that embraces the lessons of voice and accountability. EVALUATION PURPOSE AND QUESTIONS The evaluation, which is designed to inform decision-making, has the additional purpose of improving implementation of SACE by assessing program goals and activities, including their relevance, effectiveness, and management. As the Mission requests, the report addresses the following questions: 1. To what extent has the design and implementation of SACE been successful in promoting the achievement of the goal of the activity? 2. What worked in SACE’s efforts to increase the financial and technical (e.g., in terms of advocacy) capacity of targeted CSOs/clusters to enable them to engage with the government on reforms? What did not work? 3. What impact did SACE partners have in influencing the government’s development and implementation policy regarding vulnerable populations, such as women, youth, and persons with disabilities? 4. Did SACE’s public awareness activities contribute to strengthening public discourse and support for key democratic governance issues? 5. How did SACE’s support for Niger Delta implementing partners contribute to their advocacy for economic reforms/growth in that region? 5 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV EVALUATION METHODS AND LIMITATIONS The evaluation relied on a mixed-methods approach that included several qualitative data collection techniques, particularly key informant interviews (KIIs) and focus group discussions (FGDs), and a review of the SACE management and administrative reports. The evaluation team (ET) made every possible effort to minimize methodological limitations that could potentially arise, such as biases resulting from purposive sampling, poor recall, and lapses in attribution. In total, 41 KIIs and several FGDs were conducted in Abuja and in the states of Bauchi, Kaduna, Lagos, Oyo, Rivers, Imo, and Edo (the last three of which are located in the Niger Delta region). FINDINGS AND CONCLUSIONS REGARDING SACE DESIGN AND IMPLEMENTATION Without question, the anchor/cluster model has contributed significantly to the enhancement of the policy reform and advocacy capacity of participating CSOs. The model has facilitated the expansion of institutional linkages among CSOs with like-minded policy reform agendas, while enhancing the effectiveness of advocacy initiatives through constant interaction and sharing of successful strategies and procedures. Messaging capabilities have been transformed, both by the ability to articulate concrete and specific reform proposals, and by the CSOs’ improved skills to better communicate to stakeholders the purpose of proposals for policy reform. REGARDING THE IMPROVEMENT OF ADVOCACY FINANCIAL AND TECHNICAL CAPABILITIES CSOs were generally found to be committed to multi-stakeholder engagements and improved their capacity to reach other cluster members when they sought information and advocacy expertise. CSOs also increased their ability to engage in policy advocacy. While “1.0 capacity” (normally associated with strong management procedures that are assumed to lead to impacts) improved mostly among anchors, most CSOs—whether anchor or cluster—benefited from “2.0 capacity” advocacy-enhancing training that focuses on CSOs’ ability to forge alliances, be sector leaders, and respond to constituents. REGARDING THE ABILITY TO INFLUENCE POLICY DEVELOPMENT The SACE approach has proven to be effective in influencing government policy development and implementation regarding vulnerable populations. Selected regional initiatives proved their worth in addressing the basic educational needs of children (in Bauchi and Lagos), in addressing the most pressing health challenges that threaten children and their mothers (access to essential maternal/child health primary services in Kaduna), and in expanding educational opportunities and access to accommodations for the disabled (in the Federal Capital Territory (FCT) and Ibadan). REGARDING PUBLIC AWARENESS ACTIVITIES SACE public awareness activities are strengthening the nature of public discourse and support for key democratic governance reforms. As citizens are becoming more aware of their right to access public information, including financial data, some government officials are evidencing a growing willingness to comply with demands of the civil society. These changes are occurring as better-informed citizens, who USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 6 are increasingly cognizant of their rights and encouraged by the dissemination of good governance messages, apply increasing pressure on public officials to be more transparent and accountable. REGARDING ADVOCACY FOR ECONOMIC REFORMS/GROWTH IN THE NIGER DELTA SACE support for Niger Delta implementing partners made remarkable contributions to the improvement of CSO advocacy capacities and strategic engagement with policy-makers and Niger Delta institutions. Particularly salient were the achievements in the open-budget campaign, completion of hitherto abandoned projects, and inclusive growth through statutory allocations to marginalized groups and sectors. SHORT-TERM RECOMMENDATIONS IMPROVING SACE TRAINING PROCEDURES These short-term recommendations flow directly from the KIIs, as almost all the interviewees indicated that training procedures could be improved. SACE could take some relatively simple and straightforward measures to remedy issues raised, such as lengthening the time for workshops and seminars including the Annual Learning Summits (ALSs), and expeditiously providing training participants with training materials on a flash drive or via the internet prior to or after the training. SACE could also engage the services of training specialists to help design more user-friendly training procedures built around observations of current practices and input (obtained through brief questionnaires, focus groups, etc.) the trainees themselves provide. The SACE program must also evaluate means to minimize the loss of institutional memory suffered by many of its partner CSOs. Going forward, SACE should creatively liberalize the process of nominating staff for the different capacity-building sessions, while also requiring that anchor and cluster staff trained in unique skills share that knowledge with other members of their organizations. TARGETED FUNDING MECHANISMS TO ENCOURAGE/REWARD SUCCESS The joint development of cluster work plans has been effective for the design and implementation of cohesive advocacy approaches, but has, in some ways, interfered with the ability of CSOs to be quick on their feet, assume risks, and take advantage of windows of opportunity. To ease these concerns, SACE should consider establishing several new short-term funding mechanisms for specific activities not included in the routine work plan. The ET proposes a “cluster performance incentive fund” to be accessed under special circumstances by individual cluster members to support specific advocacy agenda items, for example in support of a local advocacy initiative. In addition, SACE should evaluate establishing a “windows of opportunity fund” that could be tapped by clusters as a rapid response mechanism to enable them to respond to emerging opportunities in their issue areas and regions of activity. Finally, an innovative economic development advocacy fund would be particularly appropriate for Niger Delta clusters given their focus on economic and policy reforms. THE SUSTAINABILITY ISSUE Sustainability is always a particularly difficult issue to address in a world of budget constraints and competing needs for resources. Nearly all CSOs depend on donor support, which affects institutional 7 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV sustainability due to short funding cycles. The ET believes the answer may lie in expanding SACE-provided training to include techniques for developing sustainability plans, including acquiring fundraising capabilities, during the remaining life of the program. Other lower-level and monitoring-related recommendations include: • Updating the latest SACE DQA to determine the validity of some of its performance indicators. • Evaluating the need to better capture community-level outcomes. LONG-TERM RECOMMENDATIONS The three short-term recommendations that were presented above should be considered for any future SACE follow-on programming. Given the success of the program, as evidenced by the information provided by informants interviewed, the overall orientation of the SACE program should remain unchanged, and in fact be expanded. BROADEN THE REACH OF THE SACE ADVOCACY APPROACH Follow-on programming could be expanded by including other national priority sectors, such as employment generation, housing availability, and access to clean water. These economic sectors have a major bearing on well-being, which would require CSOs’ intensified engagement with political leaders and government officials. Promoting a broader understanding among citizens of how to bring about policy change could fortify CSOs’ ability to advocate for political and economic reforms at all levels of government. A related and complementary activity would be to conduct concurrent and parallel initiatives to be focused on policy￾makers, thus taking advantage of economies of scale. Public education could also be expanded through relatively simple and affordable initiatives that could take further advantage of both conventional mass media and social media, as SACE’s programming has done. Additionally, a simplified, self-paced on-line STAR course could be developed to demonstrate the essentials of effective policy reform advocacy for interested citizens. INCREASE THE ADVOCACY VALUE OF THE STARNET The STARNET is an indispensable tool to forge sectoral and regional alliances. It is one of the most well￾regarded components of the SACE program. While the anchor/cluster model has worked, a reformulated STARNET iteration should strive to expand its thematic engagements by feeding into a national-level network of nonstate actors with the capability to support regional and national-level policies around transparency, accountability, good governance, and the rights of vulnerable and marginalized groups. Likewise, a future STARNET should strive, through more elaborate mapping and engagement, to broaden the networks of grassroots CSOs. Future programmatic activities should give priority to engaging the traditional and long-standing networks of nonstate actors. Their involvement, once equipped with 1.0 and 2.0 capabilities, could eventually revolutionize the practice of governance in Nigeria. Another needed STARNET upgrade should include giving users the capability to rely on it to conduct integrity checks. Similarly, the STARNET could be redesigned to include development-oriented community-based organizations. Networking and building alliances with these grassroots development USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 8 champions could promote long-term sustainability, even after the mandate of a possible SACE successor program expires. One alternative is to consider supporting a CSO “trust” (akin to the Southern Africa Trust), as part of a potential follow-on activity. This entity, to become self-sustaining in three to five years, would be tasked with providing research, capacity building, and technical support to the broader multi-sectoral CSO community, while assuming a wide ecosystem perspective, and housing and managing a website to broadly disseminate the existing SACE tools and other experiences generated in years to come. OTHER LONG-TERM RECOMMENDATIONS Other obvious, but just as important recommendations for a future SACE-like program include: • Improve networking mechanisms between CSOs and political leaders. While SACE had much success linking CSOs together, it barely scratched the surface in developing comparable two-way devices to ease the flow of information between CSOs and policy-makers. • Expand media access to well-structured electronic mediums (TV, radio, webinars), in which well￾trained policy change advocates could debate with policy-makers the technical and other merits of policy reform proposals. 9 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV INTRODUCTION Since the end of military rule, Nigeria has held five consecutive national elections. In March/April 2015, the latest election, which was won by an opposition party, was regarded as fair and generally peaceful. Election results brought high expectations for change and improved service delivery not only among the population at large, but among civil society organizations (CSOs) as well. However, by the mid-point of President Buhari’s term, the implicit partnership between the CSO community and the Government of Nigeria (GON) was under stress, as political forces in the National Assembly unsuccessfully attempted to curtail the independence of an increasingly vocal CSO community calling for greater government transparency and accountability. The Strengthening Advocacy and Civic Engagement (SACE) program is a U.S.$19.3 million, five-year initiative that was awarded to Chemonics in 2014, with Root Change as a strategic partner. SACE’s funding includes support provided by the Chevron-affiliated Foundation for Partnership Initiatives in the Niger Delta (PIND) for activities in the Niger Delta region. The SACE’s program follows earlier USAID assessments and program interventions, that culminated with the decision to fund a new program to continue enhancing the capacity of CSOs to influence public institutions whose functions are to serve citizens’ interests. The SACE program’s development hypothesis is “that increased civil society capacity and engagement with public institutions will influence the development and implementation of key democratic reforms at national, state, and local levels if the security situation does not deteriorate.” These goals were to be achieved through the rolling out of an innovative anchor/cluster CSO collaborative model focusing on the development of higher-order advocacy capacity development that embraces the lessons of voice and accountability. Forging strategic partnerships between CSOs, coalitions and networks, and other Nigerian critical stakeholders is central to the effectiveness of SACE’s approach. These other stakeholders include the media, reform-minded GON institutions or representatives, private sector actors, influential public figures, as well as other key stakeholders in target areas, such as grassroots organizations, professional associations, and youth groups. Thus, its emphasis is on leadership and innovation. The SACE program was fashioned by U.S. Agency for International Development (USAID) Nigeria’s Peace and Democratic Governance (PDG) Office so as to complement other initiatives, such as the Leadership, Empowerment, Advocacy, and Development (LEAD) subnational governance program and the by now completed Media Support for Strengthening Advocacy, Good Governance, and Empowerment (MESSAGE) program, while also engaging with related conflict management and elections efforts. The four components of the SACE program, being implemented on a national basis, are designed to strengthen: (1) The institutional, organizational, and technical capacity of targeted CSO coalitions and networks, to advance targeted democracy and good governance initiatives; (2) partnerships between CSO-led coalitions and networks, and targeted GON institutions and key stakeholders, to advocate for and monitor select democratic reforms aimed at strengthening transparency, accountability, and responsiveness of government institutions; (3) public awareness, discourse, and governance; (4) capacity of partner business membership organizations (BMOs) and CSOs in the Niger Delta, to advocate for inclusive economic reforms and equitable economic growth. Ultimately, SACE is expected to contribute to more transparent and accountable governance by having the public sector increasingly collaborate with the CSO community. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 10 BACKGROUND Awarded to Chemonics, the SACE program is a five-year, $19.3 million initiative that includes support provided by the Chevron-affiliated Foundation for Partnership Initiatives in the Niger Delta (PIND) for activities to be conducted in Nigeria’s Niger Delta region. Chemonics implemented the four main initiative objectives largely through the application of a novel advocacy methodology that was developed by strategic partner, Root Change. The objectives are to: • Strengthen the institutional, organizational, and technical capacity of CSOs (and their networks, coalitions, and clusters). • Strengthen partnership with government officials, discourse, and engagement on key issues. • Strengthen public awareness, discourse, and engagement on key issues. • Strengthen the capacity of BMOs and CSOs in the Niger Delta. SACE, through its CSO advocacy strengthening efforts, seeks to overcome several contextual difficulties that have historically interfered with the emergence of a strong civil society that can influence the pace of democratic reforms. These constraints include the presence of complex and endemic social problems, isolated instances of advocacy activity and success, the limited sharing of advocacy skills, and the prevailing inability to take advantage of windows of opportunity. To overcome these shortcomings, SACE is utilizing a methodological approach based on what the international community has learned regarding the management of complex and demanding change processes. These lessons suggest that advocacy activities work best when they: 1. Take into consideration the accountability context. 2. Blend supply and demand approaches. 3. Mobilize the actions of diverse actors around a common advocacy agenda. 4. Seek better fit, rather than best practice solutions. 5. Prioritize adaptive learning and innovation. 6. Use robust political economic analysis (PEA) and explore windows of opportunity. Based on these findings of a two-year global study that Root Change conducted in 22 countries on behalf of the International Federation of the Red Cross (IFRC) to assess what capacities make for high￾performing, sustainable CSOs, two principal conclusions emerged. First, aggressive “smart” networking with traditional and nontraditional partners creates high performing CSOs capable of overcoming challenging economic, social, and political country contexts. Second, higher order, “2.0” capabilities (as opposed to “1.0” capabilities, which are normally associated with strong management procedures that are assumed to lead to impacts) drive CSO performance. These may be conceptualized as the CSOs’ ability to forge alliances, be sector leaders, and respond to constituents. These “second generation” practices, based on strong system-level engagement, lead to policy impacts by promoting social capital development, embracing an ecosystem perspective, and giving pride of place to learning and adaptation. Thus, CSOs must engage in ecosystem mapping and adaptive learning, rely on feedback loops, and promote collective impacts. The desired result is to align many different actors’ actions for impact beyond what one organization can accomplish. According to SACE’s development hypothesis, transformational change requires incorporating the fundamental principles of effective voice and accountability programs, reaching beyond Capacity 1.0 inputs; generating sufficient public awareness and engagement, and effectively transferring knowledge. This expected outcome requires an ecosystems approach that aims to influence clusters of 11 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV partner organizations (CSOs, community-based organization (CBOs), policy-makers, donors, and companies) working together to create collective impacts. Such clusters are led by anchor CSOs, whose role is to coordinate and align activities of cluster members around a common advocacy agenda and strategy (i.e., lead from behind), build effective and sustainable relationships with cluster members for issue-based advocacy, through the cluster and broader sector, and promote the success of cluster members, by linking them to each other, as well as to opportunities and resources. As such, the model leads to collective impacts. Aside from the anchor/cluster feature, the methodology that Root Change has developed rests on the application of the System for Transformation and Results (STAR), a suite of tools and change processes that was devised to promote the abovementioned higher-level 2.0 organizational performance. The STAR includes eight drivers that were designed to impel the collective impact principles of the SACE advocacy approach, whose purposes are for clusters to work from a common agenda, engage in complementary activities, further constant communication, and share measurement of results. The STAR drivers in question consist of: • Adaptive strategy and tactics. • Stakeholder engagement. • Monitoring and evaluation. • Member and partner development. • Knowledge exchange. • Alliance building. • Innovation. • Promotion of public awareness. The STAR assessment tool is complemented by four other tools that were designed to enhance cluster(s) effectiveness: 1. The System for Transformation and Results Network (STARNET), which is used to map linkages among advocacy clusters. 2. STAR Feedback, to facilitate communications among cluster members. 3. STAR Cluster Coaching, the regular provision of technical assistance to support cluster activities. 4. The Annual Progress Index, a yearly evaluation of cluster member achievements along the range of STAR performance drivers. A final capacity-building feature is the Annual Learning Summit, an Abuja event that is attended by anchor and cluster members and convened to review advocacy strategies and successful advocacy initiatives. The implementation of the SACE activity began in Fiscal Year (FY) 2014 with the selection of partners and the initial establishment of advocacy clusters (in Inclusive Education, Open Budgeting, Youth Advocacy, Inclusive Health, Peace Advocacy, and Transparency in Extractives). SACE began with 16 CSO clusters, eight each at the national level and in the Niger Delta. However, in 2016, SACE ended its relationship with one of the national anchors, due to evidence of fraudulent financial practices, and in September 2017 the SACE suspended one of the Niger Delta anchor sub-grants with conditions to address similar practices. Thus, by the end of September 2017, SACE was still working with a total of 16 clusters, seven USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 12 of them national plus nine more in the Niger Delta, including two BMOs. These BMOs were two new Niger Delta region clusters that joined the SACE initiative in 2017. In FY 2015 and FY 2016, SACE began to emphasize capacity building and the gradual design and implementation of advocacy initiatives, including preliminary results. In FY 2017, the consolidation of several anchor CSOs became increasingly sustainable (sustainability being defined as having the capacity to directly manage USAID grants) and advocacy capabilities among many cluster organizations also continued to strengthen. At this stage of the activity cycle, emphasis is on the attainment of concrete advocacy results. Results have been encouraging, as in March 2017, the Mission and SACE decided to add a new outcome indicator as part of establishing the Mission’s Performance Management Plan (PMP). Some SACE-associated CSOs have qualified for grants that were awarded by other donors, thanks to their enhanced 1.0 capabilities. These SACE-associated CSOs recorded numerous achievements in unique persons who trained in various advocacy skills. Also, they initiated multiple activities in favor of marginalized groups (defined as women, youth, and the disadvantaged), all of whom apparently achieved gains in terms of sensitivity and inclusion. In addition, they introduced innovative cluster coaching practices and were able to link capacity building to results (via the Annual Learning Summit). Furthermore, SACE takes credit for the promotion of 75 bills and policies at the various government levels promoted by its clusters, as well as for the participation of many CSO stakeholders in governance oversight processes. Cross cluster collaboration has led to several government action demands, concurrently underway. In addition, SACE-affiliated organizations have been involved with several public events, such as the “1,000 Person March.” In 2015, three of the clusters working on health care issues (i.e., Disability Rights Advocacy Center (DRAC), Centre for Social Justice (CENSOJ), and Health Reform Foundation of Nigeria (HERFON)) came together to launch a public awareness campaign to influence public policy. Their core demand, which was highlighted through the march to the National Assembly, was for increased budget allocations and increased fiscal transparency in the health sector. Although the report was accepted by two jurisdictional committees, legislative results were meager. This proved to be a valuable lesson in cross cluster coordination, as the CSO community learned their demands had to be more focused. The initial effort morphed into the one percent campaign, whose focus is ensuring that the one percent statutory allocation of the federal budget to primary health care is implemented in 2018. Many of these efforts would not have been effective without SACE’s insistence on intense public advocacy, whether through traditional media or newer social media. Twitter exposures alone currently approach 50 million. Other innovative practices include engaging the entertainment sector in the Homevida Integrity Film Awards and supporting digital storytelling in Spaces for Change and the Nigeria Women’s Trust Fund Ideathon for young women participating in politics. METHODOLOGY Methodologically, the evaluation relied on a mixed-methods approach that included several qualitative data collection techniques and review of SACE management and administrative reports. The basic approach was to collect personal testimony from SACE staff members, representatives of CSOs, and key government officials regarding their views about the SACE activity. The two primary qualitative data collection techniques were key informant interviews (KIIs) and focus group discussions (FGDs), complemented by observations of SACE-related events that occurred while the evaluation team (ET) was in the field. These included a briefing by the Kaduna Commissioner of Health to CSO representatives on the rehabilitation status of primary health care clinics, as well as a briefing in Edo on a routine review 13 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV meeting of agricultural policy issues. In addition, the ET was able to observe a workshop that an anchor organization had convened in Lagos to review educational reform issues. In total, 41 KIIs were conducted in Abuja and in the states of Bauchi, Kaduna, Lagos, Oyo, Rivers, Imo, and Edo (the last three of which are in the Niger Delta region). Following the initial KIIs that were conducted in the FCT, the seat of Abuja, the ET divided into two sub￾teams and spent a week gathering data in the target Nigerian states. The ET chose the field evaluation sites following the guidelines in the Statement of Work (SOW) of the evaluation. The ET selected KII and FGD participants independently, without input from either the USAID Mission or SACE program staff. The ET did informant selection in accordance with a purposive sampling design, with key informants and FGD participants chosen to represent three respondent categories, all familiar with the SACE program: Anchor CSO representatives, cluster CSO representatives, and policy-makers. Only CSO cluster representatives participated in the FGDs. The ET informed all KII and FGD participants about the mid￾term evaluation objectives, asked to consent verbally to their participation, and promised confidentiality. The team collected the data with the assistance of guides that had been specifically developed for that purpose. The evaluation design includes the standard methodological limitations the ET encountered in the collection of field data using mixed methods. One limitation that is specific to this design was the compressed one-week period that was allowed for field work; it was particularly challenging, given the logistics associated with traveling to different Nigerian sites. In addition, the purposive sampling approach that followed could have contributed to potential selection biases. Furthermore, the ET acknowledges that, since most key informants (other than policy-makers) had ties to SACE, the methodological approach they followed may have had a built in response bias with respect to findings and conclusions. Recall biases might also be present, as respondents may have erroneously attributed to SACE program unrelated activities that were sponsored by another organization or program. The ET tried to minimize these potential biases to the extent possible. Finally, the ET had some difficulties arranging interviews with policy-makers. These individuals are frequently busy, and the ET would have needed more time than the time available to set up meetings with them. Once the main data collection phase of the evaluation was completed, two members of the ET participated in the SACE-sponsored third Annual Learning Summit in Abuja, where they were able to interview additional informants and observe the event. FINDINGS/CONCLUSIONS EVALUATION QUESTION 1 To what extent was the design and implementation of SACE’s anchor/cluster model been successful in promoting as intended key democratic reforms? • Did SACE’s anchor/cluster model work effectively? • Is SACE sufficiently focused on key results in its approach? The SACE anchor/cluster model is designed to help develop a more dynamic and empowered civil society that is capable of serving as a change agent by advocating for economic and political public policy changes and achieving results. A key component of the approach consists in providing support to CSOs for them to come together and develop common policy reform agendas, coordinate strategies, monitor results, USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 14 and share knowledge. At the core of SACE’s approach is the provision of analytical and advocacy tools, so that CSOs can help shape the policy reform process at the federal, state, and local levels through the forging of strategic and sectoral alliances, while mobilizing public opinion in support of the reform agenda. A program document notes the following: 1 “Clusters of organizations working on a clearly defined thematic issue-area, with a shared vision for change, and ‘anchored’ by a legitimate organization responsible for creating and managing collaboration, are a natural and necessary point of engagement. These clusters represent intermediaries between citizens, individual organizations and broader civil society; over time they evolve from relatively isolated approaches targeting individual actors and institutions to a more holistic ‘ecosystem’ approach, which seeks to link social accountability efforts with the broader accountability system within Nigeria. An accountability ecosystems approach considers formal and informal accountability mechanisms and engages a range of actors and institutions involved in promoting accountability, including state and nonstate actors, such as civil society, the media, government officials, institutions of accountability, and other power-holders, among others.” For the anchor/cluster model to attain its potential, it must: Interact with a civil society willing to adopt a systems perspective; include clusters that are committed to continuous learning both to enhance internal operational capabilities and collaborative links; reach a wide range of stakeholders with complementary skills; embrace feedback loops to continuously strengthen the cluster’s strategy and adaptability; practice multipronged strategies to influence a broad range of stakeholders, including the public, influencers, and decision-makers; be committed to funding strategies that are designed to foster regular and continuous communications among clusters. By combining their relative strengths, civil society alliances could influence the nature and pace of change. Ultimately, capacity-building focuses on a systems-oriented approach based on relationships, information sharing, identification of common collaborative agendas, and the alignment of diverse efforts of many actors towards achieving social change goals. For the model to be effective, anchor and cluster members must improve their 1.0 traditional management capabilities, while acquiring higher-order 2.0 advocacy capabilities (i.e., the STAR suite capabilities that are discussed under Question 2, at p. 16) to enhance their ability to influence the public policy reform agenda. By the third quarter of 2017, the number of SACE-sponsored clusters included seven national level clusters as well as nine more in the Niger Delta region, including two new clusters-the Nigeria Employers Consultative Association Network of Entrepreneurial Women (NNEW) and the Life and Peace Development Organization (LAPDO)-having already come on board. According to the STARNET’s tracking data, which are summarized in SACE’s second quarter Performance Report (June 2017), 77 national cluster members demonstrated improved performance in relating to other CSOs, a progression that was also realized among Niger Delta cluster organizations. The next regularly scheduled biannual measurement of this metric is programmed during the fourth quarter of the year. 1 Root Change, Nigeria Strengthening Advocacy and Civic Engagement (SACE) Combined Quarter 4 and Annual Report, Abuja, October 2016, pp. 4-5. 15 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Without exception, the key informants that the ET interviewed commended SACE’s anchor/cluster approach, noting the various ways in which it had enhanced, individually and collectively, CSOs’ advocacy capabilities. Common avenues that the key informants often cited to achieve these positive results include ecosystem mapping, adaptive learning, feedback loops, and collective impact assessments. These tools emphasize the model’s adaptive capacity, enabling anchors and clusters members to proactively identify windows of opportunity and take advantage of them. The model also has underlying cross sectoral applicability, a benefit used by SACE to navigate a complex landscape. In the process, SACE managed to assemble an array of clusters with a varied sectoral orientation and composed of anchor/cluster CSOs advocating for key democratic reforms in inclusive education, open budgeting, youth advocacy, inclusive health, peace advocacy, and transparency in extractives. In the implementation of this and related components, the ET identified some shortcomings, mainly associated with the (at times restricted) ability of anchor CSOs to implement mutually agreed cluster work plan activities. Due to how USAID releases grants, this process is made more complicated by the imposition of safeguards, given the high incidence of fraud in Nigeria. For the most part, key informants did not attribute this shortcoming to SACE itself; in only one instance did a key informant outlier express extreme dissatisfaction with SACE, claiming that “clusters get nothing [financially] from participating in the initiative.” A more consequential issue that was often cited in KIIs was SACE’s tendency to overload its already too short training sessions with too much material, which prevented participants from properly assimilating what was being taught. Another commonly heard adverse comment focused on lines of communication between SACE and CSO staff, which frequently caused misunderstandings and acrimony, as supposedly mutually-agreed decisions were often reversed by SACE management. During the early days of the program, this problem might be caused by a tight implementation timetable and a reformulation of the mentoring concept that had originally been proposed as the foundation of SACE’s training activities. Some key informants suggested these issues were related to the high turnover rate of the SACE’s staff, a concern that SACE’s management echoed. Finally, a last potential implementation flaw may be that some of the program’s indicators (e.g., counts of unique individuals trained on specific skills) could result in double counting, with exaggeration always being a possible reporting bias. When appropriate, the evaluation report narrative will allude to these issues. In the ET’s opinion, these relatively minor shortcomings only marginally detract from what is otherwise quite a robust program. Perceived strengths of the model are reviewed next, roughly ranked according to the relative value that key informants attributed to the anchor/cluster model. FINDINGS Among the most notable findings about SACE’s anchor/cluster model, the KIIs allowed to highlight that it is an effective mechanism in several situations, which are listed and illustrated in the following: It encourages CSOs with common advocacy agendas to come together. Although many informants mentioned that an ingrained history of non-governmental organization (NGO) collaboration existed before the SACE program, informants did commonly cite SACE’s encouragement of collaboration. What the anchor/cluster model has changed is how the various CSOs within a cluster relate to each other. As several key informants mentioned, the approach prevents CSOs from working in “isolated” silos. The model encourages mutually supportive interactions regarding advocacy initiatives—formalized in jointly USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 16 developed work plans, dissemination of advocacy training from the anchor to cluster members, and equality of opportunity in the design of advocacy approaches and adoption of innovative practices. The case of the Joint National Association of Persons with Disabilities (JONAPWD) provides a classic example. Comprised of several umbrella bodies of persons with disabilities, individual member organizations previously operated in isolation and therefore were unable to effectively advocate for the rights of their members. The acquisition of the SACE-promoted 2.0 capacities has changed the engagement dynamics of the association, transforming it from a “traditional charity model to one of equal partnership on the policy roundtable.” It facilitates the sharing of information and use of the most effective advocacy strategies. Fluid and effective interactions are at the heart of the model, driven by technology-based tools, such as STARNET, environmental mapping, and social media, among others. Embedded in the anchor/cluster model is the continued sharing of information through various venues, including the team-building effect arising from the collaborative development of an advocacy work plan and specific activities therein. These interactions are facilitated by the STAR suite of tools. As one key informant indicated, the STARNET is useful, “as it allows you to tap the experiences of other CSOs and suggests new and more effective ways to do things.” It allows to cross fertilize advocacy approaches by blurring sectoral and geographical boundaries. The anchor/cluster model is flexible by design and can accommodate different functional linkages. For example, this allows sector-oriented anchors to join forces to support initiatives from different advocacy perspectives. A current instance of this is the strategic anchor alliance between HERFON and CENSOJ, which focuses on the simultaneous promotion of reform policies to increase access to basic primary health care. In this case, HERFON takes a service delivery perspective and CENSOJ takes a fiscal perspective. It allows to craft appropriate messages to avoid misunderstandings and antagonizing policy￾making stakeholders. Since one of the fundamental underpinnings of the STAR approach is the development of effective and sustainable stakeholder collaborative linkages, the model rests on the development of mutually reinforcing relationships with the policy-making community. Thus, it is appropriate to avoid messages that may be misconstrued or that may antagonize decision-making authorities. As one key informant noted, the anchor/cluster model “prevents policy-makers from becoming confused as the advocacy is made with a single voice.” To this end, the anchor/cluster model— given its ability to assimilate input from different constituencies—is generally able to integrate different stakeholder perspectives as it focuses on shared policy objectives, while minimizing tactical differences. Conciliatory messages play a helpful role in narrowing the gap between actual or perceived policy objective differences between the advocacy and policymaking communities. In the words of a key informant, developing a common agenda and communication plan helps “understand what the government is saying, what the CSO community understands, and what the CSO community wants to communicate to the public.” 17 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Traditionally marginalized women farmers, trained by WARDC, discuss their needs with a local chief. It allows to reinforce the complementarity of related advocacy initiatives that are conducted at the various government levels. The evidence for this finding is convincing, as many key informants related that the success of advocacy campaigns that were promoted at the state or local level frequently dovetails with the reforms that were implemented at the federal level. In the health care sector, for example, initiatives that are launched in Abuja to expand access to basic health care (including increasing spending levels and associated mandates) are often complemented at the state level with related initiatives to comply with federal guidelines or access supplemental operational funds. The same applies to basic education, where states must satisfy specific criteria to qualify for federally-provided grants. In this case, state-based CSOs advocated for and succeeded in having state governments release required matching funds, so as not to forfeit federal subsidies. A parallel and coordinated advocacy campaign resulted in legislation that increased the basic education federal grant to states from two percent to three percent of the national consolidated budget. It enables the inclusion in sector-oriented clusters of media-focused CSOs to disseminate and socialize advocacy campaign objectives. Most of the key informants who were interviewed alluded to this feature of the anchor/cluster model. Its relevance lies in the ability of CSOs with a media orientation to educate the public and stakeholders, including policy-makers, while mobilizing public opinion in support of specific reform proposals. It assists with the dissemination in the CSO community of the STAR public policy advocacy suite and related tools (see response to Question 2, at p. 16). Last but not least, this has been a significant component of the model. It has contributed to the development of 2.0 capabilities among USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 18 participating clusters, particularly, but not exclusively, strengthening such skills in anchor and primary cluster organizations. This process, in turn, has helped anchors coach other secondary participating cluster CSOs. Through this mechanism, the reach of the advocacy-related SACE suite of skills has cascaded down the universe of CSOs engaged in policy reform advocacy. CONCLUSIONS The anchor/cluster model has contributed significantly to the enhancement of participating CSOs’ advocacy capacity towards policy reform. The model has facilitated the expansion of institutional linkages among CSOs with like-minded policy reform agendas, while enhancing the effectiveness of advocacy initiatives through constant interaction and sharing of successful strategies and procedures. Messaging capabilities have been transformed, both by the ability to articulate concrete and specific reform proposals, and by the CSOs’ improved skills to better communicate to stakeholders the purpose of policy reform proposals. While the ET believes that the STARNET is a great tool to empower nonstate actors, it needs to be domesticated, so that, by the end of the program, it becomes a fully-owned Nigerian commodity. While currently the CSO membership is comprised mostly of formalized organizations and associations like CSOs and NGOs, it largely excludes CBOs, faith-based organizations (FBOs), and community development associations (CDAs), except for Citizens Centre for Integrated Development and Social Rights (CCIDESOR) and Leadership Initiative for Transformation and Empowerment (LITE)-Africa that include some community-based groups as cluster members. Mapping such groups into the STARNET and involving them in cluster activities would assist with the further institutionalization of the anchor/cluster model while increasing opportunities for more sustainable and long-term impacts. Finally, the anchor/cluster model has an in-built capacity to consistently upgrade the advocacy capabilities of the CSO community as relational networks expand with additional CSOs and other stakeholders, becoming involved in SACE-related advocacy programs. IMPLEMENTATION CONCERNS At the same time, SACE has faced some implementation problems, including what some key informants described as overly demanding and information-packed training procedures that tax trainees’ ability to fully assimilate the concepts they were taught. Also, the implementation of SACE has been marred by recurring problems with personnel’s turnover, which, according to some key informants, have detracted from the program’s performance. Finally, key informants flagged to the ET what they understood to be unduly complex administrative procedures SACE imposes to release grant funds, a continual hindrance often preventing the optimal and timely implementation of work plan activities. 19 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV IS SACE SUFFICIENTLY FOCUSED? As to the second sub-question, no specific findings based on KIIs could be reported, as answers would reflect situational perspectives. The sub-question could be interpreted in three different ways: • Was the geographic focus too broad? • Did it embrace too many jurisdictional levels of government? • Was the sector coverage too dispersed? The first two interpretations can be eliminated, since, as the review of the findings above pointed out, geographic and jurisdictional diversity is a strength of the anchor/cluster model in the complex Nigerian governance context. However, it could always be alleged that the coverage of SACE’s sector may have been too dispersed. This question cannot be answered through the sectoral perspectives that the key informants provided. Motivated by their own concerns and perspectives, they would have been likely to indicate a preference for greater SACE concentration (ideally in their own sectors), alleging they would have achieved even better advocacy outcomes with more abundant resources. Instead, the ET can provide a macro-level answer, supported by evaluation findings. The SACE approach constitutes a novel, effective, and innovative way to bring about policy reforms, regardless of the development sector in which it is applied. This is a major finding with potentially significant ramifications for USAID and other donors in Nigeria. It is also a relevant finding for USAID country development strategies in other countries where CSOs are weak and must be nurtured to become effective advocates for governance policy reforms, particularly with respect to increased transparency and accountability. EVALUATION QUESTION 2 What worked in SACE’s efforts to increase the financial and technical (e.g., in terms of advocacy) capacity of targeted CSOs/clusters to enable them to engage with government on reforms? What did not work? • To what extent did the STAR result in increased capacity of targeted CSOs/clusters? • Are at least three SACE anchor partners ready to receive direct funding from the USAID? If partners are not ready to receive direct funding, are they likely to be ready by the end of calendar year 2017? The SACE initiative envisioned providing a mix of 1.0 capabilities, 2 which emphasize strengthening CSO management structures, and 2.0 capabilities, 3 which emphasize strong system-level engagement. The approach depends on the acquisition of specialized skills with regards to the application of advocacy tools, such as ecosystems mapping, adaptive learning, feedback loops, and collective impact. As the report illustrated earlier, organizations are categorized into anchors and clusters with, sometimes, overlapping roles (see response to evaluation question 1). Methodologically, the STAR system is comprised of several 2 Internal management, administrative systems and planning, policies, and procedures. 3 Social capital development, ecosystem perspective, learning, and adaptation. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 20 components: The STAR Assessment (with eight Performance Drivers),4 the STARNET (mapping advocacy clusters), the STAR Feedback, and the Annual Progress Index, as noted above. FINDINGS Improved financial and technical (advocacy) capacity. Overall, SACE’s assistance to CSOs has served to strengthen their institutional capacity, networking, and ability to engage policy-makers. SACE anchor CSOs—chosen in a rigorous selection process—and cluster members have received extensive training in both 1.0 capacity and 2.0 capacity skills since 2015. Initial capabilities and achieved competencies have varied greatly. While some CSOs have shown considerable capability improvements, both programmatically and operationally, others have not improved as much. Some SACE anchors, namely Human Development Initiatives (HDI) and Youth Alive Foundation (YAF), have expanded their competencies considerably, to the point of becoming eligible for and have received grants from other donors, such as the United Kingdom’s Department for International Development (DFID) and the MacArthur Foundation. According to Data Quality Assessments (DQA) reports, all SACE indicators comply with the data quality standards of validity, reliability, timeliness, precision, and integrity, with SACE staff verifying and validating all CSO-submitted data. To address gaps in 1.0 capabilities, especially for lagging cluster CSOs, SACE has engaged a USAID partner, Widows and Orphans Empowerment (WEWE), to assist in furthering the development of their management capabilities. Some CSO key informants indicated their SACE experience had been very good. One described the SACE program as unique: “We learned the 2.0 capabilities, they invited experienced people from CSOs and important public officials to coach us, share their experiences … it is rewarding …we’ve had many organizations in the past trying to support us, but SACE stands out because it is highly organized.” A particularly positive assessment claimed the “SACE program is excellent, the training itself is excellent … before we joined SACE, we did not have a single [administrative] policy.” Frequently, however, key informants showed concern with delays in the release of activity grants, a recurring problem compromising their ability to comply with deliverables. As a key informant related, those delays “slow us down ...we have to spend, and they reimburse us, and it takes time … I think they [SACE] are overburdened with so much work.” Improved coordination and lessons learned. Almost all CSO key informants expressed that their organizations had improved their advocacy and networking skills. Some highlighted the need for more 1.0 capacity training and, while they emphasized that they had been involved in advocacy activities for some time, they went out of their way to say the 2.0 capacity training provided by SACE is unique, effective, and impactful. The SACE has significantly built the collaborative capacities of CSOs and this has led to policy changes at all levels. The STAR approach has an embedded learning and evaluation approach that assesses organizations in relation to the larger ecosystem to bring about change. Regular data collection (e.g., through Survey Monkey), outcome harvesting, and network analysis are done with the keen and close participation of other CSOs. As the SACE’s 2016 annual report related, all SACE-engaged CSOs reported 4 Adaptive strategy and tactics, stakeholder engagement, monitoring and evaluation, member and partner development, knowledge exchange, alliance building, innovation, and public awareness. 21 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV a net change improvement in organizational capacity. This was coupled with a 39 percent improvement in the STAR index, slightly below the 42 percent target. Overall, national clusters’ net improvement ranged from a low of 29 percent for the Nigerian Association of the Blind to a high of 98 percent for the Center for Environmental Education and Development. The SACE hosts Annual Learning Summits (ALSs) to share lessons and strengthen CSOs’ capacities. “Before [SACE],” one key informant observed, “we used to be lone rangers, we thought we were big … we learnt from the 2.0 trainings the benefit of collective capacity … now if we want to make a press release we make people provide inputs.” On the other hand, SACE managerial practices often interfere with the smooth functioning of relationships and activities. In a key informant’s words, “it is frustrating, we develop a work plan, sometimes with SACE staff and our anchor … still the work plan is not approved.” Despite such problems, most key informants are pleased with how their CSO’s coordination skills have been upgraded and their relationships enhanced, with one key informant describing her CSO association with the anchor as “a good relationship … the anchor communicates well, we hold press [conferences] and issue releases [jointly].” Increased networking, commitment, and CSO passion. Generally, there has been an increase in CSO networking, coordination, and advocacy regarding accountability issues. Four anchors in particular— Centre for Information Technology and Development (CITAD), HDI, YAF, and Civil Society Legislative Advocacy Centre (CISLAC)—reported the most pronounced achievements in these areas. Interestingly, one of the major drivers for cluster involvement in networking activities was said to be CSO leaders’ long￾standing passion and commitment to specific policy reform objectives. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 22 As the ET frequently heard during the evaluation field work, there is concern about the unavailability of funds for individual CSOs to support their basic operations. Anchor CSOs receive and manage SACE funds to implement clusters’ work plans. Despite some issues associated with this process, anchor/cluster relationships were generally described as cordial and productive. The closeness of the relationship was avowed by two key informants. According to the first, “we come together to do advocacy … identify what we are going to do.” Similar feelings were conveyed by the second: “There is a group relationship, a lot of interactions … we don’t differentiate, we work as a team … the relationship helps us to leverage from each other and it helps us work with a strong voice.” Relationship building and cooperation between CSOs and policy-makers. The SACE’s support has served to bring together partners working on similar issues while sharpening their united voices. CSO key informants unanimously described favorably the capacity-building tools the program offers to strengthen CSO/GON partnerships. Cluster Coaching received specific praise, with respondents describing it as a great mechanism for CSOs to share ideas and strategies, and to learn from each other. Some policy-makers recalled invitations to SACE events as excellent venues to promote policy reforms, since these events afford an opportunity to become better informed about specific issues. Some legislators consider CSOs as especially strong allies when advocating for democratic reforms. By contrast, before TABLE 1. NETWORK IMPROVEMENT COMPARISON 2015 AND 2016 National Cluster and Anchor 2015 2016 Total Actors with Change NI-1 Total New Links NI-2 Clique Count Change Total Actors with Change NI-1 Total New Links NI-2 Clique Count Change Accountability in Education (HDI) 13 44 0 4 81 4 Accountability in Extractives (CISLAC) 12 70 13 6 421 71 Open Budgeting for Health MNCH (CENSOJ) 11 112 9 2 51 9 Quality of Health Services MNCH (HERFON) 16 20 1 3 26 5 Representation of Women in Agriculture (WARD-C) 16 27 11 5 106 14 Social Inclusion in Basic Education (JONAPWD) 6 15 0 2 35 5 Social Inclusion in Health Services (DRAC) 14 11 1 3 30 9 23 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV being exposed to the SACE approach, government officials were generally unresponsive to CSO appeals. The utilization of the STAR procedures has contributed to improved relationships between CSO representatives and public officials, as the latter now receive more refined and specific policy proposals. Several CSO key informants cited the ALSs as particularly effective events to develop institutional linkages and learn about effective advocacy practices. According to key informants, the STARNET “help[s] us locate others doing something similar ... what worked for us is delegation of responsibility … we are working together, our voice is louder.” Enhanced knowledge sharing for better advocacy outcomes. SACE-supported CSOs are generally better informed and more involved in democratic reform processes, as they demand more transparency and accountability. In the education sector, as this report indicated elsewhere, a major advocacy policy achievement was urging two state governments to provide Universal Basic Education (UBE) matching funds to access federally-provided grants. In the extractives sector, CSOs are regularly enhancing public awareness campaigns through radio and TV programs, such as “Extractives Revenue and You” and “NigeriaInfo 95.1”. Another major achievement was the drafting and dissemination of the guidelines embedded in “A Road Map on the Extractives Sector” proposal. Still, another substantive advocacy achievement was realized in the health sector when, as a result of SACE interventions, CSOs sustained their advocacy momentum by convincing the Kaduna authorities to include a family planning line item in the state budget for the first time. The state government committed to provide N100 million annually, so far having released in 2017 N45 million. This success story of policy reform can partly be attributed to improved CSO strategic communications, as the “STARNET help[s] us unite with each other and pass messages,” and because, as pointed out by another CSO key informant, the “advocacy process has really improved.” Another CSO key informant described the process as “fantastic, great exchange of knowledge.” Improved mentorship and coaching. Customized capacity building workshops were initially provided by sector experts and specialists, but value for money and other considerations forced SACE to change course and increasingly depend on its own staff to provide training and other support to anchors/clusters. By all appearances, the decision was justified and adequate, as it appears to be working well. Current support areas include STARNET relationship mapping, collaborative practices, gender, and social inclusion. As an anchor key informant noted, while expressing her overall satisfaction, “we are able to build the capacity of our target 85 organizations and 75 percent have functional policies.” One related complaint was that the “content of the workshops and the number of days to cover them … is too short.” Diversification of CSO strategies and tactics. Policy-makers have benefitted from SACE-supported training as they have become better informed about issues under their jurisdiction, while CSOs have used those opportunities to advocate for policy changes. One of the most effective strategies to make these exchanges between CSOs/policy-makers come about has been to engage influencers to schedule meetings. For example, in the Bauchi House of Assembly, the Secretary of the Education Committee has facilitated meetings between the Bauchi Coalition for Improvement of Public Expenditure Management (BACIPEM) and the committee chairman. The group has also engaged the Ministry of Education, the Ministry of Budget and Economic Development, and the State Universal Basic Education Board (SUBEB) in roundtable discussions. HERFON’s strategy in Kaduna and Katsina was to identify existing Maternal, Newborn, and Child Health (MNCH) CSOs that had benefitted from previous opportunities of capacity building through DFID’s State Accountability and Voice Initiative (SAVI) program and bring them together to advocate for budget allocations and improved health service delivery, with this process also engaging the Ward Development Committee (WDCs) at the Local Government Area (LGA) level. Enough is Enough (EiE) USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 24 has used the annual Social Media Week event to amplify the accountability and good governance message through hash tags, such as #openNASS and #OfficeofTheCitizen. Millions of Twitter followers have been reached. Some anchor CSOs, including CISLAC, HDI, CENSOJ, and Women Advocates Research and Documentation Centre (WARDC), have done likewise, but they continue relying on more conventional outreach methods, such as press briefings, press releases, media round tables, and conferences. The ET observed an HDI SACE-sponsored workshop on education accountability that was to be followed by a press conference to review results. As a cluster member indicated, the “SACE brought to us completely new techniques… we have been doing our own thing for almost 20 years, we never knew there were better ways of doing things to achieve results faster.” Engagement of marginalized groups. While in one way or another, all SACE-related CSOs are engaged with marginalized groups (i.e., women, youth, and the disabled), aside from a meeting with JONAPWD in Abuja, during the field work the ET only contacted a JONAPWD-affiliated cluster CSO in Ibadan (Nigerian Association of People with Intellectual Disability). This CSO has participated in several SACE-sponsored events. As a result of its advocacy, the Oyo state government issued a circular by which it established that all basic education for mentally disabled children must be paid by the state, a right extended as well to all disabled primary and secondary school children, regardless of disability type. Proficiency in using the STARNET. The targets of the program as to improved networking have been surpassed by national level CSOs largely thanks to their proficiency in using the STARNET, with Niger Delta CSOs gradually closing the gap. Nearly all CSOs that were assessed with the Annual Learning Index had achieved net positive change, defined as the ability to establish new partnerships and spread ideas across Nigeria’s civil society. CSO respondents reported advocacy capability improvements with regards to networking, information sharing and engaging capacities. Particularly useful, aside from the STAR advocacy training, has been the STARNET collaborative advocacy instrument, as it leads to the “pooling together of organizations.” CSOs are now more interconnected and articulate their advocacy more effectively and efficiently, as they act as a team leveraging each other’s capabilities and influence. One respondent claims that the “STARNET is like [an] electronic complementary [business] card, we use it to reach out and link with our colleagues … the quality of our advocacy has been enriched… our advocacy is evidence-based … we also speak with one voice.” Challenges have mostly arisen regarding the extent of attribution, as multiple actors could lay at least a partial claim for successful reforms. One key informant, however, was regretful about his CSO failure to take full advantage of the STARNET, but was committed to do better in the future: “We did not make very good use of the STARNET, its part of our weakness, we have a lot of people that we have mapped–we will improve.” 25 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Working with media. The activities that aimed at working with the traditional media have been successful, but their targets were not achieved as they were originally proposed, due to the withdrawal from the SACE consortium of BBC Media Trust. The program has been able to innovate and incorporate widespread use of social media, which is a tribute to the adaptability of the SACE approach, including adaptive learning. Staffing issues. The SACE program’s effectiveness appears to be undermined by constant staff attrition. An indication of the magnitude of the problem is that, during the last quarter of 2016, the headquarters for the program in Abuja had an M&E Specialist position vacant for the duration of the quarter. Given that situation, SACE’s Senior Public Awareness and Communications Advisor filled that role on an interim basis, and oversaw processing and analyzing the data that anchor CSOs submitted. Readiness to receive funding. This determination rests on the ability of anchor CSOs to achieve certain capabilities and, secondarily, on their history as recipient of donor grants. Based on these criteria, the ET determined that four CSO anchors-CISLAC, CENSOJ, HDI, and YAF–currently qualify to receive and manage USAID grants, given their comparative achievements on SACE-defined capacity improvement thresholds on 1.0 and 2.0 capabilities. CISLAC and CENSOJ have been prior recipients of donor funds, whereas this sustainability threshold was just crossed by HDI and YAF. While some other anchors￾including Africa Network for Environment and Economic Justice (ANEEJ), Lite-Africa, and SDIC-have received grants on their own from several local and international donor organizations, they have not done as well regarding some of the SACE-related capability improvements. TABLE 2. NETWORK IMPROVEMENT Area CSO Baseline 2014 2016 (total new links) Accountability in Education Human Development Initiatives (HDI) 17 35 Centre for Information Technology and Development (CITAD)* NA 54* *started April 2016 Accountability in Extractives Civil Society and Legislative Advocacy Centre (CISLAC) 6 202 Open Budgeting for Health (MNCH) Centre for Social Justice (CENSOJ) 5 44 Quality of Health Services (MNCH) Health Reform Foundation Nigeria (HERFON￾FCT) 6 20 Social Inclusion in Basic Education Joint National Association of Persons with Disabilities 4 29 Social Inclusion in Health Services Women Advocate Research and Documentation Centre (WARD-C) 12 83 Disability Rights Advocacy Center NA 27 USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 26 CONCLUSIONS CSOs were generally found to be committed to multi-stakeholder engagements and have improved their capacity to reach to other cluster members when they sought information and advocacy expertise. CSOs have undoubtedly increased their ability to engage in policy advocacy. While 1.0 capacity has improved mostly among anchors, most CSOs—whether anchor or cluster—have benefited from 2.0 advocacy￾enhancing training. Also, even though SACE has managed to exponentially increase the innovative use of social media, there still seems to be room for expanding its presence in the conventional media. EVALUATION QUESTION 3 To what extent did SACE partners successfully influence government policy development and implementation regarding vulnerable populations, such as women, youth, and persons with disabilities? The SACE performance reports show progress with respect to the achievement of outcomes and impact. According to the Policy Tracker Table in the third quarter report for April 1 to June 30, 2017, a growing number of bills and laws, as well as policy and administrative reforms, that could be largely attributed to SACE-related cluster initiatives have been achieved. These include the recent introduction of a bill on youth empowerment, which has been promoted by the Youth Alive Foundation at the Akwa Ibom State House of Assembly, and the adoption of a state disability bill in Akwa Ibom, which has been promoted by JONAPWD. In Katsina, a “Primary Health Care under One Roof” reform was approved by the State House of Assembly, and is currently waiting for the governor’s signature. Youth Alive Foundation cluster members advocate at the Akwa Ibom State House of Assembly. Several policy and administrative change proposals have been either introduced or are being implemented in the FCT and various states. These include a draft FCT policy on inclusive education, promoted by the JONAPWD-led cluster, and a Yobe State House of Assembly proposal to establish an ad-hoc education committee, advocated by CITAD. Table 3 below summarizes the achievements since the inception of the 27 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV program and indicates that a total of 75 policy initiatives that were advocated for SACE-related clusters have been carried out. These include advocacy in favor of 26 laws and bills, 24 policy reforms, and revisions of 25 administrative procedures. TABLE 3. SACE POLICY REFORM ACHIEVEMENTS SINCE THE INCEPTION OF THE PROGRAM The pace of accomplishments has been such that the USAID and SACE have agreed to revise the PMP of the activity to add a new sub-intermediate result to track program outcomes under intermediate result (IR) 2. Such achievements (also partially captured through the KIIs) were made possible in whole or in part through SACE-inspired advocacy tactics. They are associated with the development and implementation of successful reforms, regardless of scope or setting, and flow directly from the SACE advocacy playbook. They are described below and illustrated with examples that key informants provided to the ET. FINDINGS Speaking with one voice. Key informants universally agree advocacy initiatives succeed when CSOs present a united front when they lobby for a given reform. A common demand carries more weight; particularly if it entails reaching for a clearly defined policy target, instead of CSOs competing with each other and presenting poorly articulated proposals. Careful attention to the objective and the related course of action eases the policy dialogue by focusing attention on what is at stake, while eliminating distractions. Paraphrasing a FGD participant, advocacy with a single voice eliminates the level of abstraction that often separates poorly articulated advocacy from effective results-oriented initiatives. As an anchor CSO representative stated, advocacy success could be achieved when “you are now one, one of you could represent the common position.” This tactic was a constant across all the site the ET visited and is associated with nearly all the reported substantive outcomes of the SACE partner initiative. Forging cooperative relationships. Conflictive CSO-government relationships often delay or prevent advocacy campaigns from succeeding. CSOs invariably cited cooperative relationships as instrumental for the successful implementation of policy reforms; policy-makers must start viewing CSOs as “supportive rather than destructive.” While positive interactions usually take time to develop, they are likely to be fruitful when they are viewed through a win-win prism. CSOs and policy-makers alike have identified the mutual advantages to be gained by developing cooperative relationships. In this connection, the budget tracking tool has proved to be particularly useful. While CSOs gain ground implementing reform agendas, government and policy-makers also benefit. Civil society input allows government agencies to better identify constituencies’ needs, while analyses and recommendations of the CSOs’ sector contribute to the formulation of better procedures that could enhance service delivery. In some cases, CSO advocacy initiatives have contributed to the reversal of ill-advised programmatic decisions. National State Niger Delta Total Laws and Bills 11 8 7 26 Policies 9 11 4 24 Administrative Procedures 10 6 9 25 Total 30 25 20 75 USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 28 This was the case in Kaduna, where initial plans to build a hospital for government employees were discarded in favor of reallocating the related construction budget towards the rehabilitation of 255 primary health care delivery posts. The role of CSOs in this and other reform initiatives for the delivery of health services was alluded to in very favorable terms by the State Commissioner of Health, who noted that, “in most of what we do, we get [CSOs] involved … we have been working closely together … we see each other as partners.” The Secretary to the Education Committee of the State House of Assembly in Bauchi echoed those views by emphasizing gains that accrue to government through its cooperation with CSOs. They contribute, he noted, to considerable policy analysis improvements, including identification of priorities and policy guidelines, and often provide momentum for allocation of additional program resources to implement the reform in question. Come prepared and be helpful. Reform advocacy usually involves educating policy-makers – and the public at large. To succeed, such efforts must be knowledge-based, and rely on solid background research that should include specific reform targets and a step-by-step approach to achieve the desired reform. Such proposals must be informed regarding anticipated institutional, legal, personnel, and procedural requirements. They must be sensitive to budgetary implications, while at the same time clearly identifying the reform objective and policy-making target. Reform proposals must also be realistic and presented to policy-makers and government officials as achievable road maps, consistent with their mandates. This should involve the development of strategic advocacy plans incorporating solid argumentation and data. By sharing information and building rapport, “government officials come to appreciate what they really do not know,” and increasingly value what CSOs have to offer as they are better attuned to community needs. Gradually, mistrust is replaced by a sense of cooperation. A good example can be found in Bauchi, where, through the auspices of CITAD, significant CSO￾government cooperation is helping guide educational reforms. The experience is similar, but on a much smaller scale in Lagos, where a grassroots cluster CSO is advocating for disadvantaged children. The Out of School Children Foundation (OSCF) realized its voice could be heard when members of the organization showed up armed with educational information at a town hall meeting organized by HDI, which is the OSCF anchor CSO. Due to the Foundation’s advocacy, 50 children were able to return to the classroom, as the local government decided to reopen several abandoned schools. Additionally, six plots of land were set aside to build new schools. From the perspective of an anchor representative who was involved in a significant health care delivery reform, policy-makers and government officials have become so dependent on the information and ideas of CSOs that they often seek CSOs’ input when considering policy prescriptions. Advocacy for reform also hinges on discarding ineffective advocacy efforts. As a cluster organization representative advocating for improved conditions for women farmers in Osun state said, “We used to go to the Ministry with just a placard. Now we go and sit with them and make them write what they promise in case there is a change in government.” Through its advocacy, this CSO has convinced the state government to provide poor rural women with milling machines, as well as 250 acres of land, to settle destitute women farmers. A request for an additional 250 acres in a safe district to resettle poor women farmers was made and is pending. The role of enablers. A common thread running behind CSOs that have successfully advocated for policy reforms is their ability to enlist support from previously unrecognized stakeholders with the potential to have a major bearing on outcomes. SACE training and its mapping procedures have allowed effective CSO advocates to recognize the previously ignored role enablers exert in the reform process. One key informant, with more than 20 years of experience leading national advocacy initiatives on behalf 29 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV of the disabled, recently realized the marked influence that people in the “corridors of power,” including leading community figures and former government officials, have on policy-making. Partly through their auspices, the Nigerian Association of People with Intellectual Disability (NAPID), a cluster CSO of JONAPWD based in Ibadan, was able to convince government officials to expand access to basic education—a right enshrined in Nigeria’s constitution—to include disabled children. As a result, some schools are being remodeled to add necessary accommodations. A key informant from the CENSOJ-led cluster in Kaduna echoed this perspective while describing a plan for advocacy regarding the need for reform in access to basic primary health care in the state; the thoughts were “about who could help you?” By enlisting the support of diverse stakeholders, a broad CSO alliance managed to advocate for and help achieve some rather consequential health care reforms. Complementary advocacy initiatives at the national, state, and local levels. In many instances, close collaboration among clusters advocating for common agendas at different levels of government facilitates policy reforms. The ET repeatedly came across such modalities. Educational reform efforts in Lagos and in northeastern states were functionally related to advocacy efforts anchor CSOs conducted in the FCT, as were initiatives under way to expand the provision of essential primary health care and maternal/child health care services in Kaduna and elsewhere. Advocacy efforts in favor of the disabled in Oyo state have benefited from parallel efforts of like-minded CSOs seeking to expand accommodations and services for this vulnerable population across Nigeria. Expanding educational opportunities in the states of Bauchi and Lagos are supported by the ongoing work of advocacy CSOs that are engaged in the promotion of more transparent and accountable government, whether in the FCT or at the state level. The cross fertilization arising from thematic commonalities, such as more transparent budgetary procedures, goes even further, when they involve strategic alliances between clusters, which focus on common service delivery areas from different perspectives. An example of this is the strategic alliance between the HERFON and CENSOJ anchors, whose primary objective is improved delivery of basic health care. The former focuses on service delivery, while the latter addresses the health ramifications of budgetary allocations. CONCLUSIONS Several conclusions can be drawn from the ET’s findings regarding evaluation question three. The SACE approach has proven to be effective in influencing government policy development and implementation regarding vulnerable populations. Selected regional initiatives have proved their worth in addressing children’s basic educational needs (in Bauchi, Lagos), as well as in addressing the most pressing health challenges threatening children and their mothers (access to essential maternal/child health primary services in Kaduna), and expanding educational opportunities and access to accommodations for the disabled (in the FCT and Ibadan). SACE cluster partners have successfully internalized 2.0 advocacy capabilities and proven they can use them effectively. While these results are only generalizable to the selected Nigerian regions the ET visited, they and their promising outcomes have the potential to be replicated in other regions/sectors of the country by implementing the SACE advocacy capacity-building methodology. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 30 The positive results of applying the SACE advocacy approach across very different action areas suggest the methodology could be replicated with equally successful results in other national priority sectors, such as employment generation, housing, and water availability. EVALUATION QUESTION 4 Did SACE activities for public awareness contribute to strengthening public discourse and support for key democratic governance issues? • In particular, what was the influence on viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Mindset Series, Twitter, live streaming, and films? FINDINGS Leveraging the various SACE-provided capacity-building procedures, CSOs are making significant inroads furthering public awareness in favor of democratic reforms in their respective issue areas. Success has been achieved both at the national and local levels. There is evidence of effective partnerships between SACE CSOs and media to advance the advocacy agenda. In many instances, CSO clusters include media organizations. Not surprisingly, the appeal of traditional and social media varies by type of audience. Despite this difference, the strategic use of both types of media has been an effective means of broadcasting the governance issues that SACE partners championed. Use of traditional media. SACE partners have made good use of traditional media in their campaigns for the advocacy public awareness. There is a strong print, radio, and TV SACE presence at the national and local levels. Radio programs for audience participation, for example, serve as effective vehicles to advance open budget advocacy initiatives, as Social Action in Port Harcourt, Rivers State, reported. Its members are frequently invited by a Port Harcourt FM radio station to discuss issues pertaining to the state budget. In a recent program, a recording of a statement the current governor made when he was running for office was played, to remind him that he had pledged, if elected, to post the state budget on the state’s official website. A representative of the Citizens Centre for Integrated Development and Social Rights (CCIDESOR), in Owerri, offered another testimonial. The key informant recalled a question a woman raised in a recent phone-in radio program. “Could the governor tell us,” she asked, what “all these things [referring to the many construction projects under way] he is doing … are costing the state?” This attests to increased public awareness about the need for budget transparency and the willingness of citizens to request from the government increased transparency about the state budget. Similarly, the involvement in some clusters of print media journalists assures SACE partners of advocacy support as they engage policy-makers on key democratic governance issues on a pro-bono basis. These journalists, by regularly publishing newspaper feature articles, help drive home pressing advocacy issues. Use of social media. SACE partner CSOs have been trained on the use of social media and, in the process, have become adept at reaching young audiences. Thus, the increased social media presence of SACE CSOs resonates with the young—as well as the elites—while popularizing and extending the reach of advocacy messages rather quickly. SACE’s 2016 second quarter report alludes to a significant increase in public engagement through social media, including providing support for the successful hosting of the second annual “Let’s Talk Governance” sessions at Social Media Week Lagos. Aside from the more than 300 participants in attendance, the event had a Twitter reach of nearly 10 million viewers. More recent data from SACE’s third quarter report for 2017 (no page number) are: “The project’s 2.0 Mindset series in May reached 908,300 accounts and received 7,042,130 impressions.” 31 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV A SACE partner makes her pitch to a Social Media Week participant. Other examples of the social media impact associated with SACE can be culled from the presentation the program’s Chief of Party (COP) made to the ET. According to the statistics in the presentation, the Twitter exposure of the various SACE-supported CSO campaigns currently stands at more than 46 million. The reach on GovTalkNG, another effective outreach initiative, exceeded 50 million views in 2017 alone. Other popular social media SACE campaigns include are “Ten percent for youth,” “Make every kobo count,” and “Women lead agriculture.” Targeting increased transparency, accountability, and responsiveness in government institutions with such Twitter hashtags as #makeeverykobocount and #makethebudgetopen has expanded SACE partners’ advocacy reach. This innovative advocacy approach enhances the transformational influence of the campaigns for public awareness by making them akin to social movements. Through one of these campaigns, Akwa Ibom state was persuaded to upload its budget on the state website, now for two consecutive years (2016 and 2017). Similarly, the Delta state budget was made available online for the first time in 2016, thanks to a broad-based social media advocacy campaign that was led by SACE CSO partners. Just in August 2017, to cite another instance, the Imo state government was prompted to share its 2018 budget call circular with CCIDESOR and to publish it in several newspapers. Use of film. The SACE program has promoted the production and distribution of films to further the development and implementation of key democratic reforms as part of its public awareness efforts. This support to the film industry has been channeled mainly through the Integrity Film Awards, sponsored by USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 32 the Public Private Development Centre (PPDC)–Homevida–with the intent of disseminating good governance messages. One initiative promotes an annual competition for best feature film addressing these topics, while the second is intended to encourage young script-writers to address socially worthy good governance topics. The winners are supported to turn their scripts into short films. Feature films have met with relative success. In 2015, 22 feature length films competed for the USAID SACE’s Transparency and Accountability Film Prize. The winner, “Champions of Our Time,” as of October 2017, had recorded over 9,000 hits on YouTube and more than 10 million eyeballs (http://bit.ly/2kqEqbm). The 2016 competition attracted 18 films and documentaries, the feature “Oloibiri” being declared the winner. So far, it has been viewed 300,000 times on YouTube, with an additional audience of five million viewers in film festivals and movie theaters (http://bit.ly/2kqqYnP). Furthermore, the documentary category winner, “Nowhere to Run,” has been viewed over 4,000 times online and received 50,000 eyeballs (http://bit.ly/2fVvb1F). Regarding the short film category, the first SACE-supported short film on the Homevida stable, “In Apathy,” has just been released on YouTube (October 2017) and therefore it is premature to assess its potential reach and impact. Anyhow, based on the remarkable success of other Homevida short films, it is likely to have a considerable audience. A just released independent evaluation determined that an earlier Homevida short film, the 2014 “Enitan,” garnered thousands of hits on YouTube and over six million views on Africa Magic. Based on this history, the ET is persuaded that “In Apathy”, which has already been submitted to the “AFRIREVO” film festival and the Africa Movie Channel, will reach a large audience, likely in the millions. Some SACE partners have likewise produced and used short films (dramas) to further reinforce messages in their campaigns for public awareness. A good example is the short video the Africa Network for Environment and Economic Justice (ANEEJ) produced under the title “Busted,” available online on the ANEEJ website and on various social media sites, as well as on CD. According to a post on the ANEEJ website, the video was acclaimed at the Homevida-organized 2016 seminar on script writing for young dramatists, producers, and directors. The video continues to attract viewers. At the time of this evaluation, “Busted” had been viewed 277 times on the ANEEJ website, 121 on YouTube, 272 on Facebook, and 124 on Twitter. While the audience reach of some of these films has been significant, the potential for Homevida films to influence even greater audiences is considerable to the extent more resources are made available. One thing is clear: quality films are expensive to produce and promote. In its interaction with the Homevida Chairman, the ET was made aware budgetary constraints are a major challenge. Despite this challenge, the 2017 PPDC-commissioned Homevida Impact Assessment report found that since its 2010 inception, Homevida has influenced the content of over 107 Nollywood feature length films–many reaching Africa￾wide audiences-18 of them receiving Homevida awards. Moreover, the report notes that Nollywood regards the Homevida short script competition and associated training as “spurring experience on young and aspiring film makers.” It further found that Homevida contributions are well regarded in the film industry, Nollywood professionals increasingly relying on the Homevida competition criteria as templates for their own films. One result is that the dominant themes in many Nollywood productions are no longer predominantly fetish, as more and more Nollywood films embrace the development focus. Additional financial support could only multiply many-fold the future impact of good governance-oriented themes in the Nigerian film industry, given its potential to influence millions of moviegoers. Use of other modes for public awareness. The SACE program is working with several partners with broad footprints across the country to conduct activities for public awareness through complementary 33 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV platforms, such as public dialogues, town hall meetings, and citizens/community score cards. Thus far, 54 activities have been sponsored across the six geopolitical zones of the country. Concrete metrics regarding attendance are not available, since they are not reported. The outcomes of many of these activities are often disseminated through the traditional and social media, as was the case, for example, with the results of the HDI-organized Lagos workshop on educational accountability the ET observed. In the Niger Delta, to cite another instance, the results that were obtained through Citizen’s Report Cards (CRC) have been widely debated thanks to their mass media dissemination. Influence of SACE activities for the advocacy of public awareness on policy-makers. The style and delivery modes of the messages that SACE CSOs implemented to enhance public awareness advocating for key democratic reforms are becoming more effective in attracting the attention of policy￾makers. As they do so, the latter are becoming increasingly willing to engage and partner with CSOs in pursuit of democratic dividends. SACE partners’ “cooperative” advocacy style is gradually changing how policy-makers perceive those CSOs. Rather than as adversaries, mere critics, or allies of their political opponents, policy-makers are starting to regard CSOs as useful partners. This evolution has been propelled by the SACE 2.0 capacity training, as CSOs have become more analytical and adept at using reliable evidence when they formulate, present, and define reform proposals. The use of the full spectra of instruments for public awareness, coupled with improved CSO cluster advocacy skills, are strengthening civic engagement while inducing policy-makers and state institutions to take note and gradually embrace better governance procedures. Results from these positive interfaces between CSOs and state actors are likely to become more fruitful in the medium- to long-term. In many instances, crucial policy dialogues and remedial government actions are already beginning to take shape. In the Niger Delta, a good example is the partnership between Edo state Co-Operative Farmers Agency Limited (ESCFAL) and the Edo State Ministry of Agriculture. They are in the process of jointly drafting the state agriculture policy document in response to a directive issued by the state governor. The same applies with respect to ongoing legislative action to allocate 10 percent of Akwa Ibom State’s “extra” 13 percent oil-derived revenue for youth empowerment programs. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 34 Youth in Akwa Ibom advocate for empowerment programs. Just as noteworthy has been the ability of CSO clusters to rely on the STARNET and its mapping features to polish the approach of public awareness campaigns. CSOs have learned to take advantage of expertise that is not readily available to them, but that is accessible through other partner CSOs, to improve the content and design of public awareness messages. CSOs with good analytical skills have lent their support to others by providing technical reviews of budget data from targeted states, while others have contributed their legislative know-how to assess the nature of proposed legal and regulatory reforms. CENSOJ, for example, has provided training and other support to six other clusters regarding the Medium￾Term Expenditure Framework (MTEF) process, and other clusters often rely on CENSOJ’s Capital Budget Pullout tool. Consequently, CSOs are producing superior quality messages that help educate the public and support the policy reform process. CONCLUSIONS SACE activities for public awareness are strengthening the nature of public discourse and support for key democratic governance reforms. As citizens are becoming more aware of their right to access public information, including financial data, some government officials are evidencing a growing willingness to comply with civil society demands. These changes are occurring as better-informed citizens, increasingly cognizant of their rights, and encouraged by the dissemination of good governance messages, apply growing pressure on public officials to be more transparent and accountable. 35 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV The SACE program is indeed helping strengthen civil society’s ability to rally the voices of citizens towards better implementation of key governance and policy reforms at the national, state, and local levels. The SACE program’s use of traditional and social media for policy reform advocacy purposes has been effective and, in many respects, innovative. SACE can be specifically credited with some notable achievements using social media. Its emphasis on alternative means of mass communication—that are not conventionally associated with governance reforms, but that appeal to the young, such as popular films—is so far effective, with great promise for far-reaching impact. In their totality, these public awareness activities are furthering national and local dialogues about governance reforms. As a result, they are contributing to the consolidation of a critical mass of better informed citizens hungering for and increasingly willing to call for improved governance. The experience in the Niger Delta demonstrates that the inclusion of traditional and social media CSOs in clusters greatly enhances their media visibility since CSO members are committed to making advocacy issues part of the broader discourse agenda. Consequently, they contribute greatly to outcome harvesting. The SACE program should, therefore, encourage all clusters to map even more social actors and to partner more closely with them. These efforts could be further enhanced by a more targeted utilization of the so far neglected traditional media venues to convey governance reform messages to selected audiences according to regional audience segmentation criteria, such as language and relative penetration by type of media. SACE, for instance, could consider supporting the production and broadcast of reform advocacy messages as radio dramas in Hausa in the northern states given high radio listenership rates. In the South East and South zones, town criers may prove to be effective in conveying reform messages at the grass root level in communities that are relatively isolated from other communication venues. EVALUATION QUESTION 5 How did SACE’s support for Niger Delta implementing partners contribute to their advocacy for economic reforms/growth in that region? The SACE program’s support to CSOs’ advocacy for good governance and economic reform in the Niger Delta builds on several other similar program initiatives earlier undertaken by its partners in the region, especially the anchor CSOs. This had positive implications for the quick takeoff and ramping-up of activities, despite initial start-up delays. On the one hand, it provided in-roads for early engagement with government actors and other Niger Delta Institutions (NDIs) by the different clusters, as most of them still operate in already familiar terrains. On the other hand, it meant having to now deal with an institutional mindset that was not the most welcoming of CSO activities. The SACE program sought to strengthen the operational capacity of its partners in the Niger Delta as well as their advocacy and technical capacities to more effectively engage their governments and other NDIs. This has taken the form of direct training on issues such as open budget, budget tracking and analysis, advocacy capacity strengthening, use of social media, monitoring and evaluation, and partnership and network building. These have substantially improved the CSOs’ capacity to engage governance institutions in the region, with remarkable outcomes in some of the states. FINDINGS Some of the notable achievements towards economic reform and growth policy agenda in the Niger Delta are illustrated in the following. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 36 Advocacy-inspired innovations deepen impact. Following a series of capacity building and experience sharing sessions on innovative approaches to advocacy, many SACE anchor CSOs have had to rethink their individual theories of change, now adopting more cutting-edge approaches to deliver better outcomes. For example, the Social Development Integrated Centre (also known as Social Action, a CSO working on issues of social justice in the Niger Delta) under its SACE-supported Public Finance and Accountability desk, has had to adjust its theory of change and engagement approaches three times, adapting and incorporating new skills and tools for increased impact. The Executive Director noted that “with SACE, we have gradually moved from the old student-union-styled confrontational advocacy model to one that was essentially public embarrassment-oriented and now to constructive engagement through partnership and dialogue. This movement has been a difficult one as it has transformed us from our old crude ways to more cooperative partners, which is yielding better results.” The 2016 Root Change report showed a correlation between the outcomes of the cluster’s advocacy and their Net Change score. Not surprisingly, the cluster’s major improvements were in the STAR drivers of adaptive strategy and tactics, promotion of public awareness, and knowledge exchange. Voices for change were strengthened. The anchor-cluster model, including cross cluster collaborations—which STARNET reinforces—allows more organizations to identify with issues of common interest and to join forces to build a critical mass of advocates for reform. This is facilitated by the opportunity to draw on the comparative strength of cluster members to bolster the cluster’s campaign around a central theme. As a result, a symbiotic relationship grows as all cluster members, understanding their strength rests on their ability to work together, coalesce around a central advocacy issue. In doing this, individual organizations also gain by mobilizing the support of the entire cluster to propel its own specific advocacy agenda. For Gender and Development Action (GADA), a member of the African Network for Environment and Economic Justice (ANEEJ) cluster based in Port Harcourt, this has worked out well. In addition to being an active cluster member with capacity-building and networking benefits, GADA receives support from the cluster in its advocacy engagements with the Niger Delta Development Commission (NDDC). Taking advantage of the wide reach of the cluster membership, GADA is now able to more effectively oversee and engage with NDDC on NDDC projects and activities throughout the region. Ordinarily, GADA could not have done this on its own. Another cluster member, the Riverine Communities Health Organization (RIVCHO)—deploying SACE-provided advocacy capacity—also received support from other cluster members to successfully advocate for the construction of the Enerhen Road project in its home community. Essentially, all SACE partners the ET met in the Niger Delta feel SACE has given them stronger voices, making their contributions to targeted policy processes count. Also, increasing intra- and inter￾cluster collaboration on members-led advocacy themes is encouraging and inspiring more organizations to join the different clusters. In FY 2016, new CSOs and CBOs joined the cluster, with its membership reaching more than 370 organizations. Improved resource management through CSO-NDI partnerships. The SACE program’s support to Niger Delta partners has resulted in notable progress in facilitating partnerships between civil society actors and governance institutions. For instance, working under the aegis of the Niger Delta Budget Monitoring and Transparency Network, ANEEJ and its cluster members are working more closely with NDIs to achieve greater transparency, accountability, and responsiveness in the various state governments and NDIs, including the NDDC, the Ministry of Niger Delta Affairs, and the Oil and Gas Producing Area Development Commissions in Edo, Delta, Rivers and Cross River States. With the anchor CSO leading from behind, cluster members are encouraged to promote specific campaigns for reform advocacy within their respective domains. In Ondo state, for instance, in response to a targeted advocacy effort by the 37 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Community in Action Initiative (CAI), a member of the ANEEJ cluster, the State Ministry of Budget and Economic Planning published the State Development Coordination Policy Document to facilitate improved stakeholder coordination and oversight of development projects in the state. The Citizen’s Report Card (CRC) informed CAI’s advocacy, which is a product of cross cluster collaboration between the ANEEJ￾led and LITE Africa-led clusters. Similarly, in Edo state, advocacy efforts by ESCFAL and its cluster have resulted in policy discussions regarding the development of a state Agricultural Development Policy, based on the ANEEJ-LITE Africa CRC. The development is beginning to yield positive results as the state government recently set up a committee to work with the ESCFAL team to develop an inclusive state agricultural policy. Media partnerships improve advocacy outcomes. Part of the lessons from the early outcomes resulting from the Citizens Centre for Integrated Development and Social Rights (CCIDESOR) cluster’s advocacy for completion of abandoned projects in the state rests in their effective use of both traditional and social media to advance and popularize their advocacy agenda. To get the state government to revisit and complete more than 30 percent of the identified abandoned projects across six LGAs of the state, the cluster forged effective partnerships with both social media and traditional media practitioners to set the discourse agenda in the state and for mobilizing popular participation in advocacy campaigns for the completion of the projects. A top member of the State Executive specifically stated that were it not for CCIDESOR and its partners that made those abandoned projects an issue, those projects may not have been completed, by now. He added, “and you can see that the state is now making conscious efforts to ensure that it completes the projects that it has started.” The value of involving both traditional and social media as integral components of policy reform advocacy projects was also emphasized by Social Action’s open budget advocacy campaign in five states of the Niger Delta, and in getting the Akwa Ibom State House of Assembly to pass the fiscal responsibility law. Research-driven advocacy enhances impact. In addition to the abovementioned CRC-driven advocacy efforts of CAI and ESCFAL, other cluster evidence indicates that research-driven advocacy is very effective in persuading the government to act positively on targeted policy reforms. Dealing with the impact of the Imo State Abandoned Project report, a key policy-maker submitted that the government’s revisiting of this initiative and the completion of over 40 of 121 abandoned projects simply confirms the dictum “eternal vigilance as the pride of liberty.” This is also echoed across the different clusters, as the CRCs have engendered increased public discussion around targeted reform issues, thus compelling policy￾makers to act more responsively. Stronger, more enduring collaborative platforms. The anchor/cluster model’s innovative cluster coaching, capacity building, mentoring, networking, and leading from behind all combine to create an experience that effectively addresses the challenge of failed CSO coalitions and networks. One partner declared that “the cluster model is a fantastic model that is an antidote to the challenge faced in building networks and coalitions,” as it makes way for the pursuits of collective impact through shared vision and mission. Also, many Niger Delta clusters are comprised of organizations that hitherto operated in silos, each pursuing its own advocacy agenda, but now subsuming their campaign efforts under broader joint￾cluster themes. The SDIC Open Budget Cluster, which is made up of organizations across five Niger Delta states with equal passion towards the achievement of a common goal, is a typical example. Cluster membership sustained through anchors’ passion and transparency. The ET also sought to understand what drives the cluster members’ commitment to the SACE anchor advocacy goals. Most of them alluded to the open and transparent leadership of their anchor organizations. This is commendable USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 38 as SACE is consciously reawakening the need for organizational integrity, openness, and accountability, which has been the bane of several failed coalitions and networks. Government-CSO cooperative partnerships. The SACE-supported innovative approach to policy advocacy, which incorporates the peace-building “Do No Harm” principle, is another feature of the model that emerged as a major contributing factor to the improvement of government-civil society partnerships around key policy reform issues. A policy-maker in Imo state affirms “engagement with your cluster members has drastically changed the way we view CSOs. Before now we saw you people as instigated by the opposition but now you guys are more constructive, and we are even happy to open our doors and partner with you.” The cooperative partnership approach is further underscored by the collective impact maxim to not just pursue a common goal by all cluster members, but also to partner with government agencies and target NDIs on key items of the policy reform agenda. The improved capacity of Niger Delta partners to cooperate and coordinate efforts around economic reform advocacy issues and more constructively engage in policy advocacy with governments and NDIs are direct outcomes of SACE’s 2.0 capacity support, measured as a Net Change Index. The 2016 Root Change report shows that, in just one year, Niger Delta anchors and cluster members—comprising 32 different organizations—experienced an improvement of 53 percent on average in critical advocacy and partnership-building skills, over the baseline figure. This represents a 109 percent achievement of the annual target. Improved economic rights advocacy in Niger Delta communities. SACE cluster members are now deploying advocacy skills to more effectively engage NDIs in their communities and secure better economic privileges for their constituents. An example of this is a CCIDESOR cluster member, who utilized his improved advocacy skills to mobilize community members to demand an explanation from the Imo State Oil and Gas Producing Area Development Commission (IMSOPADEC) for the prolonged disruption of the community’s electricity supply. His query referenced an existing arrangement that entitles oil-producing communities in Imo state to the free supply of electricity paid for by NDDC. This advocacy effort, backed by the CCIDESOR cluster, led to a tripartite meeting involving community members, cluster representatives, and the NDDC. The result was the restoration of electricity service to the previously deprived communities. Strategic resource support repositioning partners. The SACE program’s funding and equipment support to Niger Delta anchors and clusters has effectively closed operational capacity gaps, making them better able to serve their membership needs. Increased functionality is in turn driving renewed membership commitments and CSO capacity to mobilize for advocacy on issues of common interest. For instance, responding to the reason for the near 100 percent participation by its cluster members following their meeting in Edo state, the President of ESCFAL affirmed that “before SACE, attendance at our monthly meeting had always been below 50 percent but, since SACE, the story has changed. Almost everyone now attends more punctually and very many of them owing us their membership dues now pay up more promptly. We now have more committed members and our meetings are more interesting and productive.” Also, ESCFAL and ANEEJ currently have more befitting office accommodations, standard operating systems and procedures, and a trained staff with specific competencies. All these contribute to the anchors’ improved ability to work with their clusters. CONCLUSIONS The SACE’s support for Niger Delta implementing partners has made remarkable contributions to the improvement of CSO advocacy capacities and strategic engagement with policy-makers and Niger Delta Institutions. Particularly salient have been the achievements in the open budget campaign, completion of 39 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV hitherto abandoned projects, and inclusive economic growth through statutory allocations to marginalized groups and sectors. ISSUES AND CHALLENGES Aside from some relatively minor issues associated with how the SACE program has been implemented thus far, the ET did not identify any specific issues or challenges which had not been already considered when the activity was initially launched that could derail its continued implementation, or that could interfere with a future USAID-funded civil society advocacy initiative. Thus, the initial preconditions underlying the SACE program when it began in 2014, as the evaluation SOW (pp. 4-5) highlighted, continue to hold. As the SOW states, a necessary condition for the SACE program to pursue its objectives was for the willingness of “clusters of organizations [to] work on a clearly-defined thematic issue area, with a shared vision for change, and ‘anchored’ by one organization responsible for creating and managing collaboration. These clusters become intermediaries between citizens, individual organizations, and the broader civil society sector, with the expectation that these collaborative behaviors will expand beyond discrete clusters.” The document went further by noting that, for the SACE program to work, several context￾specific necessary conditions had to be satisfied. These were as follows: • Civil society fully embraces a system perspective and a commitment to multi-stakeholder engagement. • Cluster members demonstrate a commitment to regular learning and improvement, both in their technical issue area and in their collaborative capacities. • Clusters reach out to a diverse range of stakeholder groups and actors with complementary skills and competencies. • Clusters embrace feedback loops and other learning processes to ensure continued improvement and maintenance of cluster strategy and adaptability. • Clusters engage in multipronged strategies aimed at reaching the full range of stakeholders (including the public, influencers, and decision makers). • Clusters develop funding strategies to resource regular convening and continuous communication among members. Moreover, the SOW (p. 5) notes that certain crucial assumptions had to prevail for the program to pursue its objectives, since “complex social change processes cannot reliably be attributed to a linear casual chain, but are rather embedded in a complex web of interactions among diverse actors, forces, and trends.” Therefore, the program assumes that: • No single organization can create large-scale and lasting social change alone. • Organizations contribute to processes of social change instead of controlling them. • Influencing change requires engaging with diverse actors, forming or modifying relationships and understanding of motivations to contribute to behavior change. • Voice and accountability projects succeed best when they incorporate intentional and continuous learning and iteration. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 40 • Traditional organizations’ capacity building practices do not support the kind of system-focused competencies that are needed for large-scale and sustainable social change. • A system-oriented approach to capacity building focuses on relationships, information sharing, and identifying common interests. Finally, several activity-level critical assumptions, mostly pertaining to Nigeria’s complex governance environment, were required for SACE to be implemented. These assumptions included continued and timely funding for the activity, provided that: (1) Nigeria would remain stable enough for democracy, governance, and human rights programming to continue to progress; (2) governments at all levels would continue to be willing and committed to engage; (3) target communities would embrace SACE and its programming activities. Potential challenges and issues, mainly over the short term, include the likely willingness of SACE to revise some of its training procedures, as the Recommendations/Future Directions section describes, to respond to CSO concerns. A related issue pertains to how—if at all—the program could be reformulated to address frequently voiced CSO concerns about financial issues. These include the potential introduction of small grant mechanisms to facilitate the swifter implementation of advocacy activities that are not contemplated in cluster work plans, together with alternatives to encourage the sustainability of economically vulnerable CSOs. These concerns would have to be addressed in the program design for any successor activities. Over the long term, two pressing issues emerge. The most crucial and immediate is how to “nationalize” the core components of the existing program (the SACE suite of advocacy tools) to help embed the art of constructive engagement in the country’s changing political culture to achieve needed policy reforms. This would include the domestic management of many of the SACE-promoted tools, including the well￾regarded STARNET networking mechanism, while furthering its development to gradually increase its value as a national advocacy and policy reform instrument. Aside from these issues, a redesigned SACE activity could borrow most of its activities from its initial first phase, although it should also expand its focus and educational reach to cover more economic development priority sectors, as the Recommendations/Future Directions section suggested. A redesigned activity would still face the main challenge of how to cajole Nigerian powerbrokers, both inside and outside the government, to listen and work more closely with civil society to improve the country’s governance. RECOMMENDATIONS/FUTURE DIRECTIONS In accordance with the SOW of the evaluation, the recommendations that follow address program approaches and activities that potentially could be scaled up or replicated, mainly over the short- and medium-term, while also suggesting how and in what fashion they could be implemented. A set of longer￾term recommendations, partly based on these proposals, both builds on the shorter-term suggestions and reflects what has been—in the collective view of the ET—a consequential program with many achievements. The latter recommendations are geared towards the potential development of a new civil society advocacy program once the SACE activity ends in December 2018. In proposing additional short￾to medium-term activities to be scaled up or replicated, the ET assumes the USAID/Nigeria will have the means to support these. 41 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV SHORT-TERM RECOMMENDATIONS There are short-term activities that could be scaled up, replicated, or introduced. Based on the above evaluation results and many suggested by key informants themselves, these short-term recommendations fall into three basic categories: 1. Improving SACE training procedures. 2. Establishing new, small, and targeted funding mechanisms to reward success. 3. Addressing sustainability issues. STRENGTHENING TRAINING PROCEDURES These short-term recommendations flow directly from the KIIs, as almost all the interviewees indicated that training procedures could be improved. Paradoxically, while key informants enthusiastically praised the contributions of the 2.0 capacity and other STAR tools training to their advocacy activities, they just as often complained about the difficulties they were having. These were not necessarily associated with the internalization and application of SACE-acquired advocacy knowledge, but rather with the way they perceive they were taught. The basic complaint was that too much had been taught in too short a time and with insufficient didactic and documentary support. SACE could take some relatively simple and straightforward measures to remedy this situation, such as lengthening the time for workshops and seminars. SACE could also engage the services of training specialists to help design more user-friendly training procedures built around observations of current practices and input (obtained through brief questionnaires, focus groups, etc.) the trainees themselves provide. Another suggestion that could be easily implemented would address key informants’ perceived lack of SACE’s support in making didactic material available. Many CSO informants complained about not receiving training materials, whether in hard or soft copy, during or after training sessions. Expeditiously providing them in the form of flash drives or via the Internet would alleviate this problem. The issue of compressed training time also affects the ALS. The ALS often covers broad and complex themes. In 2016, these included cluster collaboration and strategy alignment, developing and practicing critical cluster behaviors, and the role of knowledge in advocacy approaches. All these themes and other ALS activities were condensed into three-day events (the 2015 and 2016 ALSs), which most CSO key informants considered too short. In response to these concerns, SACE increased the length of the third ALS, in September 2017, to four days. The SACE should consider further increasing the duration of the ALS to five days. The SACE program must also evaluate means to minimize the loss of institutional memory suffered by many of its partner CSOs. Once staff are trained, CSOs often lose them to better-paying jobs because of their improved qualifications. Although this result may be inevitable in some cases, it has been exacerbated by SACE’s persistence on partner CSOs nominating specific point persons to attend capacity-building activities. As one anchor representative told a member of the ET, “SACE’s insistence on having a designated staff member from each organization attend its trainings leads to over capacitation and eventual exit of that staff from the organization. Most times, programs revolve around individuals trained, and it makes no sense that you capacitate only a few staff and then lose them. This can lead to the eventual collapse of the program.” This is not a major concern to other CSOs, as SACE does not seem to strictly enforce the designated staff member rule. Another key informant noted that “when they ask me to send somebody, I send [all] my staff.” The difference may have to do with travel and lodging costs, and where USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 42 specific training events are held, as SACE should not object as much to the number of trainee participants as it does to having to assume participation expenses from those coming from distant locations. Going forward, SACE should creatively liberalize the process of nominating staff for the different capacity-building sessions, while also requiring that anchor and cluster staff trained in unique skills share that knowledge with other members of their organizations. TARGETED FUNDING MECHANISMS TO ENCOURAGE/REWARD SUCCESS The joint development of cluster work plans has been effective for the design and implementation of cohesive advocacy approaches, but has, in some ways, interfered with the ability of CSOs to be quick on their feet, assume risks, and take advantage of windows of opportunity. In order to ease these concerns, SACE should consider establishing several new short-term funding mechanisms for specific activities not included in the routine work plan. The purpose would be to encourage particularly successful CSO cluster members-demonstrated via remarkable achievements in 1.0 and 2.0 skill set acquisition–to embark in innovative, rapid response, and specific activities, and thus avoiding them missing promising windows of opportunity. The former would amount to a “cluster performance incentive fund” to be accessed under special circumstances by individual cluster members to support specific advocacy agenda items, for example in support of a local advocacy initiative. Such a fund would be welcomed by some Niger Delta cluster members that lament not having sufficient resources to deploy newly acquired capacities in pursuit of community-level initiatives. While the ET lauds SACE’s objective of strengthening anchor organizations to ensure that at least three of them qualify for direct USG funding at the end of the program, we are persuaded that a cluster-level organizational capacity-strengthening performance incentive, designed to benefit best performing and most improved cluster members, will greatly encourage the achievement of results, not only in the Niger Delta, but across the width of Nigeria, too. In addition, SACE should evaluate establishing a “windows of opportunity fund” that could be tapped by clusters as a rapid response mechanism to enable them to respond to emerging opportunities in their issue areas and regions of activity. One example of the potential usefulness of such a fund would be taking advantage of the “Women August Meeting Platform” to multiply gender-based advocacy in the South East of Nigeria and rally women as key development stakeholders. Finally, an innovative economic development advocacy fund would be particularly appropriate for Niger Delta clusters given their focus on economic and policy reforms. Grants under this fund can be used to encourage innovative economic transition initiatives, an objective consistent with the overall SACE design. THE SUSTAINABILITY ISSUE Sustainability is always a particularly difficult issue to address in a world of budget constraints and competing needs for resources. The reality in Nigeria is that nearly all CSOs depend on donor support, which affects institutional sustainability due to short funding cycles. Given this reality, what is the best way to safeguard SACE’s advocacy investment, if financially weak but advocacy capable CSOs have difficulties in sustaining themselves? The answer may lie in expanding SACE-provided training beyond 1.0 and 2.0 skills, to include techniques for developing sustainability plans, including acquiring fundraising capabilities, during the remaining life of the program. This training should go beyond proposal writing skills and extend to potential procedures to generate new income streams. 43 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Additionally, with certain cluster CSOs, SACE can offer support to help cover institutional overhead costs, while encouraging these organizations to find sustainable footing in the long term. Other lower-level and monitoring-related recommendations include: • Updating the latest SACE DQA to determine the validity of some of its performance indicators, particularly the one related to counts of unique persons trained in unique skills, which may be subject to potential biases leading to exaggeration. • Evaluating the need to better capture community-level outcomes. The ET feels that some cluster CSOs do not report all the results they achieved at the local and community levels, instead assuming that only SACE-instigated initiatives must be accounted for. This leads to under￾reporting for achievements that came about through indirect SACE support. LONGER-TERM RECOMMENDATIONS The three short-term recommendations that were presented above should be considered for any future SACE follow-on programming. Given the success of the program, as evidenced by the information provided by informants interviewed, the overall orientation of the SACE program should remain unchanged, and in fact be expanded. BROADEN THE REACH OF THE SACE ADVOCACY APPROACH Follow-on activity could be expanded by including other national priority sectors, such as employment generation, housing availability, and access to clean water. These economic sectors have a major bearing on well-being, which would require CSOs’ intensified engagement with political leaders and government officials. Since the effectiveness of the SACE approach transcends sectoral boundaries, the expectation would be that clusters focusing on those issues will also achieve positive advocacy results. A future program, however, may wish to go beyond what is readily discernible. The SACE approach has considerable educational potential, since it could teach the average citizen about democratic government interaction with civil society. Promoting a broader understanding among citizens of how to bring about policy change could fortify CSOs’ ability to advocate for political and economic reforms at all levels of government. A national awareness campaign to promote advocacy literacy could be conducted in support of this objective, focusing on the benefits Nigeria’s citizens could gain from the expanded applicability of the advocacy tools introduced in the country by the SACE program. A more targeted approach would be to disseminate the essentials of the SACE advocacy approach to all CSOs, regardless of type or location. A major national CSO, for example, could be entrusted with the task of educating small grassroots organizations on the potential benefits to be gained through the acquisition of 2.0 capacity skills, even if achievements are at first only marginal. Such training could be provided through well-advertised local workshops to be conducted on a rolling basis across Nigeria. This approach presupposes the exclusion of grant support, and assumes that the transference of basic advocacy skills could be supported by a comprehensive and sustained public awareness effort, perhaps centered in the state or locality being targeted at that time. A related and complementary activity would be to conduct concurrent and parallel initiatives to be focused on policy-makers, thus taking advantage of economies of scale. This activity would enable policy-makers to become sensitive to the policy formulation advantages that would flow from increased engagement between civil society representatives and government officials. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 44 Public education could be expanded through relatively simple and affordable initiatives that could take further advantage of both conventional mass media and social media, as SACE’s programming has done. For instance, a follow-on program could publicize successful advocacy achievements to energize disadvantaged sectors of the Nigeria population not yet benefitting from them. In connection, an open access website could be established for citizens to monitor on a continuing basis successes and impediments to STAR-inspired advocacy campaigns. Results could also be disseminated through social media and other venues, including innovative outlets, such as films and TV/radio docudramas. Additionally, a simplified, self-paced on-line STAR course could be developed to demonstrate the essentials of effective policy reform advocacy for interested citizens. This course could also be used as a teaching device for high school and college students. INCREASE THE ADVOCACY VALUE OF THE STARNET The STARNET is an indispensable tool to forge sectoral and regional alliances. It is one of the most well￾regarded components of the SACE program, and any likely follow-on program must necessarily build upon it. However, the STARNET’s basic architecture should be expanded to enhance its efficacy further. While the anchor/cluster model has worked, a reformulated STARNET iteration should strive to expand its thematic engagements by feeding into a national-level network of nonstate actors with the capability to support regional and national-level policies around transparency, accountability, good governance, and the rights of vulnerable and marginalized groups. A new iteration should also continue expanding advocacy networks, with a special emphasis on linking national CSOs with CSOs at the state and LGA levels. Likewise, a future STARNET should strive, through more elaborate mapping and engagement, to broaden the networks of grassroots CSOs. The ET identified some notable advocacy successes resulting from modest and local initiatives that were sponsored by such CSOs. As there are many similar CSOs across Nigeria, a long-term strategy should be devised to encourage them to use and embrace SACE-type advocacy procedures. In the Niger Delta and other Nigerian regions, traditional institutions and community-based development associations and town unions may be enticed to join advocacy efforts with minimal support, if these institutions are properly empowered with information and organizational capacity. Future programmatic activities should give priority to engaging these traditional and long-standing networks of nonstate actors. Their involvement, once equipped with 1.0 and 2.0 capabilities, could eventually revolutionize the practice of governance in Nigeria. Another needed STARNET upgrade should include giving users the capability to rely on it to conduct integrity checks. This could be achieved by requiring that participating CSOs provide more information regarding their institutional track records and governance structures. Similarly, the STARNET could be redesigned to include development-oriented community-based organizations. Networking and building alliances with these grassroots development champions could promote long-term sustainability, even after the mandate of a possible SACE successor program expires. Lastly, copyright matters may complicate the issue of Nigerian ownership of the STARNET platform and related STAR advocacy tools, although this does not appear to be the case as many of these tools were jointly developed by SACE and Root Change teams and are USAID property. This being the case, many SACE achievements can be attributed to the systematic support Root Change provides to the program. Root Change, for instance, hosts the STARNET online, while this entity also conducts Organizational Change Surveys. To ensure ownership and sustainability, these functions should be transferred to an adequately trained local organization to manage the process, including control of the data for analysis. 45 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Two positive developments suggest this eventual outcome and were communicated to the ET. They are Root Change’s willingness to gradually “domicile” the STAR system in Nigeria, together with the fact that SACE has given much thought to the matter. The SACE team is prepared to engage the USAID to ensure, not only the national ownership of STAR, but also its long-term sustainability. One alternative is to consider supporting a CSO “trust” (akin to the Southern Africa Trust), as part of a potential follow-on activity. This entity, to become self-sustaining in three to five years, would be tasked with providing research, capacity building, and technical support to the broader multi-sectoral CSO community, while assuming a wide ecosystem perspective. In order to succeed, it should be independent and apolitical, and perhaps initially funded through an innovative multi-donor basket. The proposed trust would also house and manage a website to disseminate among the broader CSO community the existing SACE tools and other experiences generated in years to come. OTHER LONG-TERM RECOMMENDATIONS Other obvious, but just as important recommendations for a future SACE-like program include: • Improve networking mechanisms between CSOs and political leaders. While SACE had much success linking CSOs together, it barely scratched the surface in developing comparable two-way devices to ease the flow of information between CSOs and policy-makers. • Expand media access to well-structured electronic mediums (TV, radio, webinars), in which well￾trained policy change advocates could debate with policy-makers the technical and other merits of policy reform proposals. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 46 REFERENCES Adeniyi, Adesoji. “Osun women farmers demand farm settlement.” The Nation. Sunday, June 25, 2017. Advocacy Nigeria. “Building Reproductive Health Advocacy to Reduce Maternal and Infant Mortality.” Fact sheet. Kaduna State, 2017. Centre for Information Technology and Development (CITAD). “Recommendations of the Task Force Committee to Overhaul Education Sector in Bauchi State.” Bauchi, 2017 Foundation for Partnership Initiatives in the Niger Delta (PIND), “Strategic Plan 2015 – 2019,” 2015. Root Change. “Nigeria Strengthening Advocacy and Civic Engagement (SACE). Combined Quarter 4 and Annual Report.” September 2015. Root Change. “Nigeria Strengthening Advocacy and Civic Engagement (SACE). Combined Quarter 4 and Annual Report.” October 2016. ---. “Interim Outcomes.” Infographic. October 2016. ---. “Performance Data Table.” September 2014. ---. “Nigeria Strengthening Advocacy and Civic Engagement (SACE): Performance Management Plan.” April 2, 2014. ---. “Nigeria Strengthening Advocacy and Civic Engagement (SACE): Year One Work Plan.” February 14, 2014. ---. “Nigeria Strengthening Advocacy and Civic Engagement (SACE): Year Two Work Plan.” October 30, 2014. ---. “Nigeria Strengthening Advocacy and Civic Engagement (SACE): Year Three Work Plan.” October 30, 2015. ---. “Nigeria Strengthening Advocacy and Civic Engagement (SACE): Year Four Work Plan.” October 31, 2016. ---. “Report on Mid-term Evaluation of the USAID/PIND Strengthening Advocacy and Civic Engagement (SACE) Project in the Niger Delta.” November 2016. ---. “Strengthening Advocacy and Civic Engagement: Semi-Annual Report January 2 – June 30, 2014.” December 30, 2014. ---. “Strengthening Advocacy and Civic Engagement: Annual Report October 1, 2014 to September 30, 2015 and 4th Quarterly Report, July 1, 2015 to September 30, 2015.” October 30, 2015. ---. “Strengthening Advocacy and Civic Engagement: Annual Report 2014.” February 13, 2015. ---. “Strengthening Advocacy and Civic Engagement: Semi-Annual Report October 1, 2014 to March 31, 2015 and 2nd Quarterly Report January 1, 2015 to March 31, 2015.” April 30, 2015. ---. “Strengthening Advocacy and Civic Engagement: Quarterly Report, 3rd Quarter - April 1 to June 31, 2015.” July 30, 2015. ---. “Strengthening Advocacy and Civic Engagement: Semi-Annual Report October 1, 2015 to March 31, 2016 and 2nd Quarterly Report January 1, 2016 to March 31, 2016.” April 30, 2016. 47 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ---. “Strengthening Advocacy and Civic Engagement: Annual Report October 1, 2015 to September 30, 2016 and 4th Quarterly Report July 1, 2016 to September 30, 2016.” October 31, 2016. ---. “Strengthening Advocacy and Civic Engagement: Quarterly Report, 3rd Quarter - April 1 to June 301, 2017.” July 30, 2017. ---, “Strengthening Advocacy and Civic Engagement: Project Overview and Progress: USAID Mid-term Evaluation Presentation,” August 15, 2017. ---, “Impact Assessment of Processes of Homevida Films and Strategies for Expanded Viewership: A Research Study Commissioned by the Public and Private Development Centre (PPDC) on the film awards by Home Video Integrity Film Awards (Homevida), September 6, 2017. United States Agency for International Development (USAID), “Mid-Term Performance Evaluation of the Leadership, Empowerment, Advocacy and Development (LEAD) Project, Final Report,” The Mitchell Group, Inc., December 2013. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 48 ANNEXES 49 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX I: EVALUATION STATEMENT OF WORK (SOW) STRENGTHENING ADVOCACY AND CIVIC ENGAGEMENT (SACE) SOW: MIDTERM PERFORMANCE EVALUATION ACTIVITY IDENTIFICATION DATA DEVELOPMENT CONTEXT COUNTRY BACKGROUND AND DEVELOPMENT CHALLENGES With a population of about 182 million people, Nigeria is the largest country in Africa and accounts for 47 percent of West Africa’s population. It is also one of the largest oil exporters in Africa, with the largest natural gas reserves on the continent. Real gross domestic product shrank in 2016 and has barely begun to grow again in 2017. The sharp decline in oil prices since June 2014 has posed major challenges to the country’s external balance of payments and public finances. Oil accounts for nearly 90 percent of exports and roughly 75 percent of the country’s consolidated government revenues. Continued lower oil prices will continue to represent a major constraint on the ability of the government at all levels to launch some of its ambitious programs. Despite the government’s efforts to privatize and revitalize the power sector, it remains the bane of Nigeria’s economic growth, development, and public service delivery. The fifth consecutive national elections in March and April 2015, which were won by an opposition party, were considered fair and relatively peaceful by national and international election monitors when compared to the postelection violence of 2011. The success of the 2015 general elections is largely credited to the Independent National Electoral Commission (INEC) and the statesmanship of the presidential candidates. The Buhari administration is focusing on anticorruption, jobs and unemployment, the economy, and security. In the Northeast, Boko Haram remains a threat, and millions of displaced persons require assistance. There has been major destruction of infrastructure along with the loss of lives and impoverishment in this region. The Buhari administration is having a difficult time financing the major programs that it wants Development Objective Activity Title Strengthened Good Governance Strengthening Advocacy and Civic Engagement (SACE) Award Number Award Dates AID-620-C-14-00001 January 2, 2014 – January 1, 2019 Type of Contract Total Estimated Cost Cost Plus Fixed Fee $19,280,704 Contract Officer’s Representative (COR) Implementing Partner (IP) Augusta Akparanta-Emenogu Chemonics International Inc. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 50 to roll out as well as reconstructing the Northeast. Violent conflicts between crop farmers and cattle farmers (often nomadic herders) have also been increasing in frequency and intensity. In other parts of the country, kidnapping, robbery, and other crimes continue to challenge the government’s efforts to maintain security. At times in recent years, militants have resumed attacking oil infrastructure, drastically reducing output and further crippling government finances. Accelerating the creation of productive jobs through private sector growth and improvements in education remain the major medium-term challenges. To date, the pace of job creation has been inadequate, leading to increasing frustration among underemployed Nigerian youth. CIVIL SOCIETY The change of political leadership, after the electoral defeat of the People’s Democratic Party in 2015, created high expectations of change and improved service delivery for the citizenry. Halfway through President Buhari’s four-year term, civil society is having to reappraise its partnership with the government and the influence of such collaboration on the operating environment for CSOs, especially in light of the implicit threats they face from proposed legislation. Two bills were brought before the National Assembly in an attempt to constrain CSO operations in Nigeria. In 2015, a bill to restrict funding to CSOs was sent to the National Assembly. The bill sought to require approval within 90 days by the Independent Corrupt Practices Commission for any civil society group that received at least one million naira (approximately $2,700) in donor funding. The rationale for the bill was to prevent civil society from spending illicit funds or funding terrorism. In order to oppose this, CSOs came together, presented a unified front, and attended public hearings; consequently, the bill was tabled. CSOs argued that they should be regulated by the Corporate Affairs Commission, which regulates companies (and with which they are registered). The second attempt to constrain CSO was a social media bill was presented to the National Assembly in 2016. This bill sought to regulate the misuse of social media, ostensibly to stem false allegations against government officials. SACE partner CSOs have been actively involved in the activities that resulted in the first bill being tabled and in ongoing attempts to ensure that the second bill is not passed. While the civil society community admits that some form of regulation is needed, self-regulation is viewed as preferable. The 2015 USAID CSO Sustainability Index for Nigeria reports that capacity-building efforts for CSOs remained strong, especially in the period leading up to the elections. There was a slight increase in the use of strategic plans by local CSOs, but management structures remained weak, staffing deteriorated, and quite a number of CSOs lost staff to international donors who could pay higher wages. Although CSOs’ use of technology improved, quite a number of CSOs still struggled with payment of staff salaries. In fact, there was a slight decline in terms of financial sustainability. Service provision by CSOs continued in such social sectors as health, education, and governance. Although the elections divided civil society along ethnic and religious lines, CSOs’ public image remained positive. The media reported extensively on CSO activities and how such activities aided election outcomes. There was increased visibility of CSOs, especially on social media. PROGRAM DESIGN CHRONOLOGY In May 2009, the USAID/Nigeria reviewed the validity of the central finding of the PDG Office’s 2006 democracy and governance strategic assessment that a small elite, or oligarchy, continue to dominate the nation’s politics, capturing the country’s wealth to the exclusion of the vast majority of the public. The 51 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV assessment team determined that, not only was this finding still valid, but also that the situation had further deteriorated. Oligarchs remained dominant in all spheres of social, political, and economic life. Also, corruption was rampant, and the multiparty system was being threatened by the dominance of one party. Despite the negative trends, the assessment team noted some signs of light. Indeed, pockets of democratic reform were present in parts of the system, including the Supreme Court, the federal appellate courts, and some of the election tribunals. The team also noted that some oligarchs were beginning to recognize a need for reform as their power was being increasingly threatened. Based on these findings, the team recommended that the USAID/Nigeria consider opportunities for collaborating with these “reform-minded” elites, finding the space where the public interest coincides with these leaders’ self-interest and the USAID’s manageable interest. The team further articulated the development hypothesis behind this recommendation as the following: Civil society can help to create, take advantage of, and replicate the alignment of certain Nigerian leaders’ political will with public interest in order to transform pockets of reforms into a self-reinforcing system of good governance. Also in May 2009, an external team of experts conducted a mid-term evaluation of a previous USAID civil society program, that is Advocacy, Awareness, and Civic Empowerment (ADVANCE). Key recommendations of the evaluation were: 1. Increase activities to strengthen Nigerian coalitions’ and networks’ ability to effectively support and engage their client/member institutions, increase their members by “reaching down” to larger segments of the population and engaging a wider array of partners, find the proper level of institutionalization to best support their goals, and improve their financial strategies. 2. Ensure national level interventions are more targeted and focused on fewer issues. 3. Deepen impact and expand partners’ base to organizations with deeper and broader constituencies by expanding partnerships with organizations outside of the traditional sphere of coalitions, networks, and public policy and watchdog organizations. This includes increasing cross-sectoral partnerships, such as strengthening collaboration on budget tracking and advocacy initiatives. Finally, in March 2010, a USAID team of experts from Washington and Nigeria conducted consultations with an array of Nigerian stakeholders, including civil society activists, government institutions, the private sector, bilateral and multilateral donors, and USAID/Nigeria’s administrative team as well as technical specialists. The goal of the consultations was to collect the necessary information to develop a new USAID/Nigeria civil society program. The assessment/design team met with stakeholders in Abuja and other Nigerian towns to solicit ideas on how the new civil society program should be designed to help strengthen their work. The findings of the consultations substantiated the results of the ADVANCE mid-term evaluation as well as the PDG strategy update. The experts conducted a situation analysis to validate their report. The situation analysis revealed that many roadblocks to strengthening democracy remained in Nigeria at all levels of governance. Political elites were still entrenched in political structures, excluding most citizens from meaningful political participation. Violent conflict, which was triggered by political competition and communal, ethnic, religious, or resource allocation rivalries, continued to pose a major threat to democracy in Nigeria. Systemic corruption remained a key governance issue that pervaded the USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 52 daily lives of Nigerians, who believe that only unemployment is a bigger challenge to their future. Civil society still lacked both the capacity and the resources to truly engage with the government and advocate for change. Government institutions had not established partnerships with citizens or the private sector and lacked the capacity to carry out their own mandates. Nigeria remained on the USG Religious Freedom Watch List due at least in part to infringements on civil liberties perpetrated on religious grounds. In December 2013, the USAID awarded a contract to Chemonics Inc. to implement an activity to strengthen civil society’s ability to influence the development and implementation of key democratic reforms at the national, state, and local levels. The effective award date for this five-year activity was January 2, 2014. The objective of the SACE activity is to strengthen civil society’s ability to influence the development and implementation of key democratic reforms at the national, state, and local levels. The activity supports increased engagement and efficacy of civil society to be able to influence public institutions whose function it is to serve citizens’ interests. This is to be achieved through strategic partnerships between civil society coalitions and (formal or informal) networks on the one hand, and other critical stakeholders in Nigerian society on the other. The latter includes the media, reform-minded GON institutions or representatives, private sector actors, and influential public figures, as well as other key stakeholders in target areas, such as grassroots organizations, professional associations, and youth groups. The activity also explicitly aims at engaging marginalized populations (such as women, youth, and persons with disabilities) in the process and emphasizes the importance of leadership and innovation. The new civil society activity was fashioned to complement other PDG sector programs, such as the LEAD (Leadership, Empowerment, Advocacy, and Development) subnational governance program, by supporting local CSOs through strategic connections with CSO coalition and network partners. It was also expected to coordinate with the now-completed Media Support for Strengthening Advocacy, Good Governance, and Empowerment (MESSAGE) activity. The SACE’s media partner eventually pulled out of the consortium and the Contractor therefore now uses local consultants to deliver on the component that is geared to strengthening public discourse (Contract Line Item [CLIN] 3), which focuses on raising public awareness about governance challenges and building consensus on options for tackling them. Other PDG activities expected to collaborate with SACE include the conflict management and elections activities. DEVELOPMENT HYPOTHESIS AND CRITICAL ASSUMPTIONS SACE’s development hypothesis states that civil society’s increased capacity and engagement with public institutions will influence the development and implementation of key democratic reforms at national, state, and local levels, if the security situation does not deteriorate. This is implemented through an anchor-cluster model that embraces the lessons of voice and accountability and focuses on higher-order advocacy capacities. NECESSARY CONDITIONS To this end, clusters of organizations work on a clearly-defined thematic issue area, with a shared vision for change, and “anchored” by one organization responsible for creating and managing collaboration. These clusters become intermediaries between citizens, individual organizations, and the broader civil 53 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV society sector, with the expectation that these collaborative behaviors will expand beyond discrete clusters. In order for this approach to succeed, the following necessary conditions are required: • Civil society fully embraces a system perspective and a commitment to multi-stakeholder engagement. • Cluster members demonstrate a commitment to regular learning and improvement, both in their technical issue area and in their collaborative capacities. • Clusters reach out to a diverse range of stakeholder groups and actors with complementary skills and competencies. • Clusters embrace feedback loops and other learning processes to ensure continued improvement and maintenance of cluster strategy and adaptability. • Clusters engage in multipronged strategies aimed at reaching the full range of stakeholders (including the pubic, influencers, and decision-makers). • Clusters develop funding strategies to resource regular convening and continuous communication among members. ASSUMPTIONS Complex social change processes cannot reliably be attributed to a linear causal chain, but rather are embedded in a complex web of interactions among diverse actors, forces, and trends: • No single organization can create large-scale and lasting social change alone. • Organizations contribute to processes of social change instead of controlling them. • Influencing change requires engaging with diverse actors, forming or modifying relationships, and understanding motivations to contribute to behavior change. • Voice and accountability projects succeed best when they incorporate intentional and continuou s learning and iteration. • Traditional organizations’ capacity-building practices do not support the kind of system￾focused competencies needed for large-scale, sustainable social change. • A system-oriented approach to capacity building focuses on relationships, information sharing, and identifying common interests. Activity-level critical assumptions are: • Continued and timely funding for the activity. • Nigeria will be stable enough for democracy, governance, and human rights programming to continue to progress. • Governments at all levels will continue to be willing and committed to engage. • Target communities will accept SACE and its programming activities. PROGRAM GOALS AND OBJECTIVES USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 54 The SACE has four components, each of which is a CLIN: 1. Strengthened institutional, organizational, and technical capacity of targeted CSO coalitions and networks to advance targeted democracy and good governance initiatives. 2. Strengthened partnerships between CSO-led coalitions and networks and targeted GON institutions and key stakeholders to advocate for and monitor select democratic reforms aimed at strengthening transparency, accountability, and responsiveness of government institutions. 3. Strengthened public awareness, discourse, and support for key democratic governance issues, such as transparency, accountability, and good governance. 4. Strengthened capacity of partner BMOs and CSOs in the Niger Delta to advocate for inclusive economic reforms and equitable economic growth. The Chevron-affiliated Foundation for Partnership Initiatives in the Niger Delta (PIND) is the USAID’s strategic partner in the Niger Delta. The SACE complements PIND’s other activities in the region under the fourth component. The SACE supports the USAID Forward initiative and its goal of increasing the number of procurements directly with local organizations. In its first three years, SACE worked with 16 anchor civil society sub-grantees and their cluster members on agriculture, health, education, public finance, extractive industry transparency, gender, social inclusiveness, conflict resolution, and transparency and accountability in governance. The anchor CSOs are: • Centre for Information Technology and Development (CITAD), Kano, which works on education in the six northeastern states of Adamawa, Borno, Yobe, Bauchi, Gombe. and Taraba. • Civil Society Legislative Advocacy Centre (CISLAC), Abuja, which advocates for transparency and accountability in extractive industries at the national level and in target states. • Centre for Social Justice (CENSOJ) and Health Reform Foundation of Nigeria (HERFON), both based in Abuja; they advocate for improved public financial management in the health sector in the Federal Capital Territory (FCT) and Kaduna and Katsina states. • Human Development Initiatives (HDI), Lagos, which focuses on improving local governance in the FCT, Lagos, Enugu, and Kano states. • Disability Rights Advocacy Center (DRAC), Nassarawa, which advocates for improved reproductive health and rights for disabled women and girls in the FCT, Nassarawa, and Bauchi states. • Joint National Association of Persons with Disabilities (JONAPWD), Abuja, which focuses on social inclusion for persons with disabilities in the FCT and Kebbi state. • Women Advocates Research and Documentation Centre (WARDC), Lagos, which supports social inclusion for women farmers in Enugu, Osun, Kwara, and Benue states. • Citizens Center for Integrated Development and Social Rights (CCIDESOR), Imo, which works on improving public financial management at the state and local government level and ensuring accountability in Imo and Abia states. 55 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV • Africa Network for Environment and Economic Justice (ANEEJ), Edo, which advocates for increased transparency in the Niger Delta institutions and resource management in Edo and Delta states. • Social Action, Rivers, which focuses on state and local government accountability systems in Rivers, Edo, and Delta states. • Leadership Initiative for Transformation and Empowerment (LITE-Africa), Delta, which works on Niger Delta institutions and resource management in Delta and Ondo states. • Nembe City Development Foundation (NCDF), Bayelsa, which encourages peace and conflict management in the extractive industries in Bayelsa state. • Youth Alive Foundation (YAF), Akwa Ibom, which strives for social inclusion for youth in Akwa Ibom state. • Ugboroke Progressive Fish Farmers’ Association (UPFFA), Delta, which works on inclusive economic growth for fish farmers in Delta state. • Edo State Co-Operative Farmers Agency Limited (ESCFAL), Edo, which advocates for inclusive economic growth through favorable agricultural policies for farmers in Edo state. APPROACH AND IMPLEMENTATION The SACE activity was expected to achieve its objectives by: 1. Supporting six national issue clusters/coalitions/networks, each with an anchor CSO that serves as a foundation for collective action on a core advocacy issue and facilitates the efforts of participating organizations. 2. Supporting six Niger Delta CSOs and their clusters on issues of economic development and growth in the Niger Delta. 3. Directly engaging and supporting initiatives that strengthen and improve the enabling environment for CSOs to operate effectively, with a focus on political economy analysis and identification/response to windows of opportunity, the legislative space, innovation, and new technologies as a driver of change. The activity also expected to: 1. Strengthen anchors and their issue clusters/coalitions/networks through deploying Root Change’s Systems for Transformation and Results (STAR), focusing on the six performance drivers (constituent relations, monitoring and evaluation, financial sustainability, people development, issue advocacy, and sectoral leadership) as well as promoting aggressive networking and supporting partners to develop clear and time-bound “change initiatives”. 2. Strengthen multi-stakeholder engagement through utilization of the STAR suite, with an emphasis on STARNET analytics, coaching, and advanced networking and advocacy skills. 3. Invest in a community of local coaches with promising practices to engage and animate stakeholders. 4. Utilize a range of monitoring and evaluation tools to measure progress and, more importantly, learn what works and why, so that it can be shared for scale-up. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 56 5. Support partners to develop and implement public awareness initiatives on their selected issues. 6. Deploy a Capacity Development Fund (CDF), which is an Innovation and Spread the Word Fund (ISWF), to support innovative stakeholder engagement approaches, and a Niger Delta Fund (NDF). EXISTING DATA The ET will have access to the vital documents relevant to conducting this evaluation. These documents will include the technical proposal (program description), performance monitoring plans (PMPs), annual work plans, quarterly and annual reports, the PIND mid-term assessment report, and any other relevant materials documenting the management, implementation process, and results for the SACE activity permitted by the Office of Acquisition and Assistance. EVALUATION RATIONALE EVALUATION PURPOSE The contractor will conduct a mid-term performance evaluation and provide a report of the findings of the evaluation of the SACE activity that Chemonics International Inc. implemented. The evaluation is intended for decision-making, especially to bolster support and programming activities that assist the achievement of SACE’s goals, guide changes where necessary, and improve on the implementation of the activity to make it more effective in achieving its objectives. The evaluation will assess the progress made in achieving SACE’s goals and objectives, assess the relevance, effectiveness, and management of major activities in implementation focus states, determine whether expected results are being achieved, and document best fit innovations and lessons learned. The USAID also intends to incorporate evaluation findings into the design of a new civil society activity. Specifically, this external mid-term performance evaluation will: • Assess the efficacy and results of the SACE design, implementation approaches, and management structure in meeting the objectives. • Evaluate SACE’s overall performance by assessing results against stated targets and indicators, and determine if programming is on course. • Make recommendations to the USAID/Nigeria concerning possible programming changes or adjustments during the remainder of SACE’s implementation. • Analyze interventions to determine if they are strengthening civil society’s technical and financial capacity to advocate for increased participation of vulnerable groups (i.e., women, youth, and persons with disabilities) in decision-making, support the government to institutionalize transparent and accountable governance, and to advocate for economic growth in the Niger Delta region. • Make recommendations to the USAID/Nigeria regarding the design of a new civil society activity. AUDIENCE AND INTENDED USERS 57 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV The primary users of the evaluation findings are the USAID PDG Office, other Mission colleagues, and other USG officials. Other audiences include non-USG donor organizations, the Government of Nigeria (GON), and CSOs, all of which can use the findings to improve democracy and governance and human rights programming in Nigeria. The evaluation report will also serve as a reference material for program design, implementation, monitoring, evaluation, and management of future PDG activities. EVALUATION QUESTIONS The ET will develop a methodology that will answer the following key evaluation questions: 1. To what extent was the design and implementation of SACE’s anchor/cluster model successful in promoting its intended key democratic reforms? 2. What worked in SACE’s efforts to increase the financial and technical (e.g., in terms of advocacy) capacity of targeted CSOs/clusters to enable them to engage with the government on reforms? What did not work? • To what extent did Systems for Transformation and Results (STAR) result in increased capacity of targeted CSOs/clusters? • Are at least three SACE anchor partners ready to receive direct funding from the USAID? If partners are not ready to receive direct funding, are they likely to be ready by the end of calendar year 2017? 3. To what extent did SACE partners successfully influence government policy development and implementation regarding vulnerable populations, such as women, youth, and persons with disabilities? 4. Did SACE public awareness activities contribute to strengthening public discourse and support for key democratic governance issues? • In particular, what was the influence on viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Mindset Series, Twitter, live streaming, and films? 5. How did SACE’s support for Niger Delta implementing partners contribute to their advocacy for economic reforms/growth in that region? EVIDENCE OF A PARTICIPATORY APPROACH The evaluation questions were compiled with input from PDG and the USAID/Nigeria Program Office with support from The Learning Program, which was implemented by DevTech Systems. The ET will interview direct and indirect beneficiaries in focus states. These include the Contractor staff, government officials in target line, ministries and departments, PIND, CSOs that have collaborated with or received grants from SACE, community leaders, benefiting communities, women and youth leaders, and media practitioners. There will be full participation and representation of all those interested in the evaluation process to promote ownership of results. EVALUATION METHOD AND METHODOLOGY USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 58 EVALUATION DESIGN This evaluation will use mainly a qualitative approach to gain insight into the questions above. The methodology will broadly consist of personal testimony of staff members, key government officials, and CSOs, and backed up by a thorough desk study of all relevant documentation. It is expected that a minimum of half of Nigeria’s six geopolitical zones will be visited (SACE partners have activities in 25 states that represent all six zones). DATA COLLECTION METHODS The key approaches that will be used to collect and analyze data for the evaluation are: 1. Background Materials Review: Prior to conducting field work, the team will review background materials, such as the Program Description and Contract Agreement, annual and quarterly reports, performance indicators, and other public documents related to the activity. The Mission will provide these to the team in advance of their arrival in Nigeria. 2. Team Planning Meeting: The team will conduct a one-day planning meeting in Abuja, before starting the evaluation. This meeting will: Review and clarify any questions on the evaluation SOW, evaluation questions, or evaluation report table of contents; draft an initial work plan, develop a data collection plan; clarify team roles; and assign drafting responsibilities for the evaluation report. PDG will participate in some sessions of the meeting, and its outcomes will be shared with the USAID/Nigeria. The team will share its work plan on the second day. 3. Key Evaluation Steps: a. Review activity documents, including the technical proposal, annual work plans, and annual and quarterly reports. b. Engage in a one-day team planning meeting, with the purposes outlined above. c. Present a work plan to the Mission for discussion/approval. d. Conduct field visits to activity implementation areas to review program activities and meet with key stakeholders, including: Individuals that participated in SACE trainings, advocacy, and other initiatives; beneficiaries in target communities; local government officials; civil society activists. e. Conduct interviews with key informants from USAID implementing partners, the USAID, government counterparts, donor organizations, CSOs, and others, as necessary. f. Prepare a presentation and debrief for the USAID/Nigeria with major findings and recommendations. g. Prepare a draft report for the Mission after field visits. h. Prepare a final report with an executive summary that includes main findings, conclusions, and recommendations for program improvements. 4. Data Analysis Methods: The ET will develop an analysis plan and review with the USAID/Nigeria for input. For each evaluation question, the ET will explain how evaluation data will be analyzed. Data tables, as agreed upon in the analysis plan, will be generated and analyzed. 59 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 5. Methodological Strengths and Limitations; The initial implementing partners’ capacity assessment will serve as a baseline assessment report against which progress will be measured. The SACE has also developed a Policy Harvester and Policy Tracker to measure the progress of partners’ advocacy. Some of the limitations of this evaluation include: • Quality of data available. Due to the many other activities by governments, CSOs, the private sector, and other development agencies in the 25 states in which SACE partners have activities, the attribution of a particular outcome to SACE-supported interventions is often challenging. • Recall bias among beneficiaries and other stakeholders. • Heavy reliance on key informants. • Other limitations, such as travel restrictions, which may affect the ET’s travel plans. EVALUATION PRODUCTS DELIVERABLES The ET will submit the following deliverables to the USAID/Nigeria. The timeline for the submission of the deliverables will be finalized and agreed upon during the team’s planning meeting: 1. Evaluation work plan and timeline. 2. Questionnaire/Guidelines for Conducting Key Informant Interviews/Focus Group Discussions: The ET will prepare these documents during the team’s planning meeting and submit them to the USAID/Nigeria for review and approval prior to the initiation of KIIs and site visits. 3. List of Proposed Sites and List of Respondents: Abuja FCT and the states of Bauchi, Osun, Imo, and Rivers have been chosen as sites to be visited. The USAID/Nigeria will support the process of producing a list of sites to be visited and a list of respondents to be interviewed as key informants. 4. Interview Notes and Completed Surveys: The ET will submit all the interview notes and completed survey instruments they will use for the evaluation to the USAID/Nigeria, together with the data they will use for the analysis. 5. Debriefing(s): The ET Leader will regularly debrief the USAID/Nigeria on the progress being made with the evaluation during fieldwork. At the end of the fieldwork and prior to departing Nigeria, the team will debrief the USAID/Nigeria on its findings, conclusions, and recommendations. 6. Draft Evaluation Report: The ET will provide the USAID/Nigeria with a draft report that includes all the components of the final evaluation report prior to their departure from Nigeria. The USAID/Nigeria will provide written comments on the draft report to the ET within ten working days of receiving the draft report. 7. Final Evaluation Report: The final report will address the comments provided by the USAID/Nigeria and other stakeholders on the draft report. The ET Leader will revise the draft report and deliver a final revised version to the USAID/Nigeria within three weeks of receiving USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 60 the USAID’s feedback. The final report in both hard and electronic format will be submitted to the USAID/Nigeria and approval given before submission to the Development Experience Clearinghouse (DEC). REPORTING GUIDELINES The USAID’s evaluation policy requires that all evaluation SOWs include the USAID’s Criteria to Ensure the Quality of the Evaluation Report (see USAID Evaluation Policy, Appendix I). The policy also indicates that the report will outline in detail any additional expectations USAID has regarding the structure, format, and length of a report. The format for the evaluation report is as follows (number of pages is illustrative): • Executive Summary (2 pp.). • Contents (1 p.). • Introduction (1 p.). • Background (2-3 pp.). • Methodology (1 p.). • Findings/Conclusions (17-20 pp.). • Issues and Challenges (1-2 pp.). • Recommendations/Future Directions (5-10 pp.) o Recommendations on program approaches and activities that could be scaled up or replicated, how and why. o Recommendations on other approaches and strategies to use for a new civil society advocacy activity. • References. • Annexes. • Data set. Draft reports will be submitted in two hard copies and one electronic copy. The final report will be submitted in five hard copies and one electronic copy. All reports will be in English. The report must: • Distinguish clearly between findings, conclusions (based strictly on findings), and recommendations (based clearly on the evaluation findings and conclusions). • Comply with the USAID’s Evaluation Policy. • Ensure submission to the DEC after finalization. DISSEMINATION PLAN It is expected that the USAID/Nigeria will plan a dissemination session with other technical units within the Mission, implementing partners, donor community as well other stakeholders. 61 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV TEAM COMPOSITION TEAM QUALIFICATIONS The USAID/Nigeria will engage the services of a team of two local and two international consultants to work on the ET. The ET will require: 1. An International Senior Evaluation Specialist/Team Leader (IQC Labor Category – Program Development, Senior) with experience in evaluating peace, democratic governance, and human rights activities in developing countries. The team leader will finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Nigeria. At least eight years of experience in evaluation management is required. Experience in conducting evaluations or assessments of peace and democratic governance as well as human rights programming in developing countries are required. Knowledge of legislative strengthening programming is desired. Ability to produce a high-quality evaluation report in English is essential. 2. A Deputy ET Lead with at least six (6) years of experience in democracy and governance programs and some experience managing or implementing peace and democratic governance as well as human rights programming in developing countries is required. Experience is preferred in assessing capacity-building efforts in nascent democracies, and the role of civil society in strengthening citizen participation and involvement in holding governments to account. African/regional experience is desired. Ability to conduct interviews and discussions and to write well in English is essential. 3. For the two local evaluators, a good knowledge of gender issues as well as social inclusiveness is required. A fair knowledge of the operations of both social and traditional media in developing countries will be an advantage. A Nigerian with adequate knowledge of the way the national and state assemblies work will act as Sector Specialist, and one of the two local evaluators could be a former Nigerian government official with experience interacting with civil society, given the focus of SACE on that relationship. Team member experience should include graduate level economics, social science, law, and/or public finance training. In-depth knowledge of issues relating to Nigerian legislative functions and process improvement programs, improving governmental institutional capacity, and fostering civil society oversight of government institutions in developing and/or transitional economies and democracies is required. Familiarity with legislative strengthening methods and programming is essential. Some experience in conducting evaluations or assessments is expected. Ability to conduct interviews and discussions and write well in English is essential. 4. Overall, the team will need expertise in the USAID’s practices and expectations in: Program evaluation; program design and analysis; quantitative and qualitative data collection and analysis; survey design and analysis; program issues, innovations, and challenges in building civil society and legislative capacity (including financial and personnel management, including value for money); the USAID’s practices and requirements in program performance measurement. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 62 CONFLICTS OF INTEREST All ET members will provide a signed statement attesting to a lack of conflicts of interest, or describing an existing conflict of interest relative to the activity being evaluated. The USAID/Nigeria will provide the conflict of interest forms. SCHEDULING AND LOGISTICS The proposed evaluation will be funded and implemented by the Mission through The Learning Program/DevTech. DevTech will be responsible for all offshore and in-country administrative and logistical support, including identification and fielding appropriate consultants. DevTech will arrange and schedule meetings, international and local travels, hotel bookings, working/office spaces, computers, printing, and photocopying. DevTech will make all logistical arrangements, including the vehicles for travels around Abuja and other cities and should not expect any logistical support from the Mission. DevTech will also make arrangements for team meeting space and equipment support for producing the report. 63 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV TIMELINE AND LOE TABLE (CONTINUED) Preparation of the draft evaluation report, preliminary findings/PPT Aug 28-Sept 2 6 6 6 6 TABLE 4. TIMELINE AND LEVEL OF EFFORT (LOE) Item Period of Performance Number of Days Team Leader/ Int. Evaluation Expert Deputy Team Leader Local Social Sector Expert Local Social Sector Specialist Review background documents and preparation work Preparation and submission of draft workplan and draft data collection tools (protocols) Aug 1-4 4 4 1 1 Workplan/Protocols submitted to the USAID for review Aug 5 1 1 USAID reviews and provides comments on workplan/ protocols Aug 7-9 ETintegrates USAID comments and submits to the USAID Aug 11 1 1 1 1 International travel to Abuja, Nigeria Aug 12 1 1 In-brief/Team planning meetings with the USAID Aug 14 1 1 1 1 Pretest data collection tools in Abuja Aug 15 1 1 1 1 Evaluators submit finalized data collection tools to USAID Aug 16 1 1 1 1 ET collects field data in Abuja Aug 17-19 3 3 3 3 Team splits into two subteams for data collection. Team A: North (Bauchi) and West (Osun via Abuja and Ibadan connecting flight) Team B: East (Imo) and South￾South (Rivers - by road) Aug 21-26 6 6 6 6 Return to Abuja Aug 26 USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 64 Submission of preliminary findings/PPT and report outline Sept 1 0 0 0 0 Presentation of preliminary findings, exit brief, and submission of outline Sept 4 1 1 1 1 Incorporation of USAID comments to the draft report Sept 5-6 2 2 2 2 International travel out of Nigeria Sept 7 1 1 0 0 Submission of draft evaluation to the USAID Sept 8 2 2 1 1 The USAID reviews and comments on draft evaluation report Sept 11-15 The team addresses USAID comments and finalizes the report due to the USAID/Nigeria Sept 18–Oct 20 12 7 1 1 TOTAL LOE 43 38 25 25 The USAID/Nigeria will provide the following support: • Supervise the monitoring and evaluation (M&E) unit and ensure that the M&E advisor complies with the job requirements. • Ensure that confidentiality of information is maintained and that files are well kept. • Ensure that copies of relevant documents are available to the ET. • Create an enabling environment for using and learning from activity data and for promoting data- and information-sharing among activity stakeholders and to interested external audiences. • Oversee and support the above and troubleshoot, as needed or requested. 65 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 2: EVALUATION TEAM The ET comprised the following: INTERNATIONAL SENIOR EVALUATION SPECIALIST/TEAM LEADER Sergio Diaz-Briquets, PhD Diaz-Briquets’ career has focused on the interdisciplinary approach to socioeconomic development issues in the developing world, while combining field experience, management, and research. He has directed complex management and research projects under contract or directly for several government Agencies, including the U.S. Congress, the U.S. Agency for International Development (USAID), and the U.S. Department of State (USDOS), engaged in the development of programmatic approaches, and has been a staff member or consultant to international development institutions. His experience includes having served as nonresident Chief of Party for USAID-financed projects in Guatemala, Honduras, and Nigeria and for Department of State Monitoring and Evaluation initiatives in Afghanistan and Lebanon. Diaz-Briquets has led or participated in Democracy, Human Rights, and Governance (DRG) and Anti￾Corruption assessments and evaluations in countries as diverse as Azerbaijan, Cambodia, El Salvador, Nigeria, and Panama. Many of these projects have given emphasis to activities designed to mobilize broad CSO coalitions to pursue and support governance reforms. These initiatives have often been implemented in the context of related technical support efforts to enhance the efficacy of official institutions. Sustainability has always been an underlying goal if CSOs are to continue engaging in their advocacy/reform role. CSO-related efforts have included provision of capacity building assistance to mobilize public support for reforms through the design and implementation of public awareness campaigns via community outreach programs and extensive mass media reliance. Diaz-Briquets received his doctorate from the University of Pennsylvania. SENIOR LOCAL EVALUATION SPECIALIST/DEPUTY TEAM LEADER Emmanuel Uche Emmanuel Uche comes with over 25 years of experience in the development sector cutting across institutional strengthening, governance and legislative strengthening, monitoring and evaluation, transparency and anti-corruption systems strengthening, civic mobilization, and civil society’s capacity building. Emmanuel recently successfully managed the anticorruption portfolio of the GBP53million, six￾year DFID-Nigeria supported Justice for All project. Currently a doctoral student of Development Studies, Emmanuel’s experience portfolio includes serving as National M&E Advisor, Adam Smith International's Growth and Employment in States (GEMS3) project in Nigeria; a former National Coordinator of Transparency International in Nigeria and a Democracy and Governance Program Manager at the USAID/Nigeria. Emmanuel has also served as the Regional M&E Specialist (Central and West Africa) for the National Democratic Institute, Washington D.C., and M&E Specialist for the State University of New York Legislative Strengthening Program for the Afghanistan National Assembly. His other relevant experiences include the Senior M&E Specialist for USAID/Nigeria Monitoring, Evaluation Management Systems (NIGERIA/MEMS) implemented by The Mitchell Group, Washington D.C. Earlier on, Emmanuel served as Program Officer for the CIDA’s Institutional Capacity Building project in Nigeria implement by CUSO/Nigeria. Emmanuel is the President of Development Measures, Abuja and has consulted for several international and local organizations including DFID, UNDP, USAID, The World Bank, IFAD, British Council, PIND/Chevron Foundation, and for several state governments and USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 66 other governance institutions in Nigeria, including the National Assembly and the Federal Ministry of Agriculture. LOCAL EVALUATION SPECIALIST Aliyu Aminu Ahmed Mr. Aliyu is a statistician with over 17 years of experience in development work focused on program management, monitoring and evaluation (M&E), qualitative research, advocacy, and systems strengthening. Mr. Aliyu has conducted several performance evaluations and CSO capacity assessments, and has worked as Monitoring and Evaluation Expert for USAID-supported projects, National Democratic Institute, and OTI. He has extensive experience in democracy and governance (legislative strengthening, elections, and anticorruption), economic growth, and the health sector. He conducted numerous consultancies for projects supported by USAID, OTI, GIZ, UNDP, MacArthur Foundation, DFID, World Bank, and GON. Mr. Aliyu is the current President of the Nigerian Association of Evaluators and the founder of Monitoring and Evaluation Institute. He has good technical writing and computing skills. LOCAL EVALUATION SPECIALIST Onyukwu Onyukwu Onyukwu Onyukwu is an evaluator, a development economist and associate professor in the Department of Economics, University of Nigeria, Nsukka with more than twenty years of professional experience. He teaches development economics at both undergraduate and postgraduate levels among other courses. He is also a senior research fellow at the Institute for Development Studies (IDS) of the same university. He has done extensive evaluation work in the area of public expenditures and public policy development. He has supported and continues to support several NGOs and NGO networks on community advocacies on understanding gender and elimination of harmful traditional practices (including FGM/C) and other strong gender norms. Recent evaluation assignments include a public expenditure management and financial accountability review in Ogun state and a study on status and implementation challenges of the budgets of the Federal Government , that he conducted for the Fiscal Responsibility Commission in Abuja. SUPPORT STAFF The Team was also supported by DevTech Staff, Dr. Emmanuel Adegbe and Ahmad Yahuza Getso. 67 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV ANNEX 3: EVALUATION WORK PLAN INTRODUCTION Initially launched in January 2014, The SACE program was designed to increase the capacity of CSOs to influence the development and implementation of key democratic reforms at the national, state, and local government levels. The SACE program was designed to supplement other USAID/Nigeria PDG programs, including the LEAD (Leadership, Empowerment, Advocacy and Development program) that encourages CSOs’ strategic networking at the subnational level. The SACE was launched after several assessments suggested the time was ripe for such an activity, despite several continued negative governance trends in Nigeria, such as the continued dominance of the national body politic by a small oligarchy intent on continuing to appropriate Nigeria’s wealth to the detriment of the majority of the population. A Mission strategic assessment, which was conducted in 2009, concluded that, despite this political imbalance, there was some room for reform as certain reform-oriented Nigerian policy-makers, with civil society support, could be encouraged to embrace a policy reform platform. This conclusion was buttressed by recommendations that were generated during the mid-term evaluation of a previous USAID CSO support endeavor–the ADVANCE, or Advocacy, Awareness and Civic Empowerment program. Indeed, this project proposed a new initiative to continue establishing and expanding Nigerian civil society coalitions by increasing partnerships with a wider array of sectoral and cross-sectoral CSOs. This initiative aimed to enhance their social appeal, by focusing on fewer and better targeted national level interventions, and by supporting their institutionalization and financial sustainability. A further fact￾finding Mission, that included Washington-based and Nigerian experts, arrived at similar recommendations, eventually leading to the design and procurement of the SACE initiative. The purpose of this external evaluation, focusing on five key questions, is to ascertain the effectiveness with which the SACE program has thus far been implemented, and in particular how its CSO coalition￾enhancing features have fared through the application of an innovative governance enhancing advocacy methodology. Aside from those general objectives, the evaluation is intended to propose mid-term corrections, if any, as well as to suggest potential follow-up activities. PROGRAM BACKGROUND SACE was awarded to Chemonics and is a five-year, $19.3 million program that includes support provided by PIND (the Chevron-affiliated Foundation for Partnership Initiatives in the Niger Delta) for activities to be conducted in this region of Nigeria. The four main objectives of the program were implemented by Chemonics largely through the application of a novel advocacy methodology that was developed by Root Change, a crucial partner on this endeavor. They are to: • Strengthen institutional, organizational, and technical capacity of CSOs (and their networks, coalitions, and clusters). • Strengthen partnership with government, discourse, and engagement on key issues. • Strengthen public awareness, discourse and engagement on key issues. • Strengthen capacity of BMOs and CSOs in the Niger Delta. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 68 The SACE, through its CSO advocacy strengthening efforts, seeks to overcome several contextual difficulties that have historically interfered with the emergence of a strong civil society capable of influencing the pace of democratic reforms. These constraints include the presence of complex and endemic social problems, isolated instances of advocacy activity and success, the limited sharing of advocacy skills, and the prevailing inability to take advantage of windows of opportunity. In order to overcome these shortcomings, SACE is utilizing a methodological approach based on what the international community has learned regarding the management of complex and demanding change processes. These lessons suggest that advocacy programs work best when they take in consideration the accountability context, blend supply and demand approaches, mobilize the actions of diverse actors around a common advocacy agenda, seek better fit, rather than best practice solutions, prioritize adaptive learning and innovation, and use robust PEA and explore windows of opportunity. Based on these findings and on the results of a two-year global study that Root Change conducted in 22 countries on behalf of the International Federation of the Red Cross (IFRC) to assess what capacities make for high performing, sustainable CSOs, two principal conclusions emerged. First, that aggressive “smart” networking with traditional and nontraditional partners creates high-performing CSOs, capable of overcoming challenging economic, social, and political country contexts. Secondly, that there are higher order, “2.0” capabilities (as opposed to “1.0” capabilities, normally associated with strong management procedures that are assumed to lead to impacts) that drive performance, such as CSOs’ ability to forge alliances, be a sector leader, and respond to constituents. These “second generation” practices, based on strong system level engagement, lead to policy impacts by promoting social capital development, embracing an ecosystem perspective, and giving pride of place to learning and adaptation. Thus, CSOs must engage in ecosystem mapping and adaptive learning, rely on feedback loops, and promote collective impacts. The end result is to align the actions of many different actors to have an impact that goes beyond what one organization can accomplish. According to SACE’s theory of change, transformational change requires incorporating the fundamental principles of effective voice and accountability programs, reaching beyond Capacity “1.0” inputs, generating sufficient public awareness and engagement, and the effective transfer of knowledge. This expected outcome requires an ecosystems approach whereby the objective is to influence clusters of partner organizations (CSOs, CBOs, policy-makers, donors, and companies) working together to create collective impacts. Such clusters are led by an anchor CSO whose role is to coordinate and align activities of cluster members around a common advocacy agenda and strategy (Lead from Behind), build effective, sustainable relationships with cluster members for issue-based advocacy, through the cluster and broader sector, and promote the success of cluster members, by linking them to each other, as well as to opportunities and resources. As such, the model leads to collective impacts. Aside from the anchor/cluster feature, the Root Change-developed methodology rests on the application of the STAR, a tool suite and change process devised to promote higher, 2.0 organizational performance. It includes eight drivers that were designed to impel the collective impact principles of the SACE advocacy approach, whose purposes are for clusters to work from a common agenda, engage in complementary activities, further constant communication, and share measurement of results. The STAR drivers in question consist of adaptive strategy and tactics, stakeholder engagement, monitoring and evaluation, member and partner development, knowledge exchange, alliance building, innovation, and promotion of public awareness. The STAR assessment tool is complemented by four other tools, which are designed to improve the effectiveness of clusters. These tools are: The STARNET, used to map linkages among advocacy clusters; STAR Feedback, to facilitate communications among cluster 69 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV members; Star Cluster Coaching, the regular provision of TA to support cluster activities; the Annual Progress Index, an annual evaluation of cluster member achievements along the range of STAR performance drivers. A final capacity-building feature is the ALS, an Abuja event that is attended by anchor and cluster members and convened to review advocacy strategies and successful advocacy initiatives. The implementation of the SACE activity began in FY 2014 with the selection of partners and the initial establishment of advocacy clusters (in Inclusive Education, Open Budgeting, Youth Advocacy, Inclusive Health, Peace Advocacy, and Transparency in Extractives). The SACE began with 16 CSO clusters, eight each at the national level and in the Niger Delta. However, in 2016, SACE ended its relationship with one of the national anchors, due to evidence of fraudulent financial practices, and in September 2017 SACE suspended one of the Niger Delta anchor sub-grants with conditions to address similar practices. Thus, by the end of September 2017, SACE was still working with a total of 16 clusters, seven of them national plus nine more in the Niger Delta, including two BMOs, with two new Niger Delta region clusters having joined the SACE initiative in 2017. In FY 2015 and FY 2016, SACE began to emphasize capacity building and the gradual design and implementation of advocacy initiatives, including preliminary results. In FY 2017, the consolidation of several anchor CSOs became increasingly sustainable (sustainability being defined as having the capacity to directly manage USAID grants) and advocacy capabilities among many cluster organizations also continued to strengthen. At this stage of the activity cycle, emphasis is on attainment of concrete advocacy results. Results have been encouraging, as in March 2017 the Mission and SACE decided to add a new outcome indicator as part of establishing the Mission’s PMP. Some SACE-associated CSOs have qualified for grants that were awarded by other donors, thanks to their enhanced 1.0 capabilities. These SACE-associated CSOs recorded numerous achievements in unique persons who trained in various advocacy skills. Also, they initiated multiple activities in favor of marginalized groups (defined as women, youth and persons with disabilities, all of whom achieved gains in terms of sensitivity and inclusion). In addition, they introduced innovative cluster coaching practices and were able to link capacity building to results (via the Annual Learning Summit). Furthermore, SACE takes credit for the promotion of 75 bills and policies at the various government levels promoted by its clusters, as well as for the participation of many CSO stakeholders in governance oversight processes. Cross cluster collaboration has led to several government action demands, that are concurrently underway. In addition, SACE-affiliated organizations have been involved with several public events, such as the “1,000 Person March.” Many of these efforts would not have been effective without SACE’s insistence on intense public advocacy, whether through traditional media or newer social media. Twitter exposures alone currently approach 50 million. Other innovative practices include engaging the entertainment sector in the Homevida Integrity Film Awards and supporting digital storytelling in Spaces for Change and the Nigeria Women’s Trust Fund idea for young women participating in politics. EVALUATION PURPOSE AND AUDIENCE This mid-term program evaluation is primarily geared to decision-making, mainly to bolster program implementation support and improve program activities with a view to promoting the overall achievement of SACE’s goals, suggest changes if necessary, and improve program implementation procedures for SACE to become more effective in pursuing its objectives. Input for the evaluation is also expected to be considered as the Mission debates whether to fund a follow-up activity. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 70 The primary audience for this evaluation is the PDG Office at the USAID/Nigeria Mission, PIND funding partner, other Mission staff, USG personnel from other agencies, as well as other non-USG donor agencies, GON officials, and the Nigerian CSO advocacy community. The evaluation dissemination plan includes an ET team presentation of evaluation results to be made at the 2017 SACE ALS, which is scheduled in Abuja on September 18 and at which all anchor CSOs and many of their associated cluster members are expected to participate. EVALUATION QUESTIONS The ET was asked to address five specific questions, as determined by the Mission’s PDG team: 1. To what extent was the design and implementation of SACE successful in promoting achievement of the activity’s goal? • Did SACE’s anchor/cluster model work effectively? • Is SACE sufficiently focused on key results in its approach? 2. What worked in SACE’s efforts to increase the financial and technical (e.g., in terms of advocacy) capacity of targeted CSOs/clusters to enable them to engage with the government on reforms? What did not work? • Was the Systems for Transformation and Results (STAR) an effective tool in that regard? • Are at least three SACE anchor partners ready to receive direct funding from USAID? If partners are not ready to receive direct funding, are they likely to be ready by the end of calendar year 2017? 3. What impact did SACE partners have in influencing government policy development and implementation regarding vulnerable populations such as women, youth and persons with disabilities? 4. Did SACE’s activities for public awareness contribute to strengthening public discourse and support for key democratic governance issues? • In particular, what was the influence on viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Mindset Series, Twitter, live streaming, and films? 5. How did SACE’s support for Niger Delta implementing partners contribute to their advocacy for economic reforms/growth in that region? EVALUATION TIMELINE Although the evaluation formally began on August 1 with a review of program documents, development of a timeline, firming-up travel plans, and preliminary design of field research instruments, its field research phase did not get underway until the arrival of the team leader in Abuja on August 13. Following an in-brief meeting with the PDG team at the USAID Mission on Monday, August 14, and up to Friday, August 19, the ET reviewed SACE-related documents, finalized and pre-tested field research instruments, proceeded with the selection of field sites and target key informants to interview, and conducted several KI interviews in the FCT of Abuja. At this stage, the ET had been divided into two sub-teams, the first to visit the states of Bauchi and Kaduna in the north and from Kaduna travel to 71 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Lagos. The second team was to go to the states of Rivers, Imo, and Edo in the Niger Delta region. Both teams departed for their destinations the morning of Sunday, August 20, with sub-team one going to Bauchi by road arriving in the afternoon. Sub-team two flew directly from Abuja to Port Harcourt. After several KIIs and a FGD in Bauchi, on the morning of August 22, sub-team one departed Bauchi for Kaduna and arrived in the late afternoon. Two days later, and following more KIIs and another FGD in Kaduna, on Wednesday, August 25, sub-team one flew to Lagos where they met with other key informants, and also had to go on an unanticipated side visit to Ibadan, as several key informants who had promised to meet the ET members in Lagos were unable to travel. On Saturday, sub-team one flew back from Lagos to Abuja. In the interim, sub-team two began their field work in Port Harcourt, Rivers state, on August 21, traveling by car from there to Owerri, Imo state, and from there to Benin, Edo state. From Benin, sub-team two returned to Abuja, Upon their return to Abuja, the two field research sub-teams came together, began reviewing field results, and decided on writing assignments. By Thursday, August 31, they had completed a preliminary version of the report, exclusively addressing findings, conclusions, and recommendations. They sent this draft to DevTech’s headquarters in Arlington for review; then, the sub-teams forwarded a revised copy to the PDG team on Saturday, September 2. The Mission was to provide feedback by Monday, September 4, the ET then being expected to proceed with a further revision of the draft report. This was to be followed by a presentation of the evaluation results to the PDG Office and other Mission staff during an exit briefing that was scheduled for Wednesday, September 6. The Team Leader was to depart Abuja on Thursday, September 7, with the local ET members scheduled to present evaluation findings at the SACE ALS that was planned in Abuja for September 19. Plans also called for conducting several other KIIs during the ALS. Following this conference, concluding refinements were to be made to the evaluation report, a final version being forwarded to the Mission and the USAID Information Clearinghouse. EVALUATION DETAILS AND METHODOLOGY The evaluation relied on a mixed-methods approach that included several qualitative data collection techniques and analysis of SACE management and administrative reports, including the program’s Performance Management Plan (PMP) and Root Change annual reports focusing on the CSO STAR Index and Net Capacity Change assessments. Data collection methods included: • Desk review of SACE management and administrative documents. • Key Informant Interviews (KIIs): With a total of 38 interviews (in some cases, more than one KI was present as some interviews took place in small group settings although focusing on the primary KI), both in Abuja and field visit sites elsewhere in the country. • Focus Group Discussions (FGDs): With a total of four FGDs with the participation of CSO cluster members in field sites outside Abuja. • Field visits and activity observation: Following initial field work preparations in Abuja, as well as the conduct of initial KIIs in the FCT, the ET team divided into two sub-teams for separate visits to selected grantees in different geopolitical regions of Nigeria. Sub-team one went to Bauchi, Kaduna, Oyo, and Lagos states, while sub-team two traveled to Rivers, Imo, and Edo states, all in the Niger Delta region. While the sub-teams were in these sites, they were able to observe the following SACE-related activities: USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 72 o In Kaduna, a CSO advocacy visit to Dr. Paul M. Dogo, Commissioner of Health, Ministry of Health, Kaduna State, regarding the release of 45 percent of the 2017 budget allocation for family planning in the state. o In Lagos, an anchor/cluster workshop on education accountability, with the participation of multiple stakeholders, organized and anchored by HDI. o In Edo, an anchor/cluster routine review meeting on effective agricultural policies, anchored by ESCFAL in Benin City Table 5 summarizes the schedule of KIIs and FGDs, indicating when and where they were conducted. TABLE 5. SUMMARY OF INTERVIEWS AND FOCUS GROUPS Location Dates KII FGDs Subtotal Akwa Ibom State September 20, 2017 Policy Alert – Social Action Cluster members 2 Bauchi, Bauchi State August 20, 2017 Policy Maker – Education 4 August 20, 2017 Anchor - CITAD August 21, 2017 BACIPEM August 21, 2017 Policy Maker – Bauchi House of Assembly Kaduna, Kaduna State August 23, 2017 Vision Trust Foundation – CENSOJ Cluster member 5 August 22, 2017 Advocacy Nigeria – HERFON Cluster member August 22, 2017 Health Advisory Net – HERFON Cluster member August 22, 2017 RHAN – HERFON Cluster member August 23, 2017 Policy Maker – Health Lagos, Lagos State August 24, 2017 OSCEF – HDI Cluster Member 3 August 24, 2017 HDI – Anchor August 25, 2017 EiE - ISWF 73 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Ibadan, Oyo State August 25, 2017 WARDC – Cluster member 2 August 25, 2017 JONAPWD – Cluster member Port Harcourt, Rivers State August 21, 2017 SDIC (Social Action) – Anchor 4 August 21, 2017 BANGOF – Cluster member August 21, 2017 PIND Staff (Port Harcourt) August 21, 2017 CVI – Cluster member Owerri, Imo State August 22, 2017 CCIDESOR - Anchor CCIDESOR – Cluster members 6 August 22, 2017 Critical Response for Universal Xpression (CRUX) – Cluster member August 22, 2017 Policy Maker – Budget Office August 22, 2017 Policy Maker – Legislature August 22, 2017 Centre for Rural Education and Social Development – Cluster member Benin, Edo State August 23, 2017 ANEEJ - Anchor ANEEJ – Cluster members 7 August 24, 2017 ESCFAL - Anchor Policy Makers – Edo State Ministry of Agriculture August 24, 2017 Association of Women Farmers in Nigeria – Cluster member August 24, 2017 Nigeria Agricultural Students on Extension Practices (NASEP) – Cluster member Anchor/Cluster members meeting Abuja, FCT August 15 2017 SACE COP (presentation) SACE Staff (Discussion) 9 August 18, 2017 SACE COR USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 74 All KII and FGD participants were informed about the objectives of the mid-term evaluation and asked to verbally consent to participate and promised confidentiality. A sample copy of the Informed Consent statement and KII and FGD guides the ET used during the field work phases of the evaluation are shown in the following, in this annex. SAMPLING Field evaluation sites were chosen according to regional guidelines provided in the Mission’s SOW for the evaluation. The ET selected the KIIs and FGDs participants independently, without input from either the Mission or the SACE program staff, following a purposive sampling approach. At each field work site, including Abuja, the ET draw key informants from various advocacy sector areas and chose them to represent three respondent categories: Anchor CSO representatives, cluster CSO representatives, and policy-makers. All respondents were familiar with the SACE program. The ET was sensitive to gender concerns in setting up meetings. All FGD participants represented cluster CSOs. ANALYSIS About 95 percent of the KIIs were recorded and later transcribed independently by each ET member. Each team compared notes and harmonized findings. Analysis of evaluation questions were allocated to individual team members, each being primarily responsible to address one, apart from the Team Leader, who was responsible for two of them. by The ET as a whole discussed and reviewed preliminary draft responses to evaluation questions and then submitted them for further review to the Team Leader. The ET organized the findings into themes, with further clarifications gleaned from SACE-provided administrative documents. Following the USAID’s initial review of the draft evaluation report, subsequently the ET revised it twice. In this way, they could take into account Mission comments and recommendations, and then they could add further insights from the participation of two ET members in the four-day Third Annual Learning Summit, in Abuja, September 19-22, 2017, during which they had the USAID August 18, 2017 HERFON – Anchor August 18, 2017 CISLAC – Anchor August 18, 2017 ZCC – CISLAC Cluster member August 18, 2017 Social Action (SDIC) – Anchor August 18, 2017 CENSOJ – Anchor September 20, 2017 PIND Staff September 20, 2017 YAF – Cluster member GRAND TOTAL 42 75 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV opportunity to interact with anchor/cluster CSO representatives. Input for the final revision was also sought from SACE and PIND staff. BIASES AND OTHER LIMITATIONS Major evaluation limitations included the much-compressed period of time over which the evaluation was conducted, particularly given the challenging logistics conditions associated with traveling to different sites in Nigeria, potential selection biases due to the purposive sampling approach followed, and the limited number of KIIs the ET was able to conduct with policy-makers, as arranging interviews with these busy individuals generally requires more time than the ET had. Despite this concern, the ET team was able to complete KIIs with most planned respondents. The ET acknowledges that, since all key informants, other than the policy-makers, had direct SACE ties, whether as representatives of CSO grant recipients or as beneficiaries of SACE capacity-building activities, the methodology may have been impacted by a response bias (i.e., desirability bias) with respect to findings and conclusions. Also, recall biases might be present, as when a SACE-unrelated event may be attributed to the program (a possibility a respondent may erroneously attribute an achievement to SACE). These risks were minimized, however, by the ET clearly explaining the purpose of the evaluation, and by repeatedly asking key informants and FGD participants to clarify uncertain responses. USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 76 ANNEX 4: DATA COLLECTION INSTRUMENTS GENERAL INSTRUCTIONS AND INFORMED CONSENT Thank you very much for setting aside time to talk with me/us today. I/we work for DevTech, a consulting company based in the United States that has been contracted by the USAID to conduct a mid-term performance evaluation of SACE’s work towards strengthening civil society’s ability to influence the development and implementation of key democratic reforms at the national, state, and local levels. The purpose of this mid-term performance evaluation is to understand what has changed in the past few years and progress made in terms of voice, accountability and democratic reforms. This is not an evaluation of your organization or your work. For this evaluation, we are conducting interviews and FGDs in ten (10) out of twenty-three (23) focal states (including FCT) with a wide variety of COSs, government officials, media and so on. The states covered in the evaluation are Akwa Ibom, Bauchi, Delta, Edo, FCT, Imo, Kaduna, Lagos, Osun, and Rivers State. Before I/we begin, I/we want to let you know that any information or examples we discuss during this interview will not be attributed to any specific person or institution. All quotes used in the mid-term performance evaluation report will be attributed to a general stakeholder group (e.g., media, Federal Government, state government, LGA, civil society organization and so on), not by individual (unless authorized to do so) and all identifying information will be removed. You are free not to respond to any of our questions or to stop the interview at any time. The interview will take about one hour. If you don’t mind, I would like to record this conversation, solely for the purposes of listening attentively now and taking notes later. Is that all right? [BEGIN RECORDING] Before I/we begin, what questions do you have about this interview? Date: Location: Name of Respondent: Sex Title/Position: Email /Phone no: Interview Type: Verbal Informed Consent? Verbal consent to take pictures? Interviewer: Note takers: 77 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV SCENE SETTING • Please tell us a bit about your organization, history, mission, focal areas, membership, and size. • When did you join the SACE team as an Anchor or Cluster? • What were your core activity/issue areas before joining the SACE team and what was your depth of engagement as well as technical knowledge on the issue area? • What about your organizational capacity before joining SACE? USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 78 #1 SEMI-STRUCTURED KEY INFORMANT INTERVIEW (KII) GUIDE - NATIONAL ANCHOR/CLUSTER CSO REPRESENTATIVES Mid-term Performance Evaluation Strengthening Advocacy and Civic Engagement (SACE) Introduction to the interview 1. What was your engagement with increasing government transparency and accountability in the past few years? Collaborations/Strategic partnerships between civil society coalitions and networks/SACE (formal or informal) 1. How much support do you receive from SACE (financial/technical)? 2. How are you applying STAR skills in your advocacy work? If necessary, explain what the STAR is. 3. Since your involvement with SACE, has your organization improved its advocacy and civic engagement procedures? If so, could you explain how it has improved? 4. What enabled your collaboration and coordination efforts in the past few years? Hindered them? 5. If your organization partnered with other CSOs, did your partnership change the nature of your advocacy and monitoring efforts regarding increased transparency, accountability, and responsiveness in government institutions? If so, how? Collaboration and coordination: Increased engagement and efficacy of civil society to be able to influence public institutions whose function it is to serve citizens’ interests 1. With whom (CSOs) did you work with on accountability in the past few years? 2. Since your involvement with SACE, has the nature of your relationship with other like-minded CSOs changed? If so, could you explain how? 3. How did you engage the media? (Probe if they ever use information from media.) 4. How did you engage marginalized populations, such as women, youth, and the disabled in the process? Effectiveness of the anchor cluster model (voice and accountability) 1. In your advocacy role, did your CSO initiate the effort(s) or follow the lead of other CSO(s)? 2. If your CSO followed the lead of another organization, do you recall which organization? 3. Have you been pleased with the technical support provided by mentors, provided by SACE Program Advisors and mentors in workshops, or available on demand (by phone, online, etc.)? Could you comment on the strengths and weaknesses of the various technical support mechanisms? 79 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV Increased CSO engagement with public institution 1. How did you collaborate with the government in the past few years? (Probe for specific joint actions)? Did you do it in conjunction with other CSOs? 2. Could you tell us to which government institution were your efforts directed at? 3. Has your CSO advocated for policy, regulatory, or administrative procedures, already enacted or under development, since 2004? How? 4. How has your organization (or how have CSOs) facilitated policy passage, modification, or implementation? Development and implementation of democratic reforms (policies) at the national, state, and LGA levels 1. In the past few years, what accountability activities were the same as before? What was different? Policy effectiveness 1. What was the same and what changed in the democracy and governance policy environment in the past few years? 2. Did these efforts result in increased government institutional responsiveness? If so, could you elaborate? If not, to what reasons do you attribute the lack of results? 3. How did CSOs influence legislation? 4. To what factors do you attribute CSO influence? Concluding questions 1. To successfully impact government accountability in the current context, what needs to be done more of, less or differently? 2. What else would you like to tell me/us, but you didn’t because I/we didn’t ask the right question? 3. Do you have any questions for me? USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 80 #2 SEMI-STRUCTURED KEY INFORMANT INTERVIEW (KII) GUIDE - NIGER DELTA ANCHOR/CLUSTER CSO REPRESENTATIVES Mid-term Performance Evaluation Strengthening Advocacy and Civic Engagement (SACE) Introduction to the interview 1. What was your engagement with increasing government transparency and accountability in the past few years Your work under SACE 1. What was your engagement with efforts to improve the government’s responsiveness and/or economic policy reform in the Niger Delta in the past few years? 2. What specific capacity-building support did you receive under the SACE interventions? 3. How adequate was this support? (financial/technical) 4. How did the Anchor-Cluster Model help or impact what you do? (Like improving partnership, leveraging resources, increasing collective actions, etc.? Any specific examples?) 5. What level of training did you have on the STAR model? How are you using it? 6. How are you applying STAR skills in your advocacy work? Elaborate, if necessary. 7. Since your involvement with SACE, has your organization improved its advocacy and civic engagement procedures? If so, could you explain how they have improved? 8. What enabled your collaboration and coordination efforts in the past year? Hindered them? 9. If your organization partnered with other CSOs, did your partnership change the nature of your advocacy and monitoring efforts regarding increased transparency, accountability, and responsiveness in government institutions? If so, how? Collaboration and coordination 1. How effective were your cluster/BMO’s advocacy engagements on key economic policy reform issues been before the SACE intervention? How effective have they been after the SACE intervention? 2. Were there noticeable changes in the conditions in your work environment over the last few years? Can you give us specific instances of such changes/improvements? Did your work influence or impact on those changes in any way? How do you know that? 3. How would you describe your relationship with key government officials or institutions that you have engaged with? Have they become more receptive or less receptive? What do you think has contributed to this? Can you give specific instances of increased or decreased receptiveness? 4. To what extent did SACE partners successfully influence policy formulation and implementation? 5. How did you engage the media (probe if they ever use information from media)? 81 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 6. How did you engage marginalized populations, such as women, youth, and the disabled in the process? Review of the anchor cluster model (CSO voice and accountability) 1. In your advocacy role, did your CSO initiated the effort(s) or followed the lead of other CSO(s)? 2. If your CSO followed the lead of another organization, do you recall which organization? 3. How did you collaborate with the government in the past year? (Probe for specific joint actions)? Did you do it in conjunction with other CSOs? 4. Could you tell us at which government institution were your efforts directed to? 5. Has your CSO advocated for policy, regulatory, or administrative procedures, already enacted or under development, since 2014? How? 6. How did your organization (or how have CSOs) facilitate policy passage, modification, or implementation? Economic reform in the Niger Delta 1. Did the SACE interventions have concrete impact on economic activities in the Niger Delta region? If so, could you site a specific example? How did it contribute to the enhancement of social welfare? 2. If the economic impact was not anticipated, to what factors would you attribute this result? (probe; more time needed for reforms to have intended results, insufficient technical assistance, lack of financial resources …) Concluding questions 1. What else would you like to tell us that may be useful to improve results achievement of the SACE program? Do you have any questions for us? USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 82 #3 SEMI-STRUCTURED KEY INFORMANT INTERVIEW (KII) GUIDE - SACE PROGRAM ADVISORS/MENTORS Mid-term Performance Evaluation SACE Background 1. Could you please introduce yourself and your role in the SACE program? 2. Tell us a little about your professional background (probe for specific technical capacity in Organizational Development/CSOs) Capacity building by root change 1. Were you trained by Root Change on the STAR (probe for performance areas such as: communication effectiveness, impact assessment, adaptive planning, people development, analysis, and organizational learning) Engagement of CSOs on STAR 1. How are your training/advisory sessions organized? 2. How is the technical assistance provided to Anchors/Clusters? (probe for workshops, participations, internet-based, Root Change procedures) 3. Do you provide follow-up to clusters downstream training or is the follow-up provided by anchors themselves? Monitoring 1. How is capacity acquisition (Anchors/Clusters) measured? (probe for use of Root Change methodology? Others?) Determinants for results 1. Do you provide additional technical assistance following periodic capacity-building activities? 2. How do you determine the success of your technical assistance to Anchors/Clusters? Technical assistance effectiveness 1. Do you feel that your effort has had an impact on the national voice, transparency, and accountability landscape? Concluding questions 1. What else would you like to tell me/us, but you didn’t because I/we didn’t ask the right question? 83 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 2. Do you have any questions for me? USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 84 #4 FOCUS GROUP INTERVIEW GUIDE - CLUSTER MEMBER CSO REPRESENTATIVES Mid-term Performance Evaluation SACE Introduction to the interview 1. What was been your engagement with increasing government transparency and accountability in the past few years? Collaborations/Strategic partnerships between civil society coalitions and networks/SACE (formal or informal) 1. How are you conducting your cluster activities? (probe for how often the CSOs meet? How are issues prioritized?) 2. What role is the Anchor playing in your cluster activities? 3. Have you all been trained on the STAR? If necessary, explain what STAR means. How are you applying STAR skills in your advocacy work? Are there changes in the advocacy and civic engagement process as a result of applying STAR skills? How did STAR training impact your networking skills? Effectiveness of the anchor cluster model (voice and accountability) 1. In your advocacy role, did your CSOs initiate the effort(s) or followed the lead of other CSO(s)? If your CSO followed the lead of another organization, do you recall which organization? 2. Were you satisfied with the technical support provided by mentors? Could you comment on the strengths and weaknesses of the various technical support mechanisms? (probe for effectiveness of training by Program Advisors/mentors/consultants) 3. How effective is the Anchor/Cluster model for CSO advocacy? What are the challenges experienced in using the model? Collaboration with marginalized groups 1. How are you engaging marginalized populations, such as women, youth, and the disabled in your cluster activities/processes? 2. Are you engaging the media in your advocacy for marginalized groups? Increased engagement and efficacy of civil society to be able to influence public institutions whose function it is to serve citizens’ interests 1. How do you advocate to the government? 2. Has your CSOs advocated for policy, regulatory, or administrative procedures, already enacted or under development? How? 85 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 3. How did your organization (or how did your cluster CSOs) facilitate policy passage, modification, or implementation? Policy effectiveness 1. Were there any major changes/advances in economic and democratic reforms in the past few years? 2. Did these efforts result in increased government institutional responsiveness? If so, could you elaborate? If not, to what reasons do you attribute the lack of results? Policy? Concluding questions 1. To successfully impact government accountability for in the current context, what needs to be done more of, lessor differently? 2. What else would you like you tell me/us, but you didn’t because I/we didn’t ask the right question? 3. Do you have any questions for me? USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 86 #5 FOCUS GROUP INTERVIEW GUIDE - NIGER DELTA CLUSTER MEMBER CSO REPRESENTATIVES Mid-term Performance Evaluation SACE Introduction to the interview 1. What was been your engagement with increasing government transparency and accountability in the past few years? Collaborations/Strategic partnerships between civil society coalitions and networks/SACE (formal or informal) 1. How are you conducting your cluster activities (probe for how often the CSOs meet or interact with each other? How are issues prioritized?) 2. What role is the Anchor CSO playing in your cluster activities? 3. Were you all trained on the STAR? Explain, if necessary. How are you applying STAR skills in your advocacy work? Are there changes in the advocacy and civic engagement process as a result of applying STAR skills? Effectiveness of the anchor cluster model (voice and accountability) 1. In your advocacy role, did your CSO initiate the effort(s) or followed the lead of other CSO(s)? If your CSO followed the lead of another organization, do you recall which organization? 2. Were you satisfied with the technical support provided by SACE program advisors/mentors? Could you comment on the strengths and weaknesses of the various technical support mechanisms? (probe for effectiveness of training by Program Advisors/mentors/consultants) 3. How effective is the Anchor/Cluster model for CSO advocacy? What challenges did you experience in using the model? Collaboration with marginalized groups 1. How are you engaging the media in public awareness? (probe for traditional media and social media) 2. How did you engage marginalized populations, such as women, youth, and the disabled in your cluster activities / process? Increased engagement and efficacy of civil society to be able to influence public institutions whose function it is to serve citizens’ interests 1. How do you advocate to the government? 2. Did your CSOs advocate for policy, regulatory, or administrative procedures, already enacted or under development? How? 87 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 3. How did your organization (or how have your cluster CSOs) facilitate policy passage, modification, or implementation? Policy effectiveness 1. What was the same and what changed in economic and democratic reforms in the past year? 2. Did these efforts result in increased government institutional responsiveness? If so, could you elaborate? If not, to what reasons do you attribute the lack of results? Concluding questions 1. To successfully impact government accountability for in the current context, what needs to be done more of, less or differently? 2. What else would you like to tell me/us, but you didn’t because I/we didn’t ask the right question? 3. Do you have any questions for me? USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 88 #6 SEMI-STRUCTURED KEY INFORMANT INTERVIEW (KII) GUIDE - POLICY-MAKERS Mid-term Performance Evaluation SSACE Policy-makers’ perception of CSOs 1. In your opinion, what is the role of CSOs in Democracy and Governance (accountability and reform) issues? 2. Are you aware of issue-based CSO clusters working in your thematic area in the state? What was the role of these CSO clusters? 3. Can you mention the policies that SACE-supported CSOs were able to influence (whether in the formulation, legislative consideration/passage, or implementation in your state? (probe to determine which policy or policies the policy maker feels are most important) Engagement between CSO-led coalitions and networks and targeted GGON institutions 1. Are you aware of USAID/PIND/SACE-supported CSOs groups’ public awareness activities in the state? In your opinion, did they contribute to strengthening public discourse and support for key democratic governance issues? 2. How did CSOs influence your organization in terms of improved responsiveness, transparency and accountability issues in the past year? (probe for names of CSOs or networks) 3. How receptive was/were your organization/yourself to disseminate transparency, accountability, and good governance messages? Public awareness, discourse, and support for key democratic governance issues, such as transparency, accountability, and good governance 1. Do you find coordinated CSOs efforts in favour of democratic governance reforms effective? Why or why not? 2. How did the implementation of governance reforms policy/policies change in the past few years? 3. What was the influence on policy-makers viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Capacity, Twitter, live streaming, and films? 4. What advocacy approaches do you find most convincing when deciding to endorse or not a given democratic governance reform? Policy development and implementation 1. To what extent did SACE partners successfully influence government policy development and implementation? What was/were the key/thematic areas? Can you give us specific example of how CSO inputs have influenced/impacted your policies? 2. In your opinion, did CSO contributions improve policies at the national, state or local level? Could you give us a specific example in your area of responsibility? 89 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 3. How effective or constructive were the CSO’s engagements? Any suggestions for improving civic advocacy and effectiveness in the future. 4. What gaps are there in current policy? Engagement of marginalized populations, such as women, youth, and the disabled in the CSO advocacy process 1. When CSOs conduct their advocacy activities to policy-makers, do they involve marginalized groups? How? Concluding questions 1. To successfully impact government transparency and accountability in the current context, what needs to be done more of, less or differently? 2. What else would you like to tell me/us, but you didn’t because I/we didn’t ask the right question? 3. Do you have any questions for me? USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 90 ANNEX 5: LIST OF INDIVIDUALS CONTACTED USAID/NIGERIA 1. Steve Haykin, Mission Director 2. Erin Holleran, Deputy Mission Director 3. Blair A. King, Director, Office of Peace and Democratic Governance 4. Adamu Igoche, Deputy Director, Office of Peace and Democratic Governance 5. Audra Lykos, Democracy and Governance Officer, Office of Peace and Democratic Governance 6. Augusta Akparanta-Emenogu, Civil Society and Media Specialist, Office of Peace and Democratic Governance SACE 1. Charles Abani, Chief of Party PIND 1. Dara Akala, Executive Director 2. Tunji Idowu, Deputy Executive Director 3. Bose Eitokpah, Capacity Building Program Manager CLUSTER MEMBERS AND ANCHORS 1. Amina S. Kazaure, Vision Trust Foundation 2. Isah Gidado, Advocacy Nigeria (AN) 3. Hauwa L. Saulawa, State Coordinator, Advocacy Nigeria (AN) 4. Hajiya Usman, Advocacy Nigeria (AN) 5. Reverend David Ugolor, Executive Director, Africa Network for Environment and Economic Justice (ANEEJ) 6. Chief Janet Olaleye, President, Women Lead Agriculture, All Farmers Association of Nigeria 7. Joan Obaze, National Coordinator, Association of Women Farmers in Nigeria 8. Robinson Baye, Executive Director, Bayelsa Non-Governmental Organizations Forum (BANGOF) 9. Mgbemena Obidi, Executive Director, Citizens’ Centre for Integrated Development and Social Rights (CCIDESOR) 10. Christian Okororie, Executive Director, Centre for Rural Economic and Social Development 11. Eze Onyekpere, Executive Director, Centre for Social Justice (CENSOJ) 91 | SACE MID-TERM PERFORMANCE EVALUATION REPORT USAID.GOV 12. Kolawole Banwo, Civil Society Legislative Advocacy Centre (CISLAC) 13. Chinedu Bassey, Civil Society Legislative Advocacy Centre (CISLAC) 14. Kabiru Saidu Dakata, Senior Program Officer, Centre for Information Technology and Development (CITAD) 15. Sebastin Kpalam, Executive Director, Citizens’ Voice Initiative (CVI) 16. Adebisi Alokolaro, Assistant Media Manager, Enough Is Enough (EiE) 17. Nosa Collins, President, Edo State Co-Operative Farmers Agency Limited (ESCFAL) 18. Olayemi Samuel, Research and Program Officer, Human Development Initiatives (HDI) 19. Dr. Emmanuel Abanida, Health Reform Foundation of Nigeria (HERFON) 20. Olu Kayode, Joint National Association of Persons with Disability (JONAPWD) 21. Victor Oliseh Nelson, Founder, Nigeria Agriculture Student and Extension Practice (NASEP) 22. Olusola Babalola, National Coordinator, Nigeria for Change Initiative 23. Kalani Akeem Tolu, Executive Director, Out of School Children Empowerment Foundation (OSCEF) 24. Ebong Ekanem, Policy Alert 25. Tijah Bolton-Akpan, Policy Alert 26. Mariam J. Ahmad, Chairperson, Reproductive Health Advocacy Network (RHAN) 27. Ken Henshaw, Senior Program Manager, Social Action 28. Botti Isaac, Program Officer, Social Action 29. Dr. Uduak Okon, Youth Alive Foundation (YAF) 30. Anih Ndidi Patience, Intern, Zero Corruption POLICY-MAKERS 1. Babayo Yahaya, Secretary, Education Committee, State House of Assembly, Bauchi State 2. Yusuf Yerima Gamawa, former Permanent Secretary, Ministry of Education, Bauchi State 3. Emmanuel Orie Chigozie, Member (representing Ohaji/Egbema Local Government Area), State House of Assembly, Imo State 4. Victor Okereke, Director of Budget and Economic Planning, Imo State 5. Dr. Paul M. Dogo, Commissioner, Ministry of Health, Kaduna State 6. Nimi Walson-Jack, Commissioner for Planning, Rivers State USAID.GOV SACE MID-TERM PERFORMANCE EVALUATION REPORT | 92 ANNEX 6: KEY INFORMANT AND FOCUS GROUP DISCUSSION SUMMARIES (AS AN ATTACHMENT) This annex, as a zipped folder, contains the summaries of 38 KIIs and four FGDs conducted as part of the SACE Mid-term Performance Evaluation. The file names allow readers to identify whether the summary refers to a KII or a FGD, location where the KII/FGD took place, type of informant, and institutional affiliation, mostly CSOs. Specifically: • The first three letters identify whether the summary refers to a KII or a FGD. • Next, the location where the KII or FGD took place is provided. • This is followed by whether the participant was an anchor or cluster CSO representative, a policy-maker, or a member of another type of organization.