OCTOBER 2019 This publication was produced at the request of the United States Agency for International Development/Nigeria. It was prepared independently by an Evaluation Team under the Monitoring, Evaluation & Learning Activity. EVALUATION SUSTAINABILITY EVALUATION REPORT STRENGTHENING ADVOCACY AND CIVIC ENGAGEMENT ACTIVITY STRENGTHENING ADVOCACY AND CIVIC ENGAGEMENT ACTIVITY (SACE) SUSTAINABILITY EVALUATION REPORT USAID Contract No. AID-620-C-14-00001 Submitted: September 19, 2019 Prepared by: Dr. Sergio Diaz-Briquets, Senior Evaluation Specialist and Team Leader Dr. Ogoh Alubo, Local Evaluator Mr. Aliyu Aminu Ahmed, Mid-Level Evaluation Specialist Ms. Ehizefua Elens-Edeh, Local Evaluator Submitted by: Paul DeLucco, Chief of Party, USAID/Nigeria MEL Activity, DevTech Systems, Inc. Submitted to: Dr. Chidimma Anyanwu, Contracting Officer Representative, PRO, USAID/Nigeria Ms. Beth Brownson, A/Contracting Officer Representative, PRO, USAID/Nigeria Ms. Olufunke Baruwa, Civil Society and Media Specialist, PDG, USAID/Nigeria Cover photos, clockwise from top left: Charles Abani, SACE Chief of Party with SACE cluster members; SACE Annual Learning Summit 2017; Outcome Harvesting by SACE CSOs; 10% Youth Campaign Participants. Photo credit: SACE and Roots of Change. DISCLAIMER The views expressed in this report are those of the Evaluation Team and do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS ACRONYMS......................................................................................................................................................IV EXECUTIVE SUMMARY..................................................................................................................................1 EVALUATION PURPOSE ................................................................................................................................................1 PROJECT BACKGROUND..............................................................................................................................................1 EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS.................................................................................1 FINDINGS AND CONCLUSIONS ....................................................................................................................................2 EVALUATION PURPOSE & EVALUATION QUESTIONS.......................................................................7 EVALUATION PURPOSE ................................................................................................................................................7 EVALUATION QUESTIONS ............................................................................................................................................8 BACKGROUND.................................................................................................................................................8 EVALUATION METHODS & LIMITATIONS ..............................................................................................9 METHODS 9 LIMITATIONS 10 FINDINGS, CONCLUSIONS & RECOMMENDATIONS ......................................................................... 11 EVALUATION QUESTION 1.........................................................................................................................................11 Findings ................................................................................................................................................................11 Conclusions...........................................................................................................................................................20 Recommendations.................................................................................................................................................21 EVALUATION QUESTION 2.........................................................................................................................................22 Findings ................................................................................................................................................................22 Conclusions...........................................................................................................................................................28 Recommendations.................................................................................................................................................28 EVALUATION QUESTION 3.........................................................................................................................................29 Findings ................................................................................................................................................................29 Conclusions...........................................................................................................................................................32 Recommendations.................................................................................................................................................33 EVALUATION QUESTION 4..............................................................................................................................................34 Findings...............................................................................................................................................................34 Conclusions .........................................................................................................................................................38 Recommendations ..............................................................................................................................................38 ANNEXES......................................................................................................................................................... 39 ANNEX I EVALUATION STATEMENT OF WORK .................................................................................................39 ANNEX II EVALUATION METHODS AND LIMITATIONS .........................................................................................54 ANNEX III DATA COLLECTION INSTRUMENTS ....................................................................................................55 ANNEX IV ABRIDGED ORGANIZATIONAL CAPACITY ASSESSMENT TOOL (OCAT)...............................................59 ANNEX V LIST OF DOCUMENTS REVIEWED........................................................................................................60 ANNEX VI DISCLOSURE OF ANY CONFLICTS OF INTEREST...................................................................................63 ANNEX VII DATA SUMMARIES .............................................................................................................................67 ANNEX VIII INNOVATION SPREAD THE WORD UPDATES ......................................................................................74 ANNEX IX POLICY MILESTONES POST-SACE ACTIVITY .......................................................................................78 ANNEX X ENGAGEMENT OF MARGINALIZED GROUPS POST-SACE....................................................................82 iv | SACE FINAL EVALUATION REPORT ACRONYMS AKYDF Akwa Ibom Youth Development Fund ANEEJ Africa Network for Environment and Economic Justice BMO Business Membership Organization CCEPE Centre for Community Empowerment and Poverty Eradication CCIDESOR Citizens Centre for Integrated Development and Social Rights CENSOJ Centre for Social Justice CI Chemonics International CISLAC Civil Society Legislative Advocacy Centre CITAD Centre for Information Technology and Development CSO Civil society organization DRAC Disability Rights Advocacy Center ESCFAL Edo State Cooperative Farmers’ Agency Limited ET Evaluation Team FGD Focus Group Discussions HDI Human Development Initiatives HERFON Health Reform Foundation of Nigeria ISWF Innovation and Spread the Word Fund JONAPWD Joint National Association of Persons with Disability KII Key Informant Interview LAPDO Life and Peace Development Organization LGA Local Government Area LITE-Africa Leadership Initiative for Transformation and Empowerment NCDF Nembe City Development Foundation NECA Nigeria Employer’s Consultative Association NGO Non-Governmental Organization NNEW NECA’s Network of Entrepreneurial Women OCA Organizational Capacity Assessment OGP Open Government Partnership OPADEC Oil Producing Areas Development Commission (by state) PANDO Formerly STARNET PIND Foundation for Partnership Initiatives in the Niger Delta PWD Persons With Disabilities S4C Spaces for Change SACE Strengthening Advocacy and Civic Engagement Social Action Social Development Integrity Centre STAR System for Transformation and Results STARNET System for Transformation and Results Network SUBEB Universal Basic Education Board UBE Universal Basic Education UBEC Universal Basic Education Committee UNDP United Nations Development Program UPFFA Ugboroke Progressive Fish Farmers Association USAID United States Agency for International Development WARDC Women Advocates Research and Documentation Centre WEWE Widows and Orphans Empowerment Organization YAF Youth Alive Foundation 1 | SACE FINAL EVALUATION REPORT EXECUTIVE SUMMARY EVALUATION PURPOSE This evaluation is intended to ascertain the progress made by the Strengthening Advocacy and Civic Engagement Activity (SACE) toward sustaining goals and objectives beyond funding by United States Agency for International Development (USAID). USAID intends to use this evaluation’s findings to inform the design and implementation of new civil society activities. PROJECT BACKGROUND In December 2013, USAID awarded Chemonics International (CI), a US-based international consulting firm, a five-year, $19.3 million contract (USAID Contract No. AID-620-C-14-00001) for the implementation of SACE to aid Nigerian civil society’s ability to influence the development and implementation of key democratic reforms at the national, state, and local levels. The primary goals and objectives of SACE included strengthening the institutional, organizational, and technical capacity of civil society organizations (CSOs) to advance democracy and good governance initiatives. Implementation included strengthening partnerships between CSO networks and government institutions; identifying key stakeholders to advocate for and monitor specific democratic reforms aimed at building transparency, accountability, and responsiveness of government institutions; strengthening public awareness, discourse and support for key democratic governance issues; and strengthening the capacity of partner business membership organizations (BMO) and CSOs in the Niger Delta to advocate for inclusive economic reforms and equitable economic growth. The latter effort was made in association with the Chevron￾affiliated Foundation for Partnership Initiatives in the Niger Delta (PIND). EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS This sustainability evaluation relies on a mixed rapid-appraisal approach that includes qualitative and quantitative components designed to generate insights regarding the various questions elaborated in the Statement of Work (SOW, Annex I). The evaluation includes the analysis of personal expert opinions obtained through key informant interviews (KII) and focus group discussions (FGD) conducted with former SACE staff members, representatives from anchor and cluster CSOs, and key government officials. Respondents were engaged in 11 of the 24 SACE Activity states. The evaluation’s three quantitative components, consisting of two electronic surveys and a questionnaire, were used to conduct an Organizational Capacity Assessment (OCA). One survey was sent to all former SACE anchor and cluster CSOs that participated in the Activity, while the other survey was distributed to participants in the two close-out events sponsored by SACE. The SOW elaborated four questions for the ET to respond to: 1. Has the SACE Activity been successful in achieving the Activity’s goals and objectives post activity close-out? 2. Did organizational and technical advocacy skills acquired under the Activity’s initiative help clusters to promote policy reform? How were various advocacy tools used? Did SACE partners manage to influence government policy development and implementation regarding vulnerable populations and could these achievements be attributed to SACE? 2 | SACE FINAL EVALUATION REPORT 3. What impact have SACE partners had in influencing government policy development and implementation regarding vulnerable populations such as women, youth, and persons with disabilities post close-out through advocacy and traditional and social media engagements? 4. How has the SACE Activity support contributed to economic reforms/growth in the Niger Delta region? Since the evaluation relied on a rapid-mixed-method approach, the standard limitations associated with such an approach apply; these include potential recall bias and an inability to contact some key informants. Further, survey response rates were lower than desired. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 – To what extent has the implementation of SACE been successful in promoting achievement of the activity’s goals and objectives post close-out? On the basis of the evidence examined by the ET, SACE made many achievements during implementation that continue to produce results well after close-out of the Activity. These achievements include upgrading the organizational capacity for anchors and the advocacy capacities for clusters. These institutional capacity gains were matched with important policy-reform gains to the benefit of vulnerable populations, budgetary transparency, education, and health, among other policy achievements. Regarding the effectiveness of the SACE anchor/cluster model and its sustainability beyond USAID support after close-out, the achievement of goals and objectives varied from site to site based on myriad factors. Both anchor and cluster CSOs provided positive assessments of the Activity, even those that did not seem likely to survive past the Activity’s end-date. In general, although most anchor CSOs mentioned advocacy trainings as a success, according to respondents more time dedicated to capacity￾building training is still needed. CSOs have continued advocating for reform, often seeking to deepen their impact, and in some cases have monitored to confirm that prior policy-reform gains are being properly implemented. Existing clusters continue to function, and new ones are being established. Since close-out, several anchors have secured grants from international donors. Across the field sites visited, the question of whether the anchor/cluster model worked beyond USAID support could generally be answered in the affirmative, but this response must be qualified in some important respects. Not surprisingly, the achievement of goals and objectives post-close-out has varied from site to site under the influence of individual factors. As a general rule, those clusters judged by the ET as more successful in sustaining the SACE goals and objectives post close-out provided very positive assessments of what they had learned from its approach. Interestingly, however, some anchor CSOs that appear less likely to sustain the SACE approach with equal vigor were just as likely to echo those positive assessments. The question, then, arises: What accounts for this divergence? It appears that for some anchors the advocacy skills acquisition process—while generally successful—was truncated just as it was gaining momentum and reaching the point where it could have borne fruit over a longer time. Lack of funding from other donors for some anchor CSOs has also impacted close-out performance. Yet another impediment may have been the unwillingness of policy-makers to engage in policy dialogue with civil society. The ET interviewed five out of the seven Innovation and Spread the Word Fund (ISWF) grant recipients to identify the status of their innovations following the close-out of SACE. All five ISWF grantees have 3 | SACE FINAL EVALUATION REPORT continued implementing their innovations, with one in particular, ConsTrack, having the potential to increase transparency of government operations across Nigeria. Based on the ET’s findings, the SACE model and methodology have achieved the intended results and continue being utilized post SACE close-out. Therefore, if a future CSO activity were to address SACE implementation shortcomings as detailed in this report, this advocacy approach would be sustainable and contribute to needed policy reforms across the country. Recommendations: The ET’s findings include suggestions by key informants that any follow-on civil society advocacy activity should: ▪ Conduct a needs assessment to determine CSO requirements and thereby develop training programs specific to the competencies of different CSOs; ▪ Provide more direct contractor training to cluster members than SACE did, as most of the training provided by the Activity was geared primarily to the benefit of anchor CSOs, and only secondarily to cluster CSOs; ▪ Develop training programs, facilitated by good-governance-oriented CSOs, specifically designed to educate elected officials at all levels of government about the citizens’ role in the formulation of public policies; ▪ Implement training programs for career-level public officials at all echelons of government, because a lack of understanding regarding the role of civil society in public policy formulation has obstructed the advocacy process. EVALUATION QUESTION 2 - How have SACE partners’ financial and technical advocacy skills (Capacity 1.0 and 2.0) influenced their capacity to engage with government regarding reforms? What has worked and what has not worked? Respondents were generally eager to share their CSO’s experiences with the SACE advocacy tools with members of the ET. During the KIIs and FGDs, the vast majority of cluster anchors and members retrieved the names of the various tools without prompting. Equally telling was the ease with which the informants recalled how the tools had been used while the project was being implemented and the extent to which they continue to be used, or not, after close-out. Informants indicated that several SACE tools such as System for Transformation and Results Network (STARNET, now PANDO), Outcome Harvester, and Policy Tracker had been useful, with the caveat that some anchor/clusters took greater advantage of some tools than others, as some were more user-friendly. Some tools, such as Policy Tracker, Outcome Harvester, and Cluster Coaching, were found to be more enduring and continue to be used more often than the STARNET. Informants commented favorably upon other features of the SACE approach, including the power of collective advocacy, speaking with one voice, and “leading from behind,” i.e. the notion that anchors should encourage cluster members to take the lead in specific initiatives. Most anchor/cluster organizations have continued SACE programming since the activity ended in 2018, although many reported the intensity of their advocacy has suffered. An important factor noted by informants in practically every site visited was the distraction caused by the early 2019 general elections, which drew attention away from all other advocacy activities in which governance-related CSOs were engaged. Regarding reliance and continued use of the SACE tools, the findings garnered by the ET were more mixed. Some informants praised STARNET, while others were far more reserved in their assessments of that tool. It appears that past and continued use of STARNET is mediated by several 4 | SACE FINAL EVALUATION REPORT factors. One was reportedly the insistence on the part of SACE staff that participating CSOs utilize the tool while the project was underway. Following the close-out, this adherence seems to have waned. The Cluster Coaching, Policy Tracker, and Outcome Harvester were cited as the most useful and frequently used tools. Regarding future use, the evaluation findings show that CSOs plan to continue using these tools, although some questions were raised regarding continued use, for technical and cost issues, of STARNET. To address the evaluation question as to whether former SACE partners are ready to receive direct funding from USAID, the ET conducted a “Capacity 1.0” OCA, with the aim of assessing critical elements for effective organizational management. The relative capacity of the various anchor CSOs has changed but little since the previous 2018 assessment. Seven out of the fourteen CSOs assessed, however, appear qualified to receive direct USAID funding. On the basis of the findings, it can be concluded that SACE tools have played a useful role in enhancing the advocacy capacities of the partner CSOs; these skills and related tools are likely to continue to be used in policy-reform advocacy. However, the organizational capacity training was only partially successful in building administrative and financial capacity to manage grants, and therefore impact sustainability prospects. Only half of the anchors currently qualify to directly manage USAID funds. Essentially, the amount of training provided was insufficient to enable some CSOs to achieve the desired level of competency; further training provided over a longer period of time will be necessary to overcome organizational deficiencies. Recommendations: The ET’s findings indicate, and key informants recommended, that any follow-on civil society advocacy activity should: ▪ Impart intensive training regarding financial and fund-raising skills, including proposal writing; ▪ Rely, to the extent possible, on more flexible administrative procedures; ▪ Evaluate program approaches to gradually improve organizational capacities among Nigerian CSOs, independently of whether or not they receive USAID support; ▪ Provide support for the establishment, maintenance, and update of a clearly identifiable, user￾friendly website to inform CSOs about the availability of the SACE methodology; ▪ Support permanent establishment of a national CSO-advocacy institute. EVALUATION QUESTION 3 – What impact has SACE had in influencing government policy development and implementation regarding vulnerable populations such as women, youth, and persons with disabilities within the last six months post-SACE? Due to SACE efforts, clusters were intended to serve, and include, marginalized populations such as women, youth, and persons with disabilities (PWD) in their policy-reform agendas. Some clusters were convened to address their policy-reform objectives, while others embedded some marginalized-population participation policy into their agendas. These policy accomplishments have not ended, as some clusters have continued to advocate for further reforms but are demanding that those already approved be implemented as required by law. The ET was able to document numerous SACE accomplishments. These ranged from assisting partner￾anchor CSOs and their clusters to increase their technical organizational capabilities to instilling a more sophisticated culture of policy-reform advocacy. After the SACE close-out, most anchor/clusters have 5 | SACE FINAL EVALUATION REPORT repurposed their advocacy strategies to some extent, while continuing to foster sustainable relationships with policy-makers. These built on policy-achievement milestones that are partly attributable to SACE interventions. Project achievements were documented in numerous ways by SACE staff and continued after the SACE Activity close-out. One of SACE’s predictors of eventual advocacy success was the ability of anchor CSOs to secure funding from institutions other than USAID. The evaluation findings show that anchor CSOs are receiving more awards than in the past; several former anchors have diversified their funding sources significantly by winning more than one grant from sources other than USAID. The sustainability of media activities appears to be mixed. While across all Activity sites a fruitful CSO/conventional media relationship seems to have flourished and continues to be active, the outcome of social media is not as clear-cut. The latter has a distinctive characteristic as a primary means for communication and coordination, even making up for functions previously played by SACE through its learning and sharing events. Reliance on social media has increased as an alternative to some of the alliance￾building services previously secured through STARNET. On the other hand, some social-media applications have faltered while others are poised to succeed far beyond initial expectations. The evaluation was unable to ascertain if the SACE impact on Nollywood and other commercial mass media will be lasting in the absence of well-organized and orchestrated advocacy promotion efforts. In conclusion, SACE accomplished its objective by increasing participation of marginalized populations and helping to formulate policy reforms to their benefit. It also contributed to many reforms in the area of maternal and child health, education, budget transparency, marginalized women farmers and stall sellers, and other sectors. Recommendations: The ET’s findings include suggestions from key informants that any follow-on civil society advocacy activity should: ▪ Consider having USAID play a decisive role in fomenting a Nigeria-wide advocacy policy￾reform initiative based on the principles of the Open Society Partnership; ▪ Be more sensitive to the unique concerns of the PWD community, as well as the youth and women who are currently virtually excluded from decision making and policy formulation and implementation; ▪ Support a public-awareness campaign to educate the public as to why set-aside government programs on behalf of PWD should be endorsed; ▪ Host human-development seminars or similar training on software application development, so that future CSO advocacy activities may benefit from technological innovations similar to the SACE-sponsored ISWF that are currently helping promote greater government transparency and accountability. EVALUATION QUESTION 4 – How has SACE’s support for Niger Delta implementing partners contributed to their advocacy for economic reforms/growth in that region? Desk reviews, KIIs, and FGDs show several milestones have been achieved by SACE partners in the Niger Delta. One of these is the establishment in all six Niger Delta states of Oil Producing Areas Development Commissions (OPADEC), organizations designed to harness oil revenue to enable the government to meet the financial needs of their populations. This has facilitated more transparency in channeling oil revenue into areas of need as identified by beneficiaries in various localities, while improving citizen engagement in more transparent budgeting. 6 | SACE FINAL EVALUATION REPORT Although CSOs operate in difficult terrain, circumstances, and situations, it was obvious that policy-reform advocacy has made some significant differences, especially in the Niger Delta, as reflected in major achievements in budgetary transparency and improvement in living conditions of the marginalized population. Some of the anchor CSOs, such as the African Network for Environment and Economic Justice (ANEEJ) and Youth Alive Foundation (YAF), have received substantial funding from international agencies such as the United Kingdom’s Department for International Development (DFID). In conclusion, SACE has contributed to economic development and overall policy reforms to bring more regional development. However, this groundwork should be considered as laying the foundation on which to build future engagements between CSOs and government. Recommendations: The ET’s findings include suggestions by key informants that any follow-on civil society advocacy activity should: ▪ Continue to expand the policy-reform advocacy process in the Niger Delta; ▪ Consider maintaining the mutually supportive relationships with PIND; and, ▪ Expand, to the extent possible, future CSO activity collaboration with other international donors. 7 | SACE FINAL EVALUATION REPORT EVALUATION PURPOSE & EVALUATION QUESTIONS The United States Agency for International Development (USAID) contracted DevTech Systems, Inc. to analyze the sustainability of Strengthening Advocacy and Civic Engagement Activity (SACE) after close￾out. USAID hopes that this evaluation will determine if its interventions have contributed to civil society’s technical and financial capacity to advocate for: ▪ Increased participation of vulnerable groups, e.g. women, youth, and persons with disabilities (PWDs); ▪ Transparent decision-making; ▪ Government support to institutionalizing transparent and accountable governance; and, ▪ Economic growth in the Niger Delta region. Requested recommendations are intended to inform the design and implementation of a new civil society activity by USAID/Nigeria. EVALUATION PURPOSE This evaluation was designed to ascertain the progress made by SACE partners in sustaining the project’s goals and objectives beyond USAID’s funding. USAID also intends to incorporate these evaluation findings into the design and/or implementation of a new civil society activity. Specifically, this external sustainability evaluation is tasked to: 1. Develop key evaluation questions and a methodology to answer these questions: ▪ To what extent has the implementation of SACE been successful in promoting achievement of the Activity’s goals and objectives after close-out? ▪ Has the SACE’s anchor/cluster model worked effectively without/beyond USAID support? How have the Innovation Spread the Word (ISWF) grantees fared? ▪ How have SACE partners’ financial and technical advocacy skills (Capacity 1.0 and 2.0) influenced their capacity to engage with government on reforms? What has worked and what has not worked? ▪ Have Systems for Transformation and Results Network (STARNET, now PANDO), Outcome Harvester, Policy Tracker, and other tools been effective in that regard? ▪ Are there additional SACE anchor partners ready to receive direct funding from USAID? ▪ What impact have SACE partners had in influencing government policy development and implementation regarding vulnerable populations such as women, youth, and PWD within the last six months post-SACE? For what documented policy mileages can SACE claim credit? ▪ In particular, what has been the influence on viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Mindset Series, Twitter, live streaming, and films? ▪ How has SACE’s support for Niger Delta implementing partners contributed to their advocacy for economic reforms/growth in that region? 2. Evaluate progress made by partners within six months after close-out and whether partners are still using the tools and learnings from the Activity to inform program design and implementation and are open to continue doing so beyond the evaluation period. 8 | SACE FINAL EVALUATION REPORT 3. Analyze SACE’s approach to determine if it has strengthened civil society’s technical and financial capacity to: advocate for increased participation of vulnerable groups (women, youth, and PWD) in decision-making; support government to institutionalize transparent and accountable governance; and advocate for economic growth in the Niger Delta region. 4. Make recommendations to USAID/Nigeria regarding the design and/or implementation of a new civil society activity. EVALUATION QUESTIONS Evaluation Question 1: To what extent has the implementation of SACE been successful in promoting achievement of the activity’s goals and objectives after close-out? ▪ Has the SACE’s anchor/cluster model worked effectively without/beyond USAID support? How are they funded or sustained? ▪ How have the Innovation Spread the Word (ISWF) grantees fared? Evaluation Question 2: How have SACE partners’ financial and technical advocacy skills (Capacity 1.0 and 2.0) influenced their ability to engage with government on reforms? What has worked and what has not worked? ▪ Have Systems for Transformation and Results Network (STARNET, now PANDO), Outcome Harvester, Policy Tracker, and other tools been effective tools in that regard? ▪ To what extent are the SACE partners using the tools and learning from the Activity to inform program design and implementation, and are they open to continue doing so beyond the evaluation period? ▪ Are there additional SACE anchor partners ready to receive direct funding from USAID? Evaluation Question 3: What impact have SACE partners had in influencing government policy development and implementation regarding vulnerable populations such as women, youth, and PWD within the last six months after SACE? ▪ For what documented policy milestones can SACE claim credit? ▪ In particular, what has been the influence on viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Mindset Series, Twitter, live streaming, and films? Evaluation Question 4: How has SACE’s support for Niger Delta implementing partners contributed to their advocacy for economic reforms/growth in that region? ▪ To what extent do achieved results continue to be relevant to the needs of Niger Delta implementing partners? Evaluation Question 5: Make recommendations to USAID/Nigeria regarding the design and/or implementation of a new civil society activity. BACKGROUND In December 2013, USAID awarded Chemonics International (CI), a US-based international consulting company, a five-year, $19.3 million contract to intended to strengthen civil society’s ability to influence the development and implementation of key democratic reforms at the national, state, and local levels in 9 | SACE FINAL EVALUATION REPORT Nigeria. SACE activities, from January 2, 2014 through January 1, 2019, supported increased engagement and efficacy of civil society to influence public institutions whose function is to serve citizens’ interests. This was achieved through strategic partnerships between civil society coalitions and networks, formal or informal, on the one hand, and other critical stakeholders in Nigerian society on the other. The latter included the media, reform-minded government institutions or representatives, private sector actors, influential public figures, grassroots organizations, professional associations, and youth groups. The Activity also explicitly engaged marginalized populations, such as women, youth, and persons with disabilities in the process, and it emphasized the importance of leadership and innovation. Some of the Activity’s primary goals and objectives included: strengthening institutional, organizational, and technical capacity of targeted civil society organization (CSO) coalitions and networks to advance democracy and good-governance initiatives; strengthening partnerships between CSO-led coalitions and networks and selected government institutions and key stakeholders to advocate for and monitor specific democratic reforms aimed at strengthening transparency, accountability, and responsiveness of government institutions; strengthening public awareness, discourse and support for key democratic governance issues such as transparency, accountability and good governance; and, strengthening capacity of partner business membership organizations (BMOs) and CSOs in the Niger Delta to advocate for inclusive economic reforms and equitable economic growth. In the Niger Delta region, these goals and objectives were to be achieved by working closely with the Chevron-affiliated Foundation for Partnership Initiatives in the Niger Delta (PIND), USAID’s strategic partner, as SACE was to complement PIND’s other activities in the region. The Activity supported eight national issue clusters/coalitions/networks, each with an anchor CSO that provided the foundation for collective action on a core advocacy issue and facilitated the efforts of participating organizations. These national coalitions operated in tandem with seven Niger Delta CSOs and their clusters on issues of economic development and growth in the Niger Delta, as well as three Niger Delta BMOs. The purpose of the coalitions was to directly engage and support initiatives that would strengthen and improve the enabling environment for CSOs to operate effectively, with a focus on political economy analysis and identification/response to windows of opportunity, the legislative space, innovation and new technologies as a driver of change. During the Activity’s implementation, the actual number of active anchor/clusters changed as some were added and others discontinued participation. EVALUATION METHODS & LIMITATIONS METHODS The evaluation primarily relied on interviews with experts obtained through key informant interviews (KIIs) of former SACE staff members; representatives from anchor and cluster CSOs and selected project beneficiaries; and key government officials. Qualitative data gathering also relied on information gathered through focus group discussions (FGDs) held with CSO cluster members. The methodology also included a desk review of various SACE-related administrative reports - before and after fieldwork was conducted - supplemented by direct visual inspection during field visits of documents evidencing acquisition of organizational capacity expertise, including financial policies, personnel policies, guidelines, manuals, etc. Respondents were engaged in 11 of the 24 SACE Activity states, i.e. Akwa Ibom, Federal Capital Territory, Kaduna, Bauchi, Kano, Lagos, Kwara, Ondo, Edo, Imo, and Rivers. For further details regarding sites visited, number and type of KIIs conducted and other data, please see the evaluation work plan. 10 | SACE FINAL EVALUATION REPORT The evaluation also relied on quantitative data analysis through surveys and an organizational capacity assessment. Closed-ended electronic survey questionnaires were sent to anchor and cluster CSO representatives and participants in other SACE-sponsored events. The first survey, the Successes, Challenges, and Sustainability Issues Survey, was intended for CSOs that had participated in the SACE Activity. The second, the Share-out Survey, was meant to elicit feedback from CSO representatives who were invited to participate in workshops sponsored by SACE in Abuja and Port Harcourt to expose CSOs not involved with SACE to its tools and results. These two questionnaires also included a few open-ended questions. A final structured questionnaire was sent to all anchor CSOs as part of an Organizational Capacity Assessment. LIMITATIONS In an evaluation design of this type, respondent selection bias can always creep in and the ET was unable to entirely control these biases. For instance, although the ET was provided a list of potential respondents, it had no control over who would be available. Now that the activity is over, difficulties reaching respondents became a concern as some CSO representatives (previously readily available) were either occupied with other activities or simply unreachable. Policy-makers were often unavailable for an interview. A related but inevitable bias, given the evaluation design, is that the vast majority of respondents, except policy-makers and respondents to the Share-out Survey, had had a previous direct association with the SACE Activity; this may or may not have colored their responses in unknowable ways. For example, respondents may have been predisposed to give favorable assessments with the expectation of receiving further USAID funding. Recall bias may also be a potential source of error. Response rates to the surveys could also influence the solidity with which conclusions reached on the basis of triangulation of results were arrived at. Particularly disappointing was the low response rate of only 21 percent (24 out of 115 potential respondents) received from those to whom the Successes, Challenges and Sustainability Issues Survey was sent, despite repeated rejoinders. As the ET eventually learned, many of these potential respondents were reticent to comply unless they were told to do so by their respective anchors, something that apparently did not happen. The response rate of 46 percent for the Share-Out Survey was much better, 23 out of 51 potential respondents returning completed questionnaires. 11 | SACE FINAL EVALUATION REPORT FINDINGS, CONCLUSIONS & RECOMMENDATIONS EVALUATION QUESTION 1 FINDINGS EVALUATION QUESTION 1 – To what extent has the implementation of SACE been successful in promoting achievement of the activity’s goals and objectives after close-out? SACE had many achievements while it was implemented and continues performing well after close-out. Institutional capacity gains were matched with important policy-reform gains to the benefit of vulnerable populations, budgetary transparency, education, and health, among others. These achievements include upgrading the organizational capacity for anchors and advocacy capacities for clusters. Of particular note were the establishment of Oil Producing Development Commissions (OPADECs) in the Niger Delta, more inclusion of marginalized groups such as market women and PWDs, additional funding to women farmers in Ondo State and, in some instances, sustainable relationships with policy-makers and staff of State ministries, departments, and agencies. In some cases, advocacy efforts did not yield desired results because policy-makers were not receptive to engaging with CSOs, claiming to be too busy, not authorized to speak on some issues, or were unwilling to discuss policy-reform issues with CSO representatives. Specific instances that could be cited include Women Advocates Research and Documentation Centre (WARDC)’s inability to engage with policy-makers to advocate for the Gender Bill in Enugu State and issues the Human Development Institute (HDI) had with the Lagos State Universal Basic Education Board (SUBEB) regarding the release of budget data. Has the SACE anchor/cluster model worked effectively without/beyond USAID support? How are they funded or sustained? The achievement of goals and objectives after close-out has varied from site to site under the influence of myriad factors. The most significant appears to be how effectively individual clusters internalized and applied the SACE advocacy methodology. Another important determinant may have been the initial institutional strength of individual anchors when joining SACE. A related factor is the advocacy success achieved while operating under SACE and the implications of these achievements for future performance. Other success predictors are related to the nature of the political setting—in SACE’s terminology, whether or not policy-reform opportunities were revealed through the ongoing political economy analysis—and the often-present but imponderable nature of leadership, whether on the part of policy￾makers or key CSO leaders driving the reform agenda. SACE’s sustainability legacy is largely determined by how these elements mediated the effectiveness with which its objectives were pursued. In the view of the ET, a training program is needed to sensitize policy-makers about the need to engage civil society in public policy formulation. For the reform to succeed, political leadership is necessary. In general, the most-successful clusters in sustaining SACE’s goals and objectives post-close-out provided very positive assessments of what they had learned from its approach. Respondents noted, for example, that the SACE-targeted advocacy strategy proved to be far more effective than traditional advocacy approaches; it made CSOs more focused. They also pointed out as a result of SACE, they had managed to transform formerly antagonistic relationships with government officials into collaborative efforts, as one anchor representative observed: 12 | SACE FINAL EVALUATION REPORT “… we used to be a ‘stone-throwing organization’ because we didn’t believe in sitting down to talk or have a conversation with government, but the SACE project brought us closer to listen to what they had to say, to sit with them, and discuss policy issues.” Without exception, key informants and FGD participants like the one cited immediately below reported highly positive views about the SACE anchor/cluster approach and its tools. “…as a game changer, (the model) was able to garner the full force of collective impact through the combined advocacy work of a group of CSOs striving to reach a common outcome. The model has become strategically embedded in the way we pursue policy-reform advocacy objectives. Initially, some of the cluster members did not understand some of the issues, but eventually, with the cluster model and leadership from behind, they managed to further engage with their elected officials.” As the quotation suggests, informants repeatedly referred to the value of collective advocacy. Many mentioned a continued reliance on many of the advocacy tools SACE disseminated among its anchor and cluster members, respondents indicating the tools are consistently applied and often used to train new advocacy partners. However, even those anchor CSOs which appeared to be less likely to sustain the SACE approach were just as likely to echo positive assessments of the SACE tool and model. While the activity performed well at building organizational (Capacity 1.0) and advocacy (Capacity 2.0) capacities for most anchors and some cluster members, it fell short with others. Such an outcome was partially predictable as initial conditions differed from CSO to CSO, even though institutional capacity￾building efforts were extended to all anchors. Some of the former SACE staff members noted that toward the end of the Activity, some initially notorious weak anchors were performing rather well. Although cluster members benefited too, anchors were the primary object of attention (e.g., in terms of training and funding). What appears to account for this outcome is that, for some anchors, the advocacy skills￾acquisition capacity process was too short to achieve greater results. SUCCESSFUL CSOS More successful CSOs shared certain features. In general, they: ▪ continued advocacy efforts in connection to the deepening or implementation of past policy￾reform efforts, or the pursuit of related policy-reform objectives; ▪ sustained and effectively utilized SACE-disseminated advocacy tools in long-standing or new policy-reform endeavors; ▪ developed a robust administrative capacity, demonstrated through past and contemporaneous organizational capacity assessments; and ▪ applied the SACE toolkit in the pursuit of policy-reform goals and objectives initiated under the auspices of donors other than USAID since SACE ended in early 2019. According to these criteria, several anchor CSOs were able to achieve significant advocacy capacity but not a strong organizational capacity, and vice-versa. In at least one instance, the anchor, Citizen’s Centre for Integrated Development and Social Rights failed on both accounts, as it achieved neither advocacy nor organizational capacity. Those anchors that succeeded in both have managed to gain grant support from agencies other than USAID, such as the Department for International Development (DFID), the MacArthur Foundation and Bread for the World. 13 | SACE FINAL EVALUATION REPORT Six anchors succeeded in achieving the desired advocacy and organizational capabilities. They include the HDI in Lagos; African Network for Environment and Economic Justice (ANEEJ), Benin in the Niger Delta; Centre for Social Justice (CENSOJ) in Abuja, FCT; and the Youth Alive Foundation (YAF) in the Niger Delta and Abuja, whose achievements are described below. Other anchor success stories (that have developed organizational capacity, including strong structures for the management of donor funds) not described in detail include the Civil Society Legislative Advocacy Centre (CISLAC) and the Life and Peace Development Organization (LAPDO). Still other anchors, while meeting most of these criteria, were unable to attain a high level of performance on all organizational criteria and thus their sustainability, as defined for the purposes of this evaluation, is not assured. Human Development Initiative (HDI): The advocacy successes achieved by HDI while advocating for budget transparency in, and for increasing the quality and accessibility of, primary and secondary education have persisted not only after close-out, but in fact may be accelerating. The CSO’s capabilities and achievements are increasingly recognized by the international donor community; in 2019, the MacArthur Foundation awarded a grant to HDI to further the advocacy work pursued under SACE auspices. Other organizations, such as EDU Planner, have approached HDI to request its assistance in developing approaches and implementing educational strategies. As documented in recent proposals submitted to donors, in all instances HDI and its partners are committed to the SACE-inspired advocacy tools in their forthcoming initiatives. No better evidence of this intent is that, in these new endeavors, HDI will continue to rely on its original SACE cluster—as will be the case in Enugu— while either adding new CSO members or, as in the case of Lagos, substituting former cluster members with others better suited to the new advocacy task. STARNET (now PANDO) continues to be used to identify new partners and was described as a “very, very useful tool,” with members of the original cluster still remaining in regular contact through a WhatsApp group. HDI’s attractiveness for donor funding increased as it developed more substantive technical, organizational, and financial skills through the 1.0 Capacity training provided by SACE, substantiated by a 2018 Organizational Capacity Assessment and further validated by the ET’s own capacity assessment. Africa Network for Environment and Economic Justice (ANEEJ): Regarding the first criteria enunciated above, ANEEJ continues to be an effective advocacy change agent in the Niger Delta. While it proved its mettle, it continues to lead its cluster from behind after close-out by advocating for enhanced transparency and accountability in resource management. In 2019, with cooperation from the CSO community and government officials in the Niger Delta, ANEEJ embarked on a new initiative. The “Open Niger Delta” is designed to forge an action plan to institutionalize transparency and accountability practices in the region by embracing the Open Government Partnership (OGP) principles, a global initiative already endorsed by the Federal Government in 2016. If these principles are implemented in the Niger Delta, they would provide the foundation for considerable further transparency in the way Niger Delta Development Commissions operate, a necessary step in consolidating prior gains achieved by ANEEJ and its partners. State-specific efforts around the OGP are already under way in Edo and Ondo States. The anchor/cluster model is at the heart of these efforts, as well as SACE advocacy tools, notably Cluster Coaching and Outcome Harvester. ANEEJ has been successful not only due to the governance reforms it helped achieve through its collective advocacy work, but also because of the organizational and financial management gains made through its participation in the Capacity 1.0 institutional development training provided by SACE. Such achievements have been validated by several organizational capacity assessments, including one concluded (August 2019) by the ET. “The technical support received (from SACE) was exceptional,” a key informant noted, and 14 | SACE FINAL EVALUATION REPORT “contributed immensely to the growth and development of ANEEJ.” The donor community has taken note of ANEEJ’s ability to make a difference. In months to come, ANEEJ is slated to showcase its achievements at The World Bank in Washington and at Chatham House in London. Equally consequential is the grant support ANEEJ has attracted since the SACE project ended. ANEEJ’s current primary funders are Bread for the World, a German grant-making institution (this €650,000 grant will run through April 2020) and DFID through a £1.5 million anti-corruption grant awarded in April 2019. Centre for Social Justice (CENSOJ): CENSOJ continues to actively pursue reforms in public financing for maternal/child health, including ensuring that the government complies with the one percent maternal/child health federal-funding allocation. This monitoring activity is implemented through a broad coalition to which the CENSOJ-led cluster is heavily committed. The SACE approach continues to be central to CENSOJ’s advocacy initiatives. Recently, the CSO developed a new cluster to achieve additional policy-reform goals (the Government Audit Reform cluster). CENSOJ depends on SACE tools in their advocacy work. In constituting the Government Audit Reform cluster, CENSOJ relies on the identification of potential cluster partners on STARNET (now PANDO). CENSOJ’s national and state coordinating efforts also rest on Cluster Coaching and other SACE-training tools, which are disseminated to new advocacy partners for capacity-building. As noted by a respondent, “we have adopted the SACE toolkit and continue using what we learned as a crucial input in designing our advocacy strategy.” Acquisition of 1.0 Capacity has made CENSOJ a more trustworthy grant recipient for donors as it now has strengthened ability for grants management. In 2019, domestic and international donors favored CENSOJ by awarding them several grants, including four new 2019 separate grants: the USAID North East Regional Initiative, a Chemonics-managed project; a component of the International Budget Partnership; a European Union-financed Audit Reform grant; and a Bench Marking Budget for Extractives funded by the Facility for Oil Sector Transparency in Nigeria. In addition, CENSOJ has been short-listed for a potential United Nations Development Program (UNDP) Spotlight Initiative award. Youth Alive Foundation (YAF): With offices in Uyo, Akwa Ibom in the Niger Delta, and in Abuja, this CSO focuses primarily on Nigerian youth in governance and development processes. It was a SACE Activity anchor and coordinated activities of 53 cluster nongovernmental organizations (NGOs). One of its notable achievements was the mobilization around and eventual passage of the Akwa Ibom Youth Development Fund bill (AKYDF), whose intent is to commit ten percent of the annual State budget (about N3 billion) to youth development. This bill was passed by the Akwa Ibom House of Assembly in February of 2019 but must be signed by the Governor to become law. YAF was also awarded one of the SACE ISWF grants, which funds were used to develop the “Your Excellency” app to engage youth in governance issues, an app currently being updated. YAF networked with other CSOs in the Niger Delta region and beyond; ANEEJ and Edo State Cooperative Farmers’ Agency Limited (ESCFAL) were mentioned as regular contacts. Further afield, CSOs from Kogi and other states have come to understudy how the AKYDF was conceived and passed, to enable replication. YAF has continued to use SACE tools in advocacy and its cluster members have become well￾known in the community. In addition to issues of inclusion of youth, women, and PWD in decision-making and participation in policy formulation and implementation, cluster members now engage in budget tracking to compel the government to “kene budget,” or follow the budget. Since the closeout of SACE, YAF has obtained other sources of support, such as a £2 million grant from DFID, as well as grants from 15 | SACE FINAL EVALUATION REPORT the PIND, the International Foundation for Electoral Systems, and the Ford Foundation. Interactions with cluster members and policy-makers indicate that YAF is highly esteemed for its advocacy work. LESS-SUCCESSFUL CSOS There are CSOs less likely to consolidate and sustain their past performance, due to SACE’s inability to impart sufficient Capacity 1.0 and Capacity 2.0 training to allow these anchor CSOs to overcome institutional weaknesses to become sustainable and effective change agents. This limitation extends to a current incapacity to attract external funding sources to successfully promote policy-reform agendas. Anchors in this situation include those below. NECA’s Network of Entrepreneurial Women (NNEW): NNEW is a CSO working for the achievement of the rights and opportunities for women. It led the initiative in ensuring that market women are not mistreated, exploited, or subjected to continuous harassment and multiple taxation by corrupt government officials. When viewed as the effective and committed mobilization of one of the most vulnerable and abused Nigerian constituency, the informal sector women stall sellers, its achievements are impressive; however, this anchor has yet to gain donor support despite attempts to do so, partly due to management shortcomings. SACE’s impact on the effectiveness of NNEW can only be described as transformational. As one informant related to the ET, “our intervention with the SACE project gave us the boldness to interface with the government.” The engagement gave NNEW enough confidence to expand its advocacy reach. “Before SACE, NNEW was working with only three policies, now we are focusing on 19.” The SACE tools were and continue to be essential to the NNEW advocacy work. “(STARNET) was used to harmonize and gather all clusters and anchor in one database (while) we still make use of the Outcome Harvester regularly … in tracking our success stories from budget narrative and work plans … we review and update all our activities every three months using these tools (and, in the post-SACE environment) still use these tools for PIND projects.” Cluster communication is maintained through a WhatsApp link. Among other NNEW achievements have been the institution of policies to prevent the harassment of women stall sellers and the establishment of new markets designed for women traders, including specially designed sections to accommodate the needs of disabled women traders. These achievements were made possible by the collective impact of multiple women groups joining efforts under the NNEW cluster that are still active today. Even though the organizational and financial capacities of NNEW and some of its cluster members were strengthened by SACE’s Capacity 1.0 training, however, this anchor has yet to gain donor support despite attempts to do so, partly due to persistent management shortcomings. NNEW’s inability to gain access to external support threatens its long-term viability. In spite of this, the NNEW is a shining SACE example of what a humble group of beneficiaries under a CSO umbrella can achieve. NNEW convincingly managed to bring about changes in a policy environment previously regarded as resistant to change, while improving the livelihood and economic prospects for vulnerable women and their children. Although it continues to be engaged with PIND-supported initiatives and receives limited government support, NNEW’s viability rests on the financial support it receives from its membership. Centre for Information Technology and Development (CITAD): While participating in SACE, CITAD made considerable progress advocating for increased education funding for states in the Northeast. By engaging the State House of Assembly, budget allocations increased by the time of the SACE close-out from nine percent to 18 percent in Bauchi, Adamawa, and Yobe, and 12 percent in Borno. 16 | SACE FINAL EVALUATION REPORT CITAD has extended its advocacy interest into new areas such as peace building, anti-hate speech, gender￾based violence, and PWD. These efforts are loosely coordinated since CITAD lacks the resources to convene cluster coordination meetings or capacity to train others on how to utilize the System for Transformation and Results (STAR) suite of advocacy tools. To overcome these shortcomings, CITAD must further develop its 1.0 and 2.0 Capacities. According to an informant, getting through to policy￾makers in this part of Nigeria continues to be difficult and additional advocacy trainings are necessary to open the door to dialogue. Also, given USAID status as a major health-sector donor, when CITAD was receiving USAID funding they policy-makers were more responsive. Additional 1.0 capacity-building is necessary if CITAD is to gain further financial support from USAID or other donors. Results of the organizational capacity assessment reveal that CITAD capabilities remain below the threshold necessary to receive grant funds. While CITAD continues to partially rely on some SACE tools—primarily Outcome Harvester—to guide its activities, it has dispensed with STARNET due to complexity. Through a forthcoming North East Book Education and Exhibition Conference to be organized in collaboration with the government of Bauchi State, CITAD hopes to partly revitalize the anchor/cluster model but, with scarce funding, this may only be partially achieved. The end of SACE programming prevented CITAD from further developing its organizational capabilities. This largely explains why the CSO has failed to attract outside sources of funding, despite some impressive advocacy achievements. Health Reform Foundation of Nigeria (HERFON): The end of the Activity has led to the erosion of the HERFON-led cluster after attaining significant health-sector achievements in Kaduna State. While some of these achievements have been consolidated, such as CSOs partaking in policy deliberations of government steering committees, to a great extent the advocacy capacity formerly exercised by HERFON and its cluster has dissipated. As noted by a respondent, since the SACE close-out in-person coordination meetings, formerly held on a routine basis, are no longer scheduled; instead, most cluster coordination today is being conducted through social media. In some ways, the collaborative approach has been abandoned, as each CSO is consumed with its primary engagement as coordination (including documentation of advocacy processes) has become difficult to sustain. Thus, the anchor role previously held by HERFON now is assumed by whichever CSO is focused on a particular issue. Previous coordination achieved through the use of the Cluster Coaching or Outcome Harvester tools is a thing of the past, as HERFON no longer evaluates what one CSO is achieving vis à vis what others are doing. Some of these problems have arisen because some CSO members did not receive as much SACE training as they should have, while others could be traced directly to a shortage of funds to coordinate activities. Despite SACE’s efforts to make HERFON stronger and more capable of accessing outside sources of funding, the anchor CSO never achieved a sufficient proficiency level to manage external funds. Thus, insufficient organizational and advocacy training after SACE’s activity close-out has handicapped HERFON’s ability to continue acting as a successful advocate for reforms in the health care sector. Social Development Integrated Centre (Social Action): Social Action advocated successfully for budget-transparency reforms in the Niger Delta states of Edo, Bayelsa, Delta, Akwa Ibom, and Rivers. Recently, Social Action has also seen how alignment of local political forces affects the achievement of transparency objectives. This process has played out differently in various states. In some states, Social Action has been able to sustain reforms, while in others considerable backsliding has occurred. Reforms in Delta State remain established and the state government has stated its willingness to continue 17 | SACE FINAL EVALUATION REPORT collaborating with Social Action. However, in Bayelsa State the government agreed to Social Action’s request to establish a website to publicize income and expenditure data, but it never made the website public. Furthermore, the State unilaterally decided to cease releasing the budget, something it had done in the past. Comparable policy-reform reversals have been reported in Akwa Ibom as well, where after making the state budget public for two years, those figures again disappeared from public view. According to Social Action, these setbacks could have been prevented had the cluster acquired better advocacy skills engaging policy-makers at the state rather than the local level, where the genesis of some of its more consequential accomplishments may be found. As one informant noted, SACE’s work remains unfinished, as the training it provided was insufficient to allow Social Action and its partners to acquire the skills required to appropriately influence the policy-reform process at the state level. In Social Action’s case, the achievement of Capacity 1.0 competency—reflected in the award of a MacArthur Foundation anti-corruption grant to the organization—has not been sufficient to overcome Capacity 2.0 limitations, traceable to SACE’s inability to provide more refined advocacy training as the Activity closed. It is possible that Social Action did not have the necessary capacity to assimilate the SACE-provided advocacy training or had to contend with more recalcitrant state policy-makers, less receptive than local ones to civil society calls for policy reform. A QUANTITATIVE VIEW OF THE SACE MODEL SUSTAINABILITY With more than 50,000 NGOs and CSOs, the landscape of Nigerian social organizations contains a rich and varied tapestry of committed stakeholders seeking to find solutions to the country’s many challenges of poverty and marginalization. The SACE approach, as suggested by our findings, offers a potentially viable option that many of these social organizations may be willing to embrace, were they to believe it would be useful to advance their institutional agendas. While the scope of work for this evaluation did not specifically include assessing the likelihood of replicating the Activity beyond the anchors and cluster members involved, the Activity invited non-SACE CSOs to participate in two “Share-out Workshop” (dissemination) events at the end of the Activity to discuss lessons learned. The ET designed and circulated a survey questionnaire (Annex 9 in the Work Plan) sent via email to all (51) participants of the Abuja and Port Harcourt events. Survey results may be found in Annex VII of this report. About half of the respondents answered the survey (23 of 51, a 46 percent response rate). The majority found the “Share-out Workshop” useful, since participants were exposed to the SACE tools suite. The value of the collective-impact approach was reportedly most useful, with about half of those responding claiming to use some of the SACE tools—mostly at the state level—with a nearly equal number indicating they had been searching for additional information regarding the SACE advocacy approach. The range of advocacy topics addressed by these respondents was broad, ranging from health and education reforms, policies to increase agricultural value chains, implementation of the OGP, and reforming the criminal justice code. In summary, survey results suggest broader dissemination of the SACE tools among the CSO community would be welcome; there is a need for more effective means to bring about policy reforms. How have the Innovation and Spread the Word grantees fared? Noted in the SACE administrative reports and confirmed during interviews, partners during SACE’s engagement with the clusters had difficulties identifying concepts to integrate innovative approaches in their advocacy and civic engagement plans. As part of SACE’s activity implementation, it was decided to set up an Innovation and Spread the Word Fund (ISWF) to support “promising practices” and nurture innovations to improve transparency and accountability. As part of this initiative, SACE drove innovations 18 | SACE FINAL EVALUATION REPORT and published criteria for the launch of the ISWF Fund. The call for innovations was specifically designed to promote improvements in engagement and public awareness. The ISWF approach included trainings and SACE award of grants to selected ISWF after a call for proposal. After carefully selecting partners to provide technical support, seven innovation organizational partners were chosen. The ET was able to interview five of the seven grant recipients to identify the status of their innovations following the end of the Activity. Some partner organizations were able to sustain their momentum, while others have integrated their projects into newly funded activities. Some of these organizations are currently upgrading their innovations and continue using them in support of various advocacy campaigns. Brief summaries of the innovations, together with descriptions of successes and challenges they face (including sustainability outlooks) may be found in Annex VIII of this evaluation report. This section describes three of those innovations, while giving added emphasis to the one (ConsTrack) that, from the ET’s perspective, has the most transformational potential and is worthy of replication. Digital Story Telling is an innovation developed by Spaces for Change (S4C) to allow youth in the Lagos slums to communicate some of the painful experiences poor residents of the city confront on a daily basis with policy-makers and a wider audience. By filming videos in their communities and sharing them at public events, the young men and women of Digital Storytelling—trained through an ISWF grant to S4C—have acquired production skills (videography, script writing) to vividly portray the human costs of evictions and lack of community services. One example was a video used to illustrate what tragic consequences were associated with the inability of the Lagos State Fire Services to reach slum residents, as it highlighted the loss of life due to slow response times. As a result of the ensuing dialogue the video provoked, S4C was invited to collaborate with the Lagos State Fire Services to identify innovative ways to more readily access the slums and save lives. Other equally impactful videos have addressed the issue of forcible evictions and what they entail for the welfare of poor women and their children. An unanticipated benefit has been that, as a result of the training imparted under the ISWF grant, S4C added to its permanent staff some of the more skilled young activists; others have managed to gain employment with other community-based organizations. Still others decided to become start-up photography and videography entrepreneurs. “Your Excellency” is a mobile application developed by YAF, designed to appeal to youth by involving them in games that deal with governance and civic engagement. The idea behind the app was the realization, following an assessment, that youth were generally apathetic towards the issue of governance in Nigeria. As a result, YAF was reluctant to sponsor additional seminars on Youth Civic Engagement. The conclusion was that gaming could become a more enticing platform for youth to learn about these issues, as the app would also satisfy some of their educational and entertainment objectives. As a result, the “Your Excellency” gaming application was born. Currently it has been downloaded more than 500 times and youth are continuing to access the app since the Activity ended. To enhance its appeal further, YAF is in the process of upgrading the “Your Excellency” app to provide more governance and civic￾engagement-related content to attract more young users. ConsTrack is a technology application managed by Order Paper that works by tracking how constituency projects are implemented. This enables citizens to track their legislator’s constituency projects and what they were to accomplish in the process (see Box below). The app can be downloaded from the Google Play Store and, since the SACE project ended, from iOS. Among other accomplishments, the development of the ConsTrack app has led to a relationship-building process among various stakeholders such as the media, members of Parliament, and other agencies. While currently operating in three states, there is 19 | SACE FINAL EVALUATION REPORT potential to scale the application to Nigeria’s 36 states. To that effect, the technology underlying the app has been upgraded to reach all parts of the country since SACE ended. Order Paper’s sustainability plan is to identify more donors to further expand the application to other States and constituencies. Some National Assembly delegates and officials at the Ministry of Budget and National Planning have become receptive to the app. The Ministry’s Director of Monitoring and Evaluation has discussed with Order Paper the possibility of transforming ConsTrack into a national tracking tool, thus extending its reach beyond the three original states were the technology was deployed. As a result, Order Paper was requested to submit a Concept Note to the Ministry, to include ConsTrack as a line item in a budget-support request for a donor. In July of 2019, the national chapter of the Nigerian Union of Journalists (NUJ), an inclusive union of media professionals, indicated its willingness to collaborate with Order Paper on a Constituency Project Monitoring using ConsTrack. There is a Memorandum of Understanding in place which both parties should sign in September. The implication is that the app will provide the reach to properly track projects in remote Nigerian communities using NUJ-accredited community journalists who are knowledgeable about their localities. These tracked projects would also be given the publicity they deserve. Discussions have also been held about integrating the ConsTrack app into the Open Budget - Improved Service Delivery Commitment of the Fiscal Transparency Working Group of the 2nd National Action Plan (NAP) of the OGP in Nigeria. The app, among others, would be used to monitor citizens’ satisfaction with the services and responsiveness of the MDAs. Furthermore, SERVICOM, a Federal Government initiative with the mission of improving citizen satisfaction by promoting excellence in public services, has indicated its desire to collaborate with ConsTrack on project delivery, as stated in a letter seeking a partnership. ConsTrack: An Innovative Spread-the-Word Sustainability Success Story In partnership with Deji360, a television program, ConsTrack exposed how a budgetary allocation for a constituency project in the small and remote community of Kwaku, about 50 kilometers from Abuja, was in reality a corrupt scheme. The line item in question, intended to build an earthen dam, was part of the 2016 federal budget and directed N150 million to the Ministry of Water Resources. The investigation revealed the Kwaku citizens were unaware of the project as they had never been consulted and were shocked when informed such a large sum had been assigned for an alleged community project. When the legislator representing the district was confronted on camera, he claimed that when informed of the line item’s approval, he dispatched a team of surveyors who reported Kwaku was so inaccessible, heavy equipment would be unable to reach the proposed dam site. On that basis, the funds were reallocated and remain unaccounted for. The expose, initially broadcast in late 2018, also revealed the legislator in question controlled a further N760 million for six other community projects—none of which had been executed. As a result of the revelations, the legislator lost his bid for re-election in 2019. More information on the story, which was shown on national and cable television (DSTV) may be found at: https://www.youtube.com/watch?v=ZrFGxmOL65E&t=84s https://www.youtube.com/watch?v=I7MLeZpIlCA. 20 | SACE FINAL EVALUATION REPORT CONCLUSIONS The ET’s findings indicate that SACE successfully achieved the majority of the Activity’s goals and objectives. It contributed to strengthening the organizational and advocacy capacity of anchors and cluster CSO coalitions while managing to promote policy reforms in favor of marginalized populations and other objectives across various sectors at the national and state levels. These sectors included budgetary transparency, educational reforms, improved maternal/child health care, and better working conditions for women farmers and stall sellers. The ISWF also has been sustainable post-SACE. These achievements were made possible by the application of the anchor/cluster model and related SACE tools. While some reforms have yielded substantial gains, PWD for example, others have not come to full fruition. Other possibly consequential reforms have not borne fruit—the policy change process is complex, slow, and likely to face many obstacles. In some instances, anchors failed to achieve the capacity to operate independent of USAID support (as defined by their ability to receive grants from other donors) due to insufficient training associated with time limitations related to a project’s life cycle; SACE did not provide as much support as was needed. The field work confirmed that formerly associated CSOs continue to apply the SACE methodology. Overall, several factors appear to favor sustainability, while others undermine it. Those likelier to sustain had the initial institutional capacity of individual anchor CSOs and the ability to assimilate the SACE-imparted organizational and advocacy capabilities. Good examples of these CSOs are ANEEJ, CENSOJ, and HDI. Some initially weaker and less known anchor CSOs, such as YAF, were able to succeed through effective leadership, particularly if exerted by highly motivated younger individuals not content with the status quo and willing to embrace new ways of doing things. LAPDO offers another example of dynamic leadership, as even though this anchor joined SACE late, it achieved many successes in a short period of time. Therefore, the role of leadership, together with initial institutional conditions, appears to be of some significance in ensuring sustainability. The willingness of some clusters to build on the successes of other clusters was also noted, even those working in unrelated topics. LAPDO, for example, took advantage of ANEEJ achievements in its open-budgeting initiative to negotiate with Ondo State officials for increased budget allocations for women farmers. Another major marker of sustainability is the continued CSO commitment to join forces and advocate for common causes with one voice. Continued reliance on the anchor/cluster model is a predictor of advocacy success, as is the persistent application of some of the SACE tools. The continued success of WARDC, CENSOJ, and YAF attest to this usefulness. In some instances, however, the application of such SACE tools may be a necessary but insufficient condition to overcome other constraints. The converse is also true—originally weak anchor CSOs could not overcome organizational and occasionally advocacy deficits, despite the training provided by SACE. In several instances, effective leadership and civic commitment helped account for advocacy successes, even though (by the criteria used in this evaluation) the sustainability threshold may not have been crossed. That is the case, for example, with NNEW, CITAD, and HERFON; while they have not become “technically” sustainable in the sense that they have been unable to receive donor support, they scored impressive achievements in pursuit of major policy reforms in maternal and child health, increased educational funding for youth, and more favorable policies for poor market women. Some of these achievements may be jeopardized as former anchor CSOs can no longer count on SACE support. For example, there is evidence some of these anchors 21 | SACE FINAL EVALUATION REPORT are no longer capable of sustaining prior “speaking with one voice” advocacy initiatives, as they currently lack funds to coordinate activities. In some cases, the support provided by SACE extended beyond technical and financial assistance. Given the major role USAID and other American institutions play in supporting health care programs in Nigeria, policy-makers in this sector may have been more receptive to advocacy initiatives sponsored by anchors, such as CENSOJ and HERFON, than they otherwise would have been. This conclusion, suggested unambiguously by a HERFON associate, is at the heart of some of the recommendations made by the ET regarding a need to educate the policy-making community at all levels of government about their duty in a democratic society to listen to and collaborate with civil society in the enactment of public policy. The ET was able to document a number of instances of unfulfilled political commitments, including backsliding with respect to some initial achievements in the Niger Delta. Similarly, the ET saw how fruitful the collaboration could be between reform-minded policy-makers and an effective CSO leadership, as demonstrated, for example, by Joint National Association of Persons with Disability (JONAPWD), advocating on behalf of PWD; LAPDO in favor of women farmers; HDI to benefit young students; or ANEJJ for the establishment of OPADECs in the Niger Delta. RECOMMENDATIONS On the basis of the ET’s findings, and suggestions by key informants, it is recommended that any follow￾on civil society advocacy activity should: ▪ be sensitive to the fact that training requirements may vary significantly from one CSO to the next, and plan training intensity differentially, assuming it will take longer for some CSOs than others to fully develop organizational and advocacy skills; ▪ provide more direct contractor training to cluster members than SACE did, as most of the training provided by the Activity was geared primarily to benefit anchor CSOs and only secondarily to cluster CSOs; ▪ develop training programs specifically designed to educate elected officials at all levels of government about the citizens’ role in the formulation of public policies, to inculcate and/or reinforce reform-advocacy tendencies and encourage stakeholder collaboration. Such training could be included as part of the orientation training already offered to elected officials when assuming office at the Federal level and should be provided as well for elected officials at the State and local levels; ▪ select and train good-governance-oriented CSOs as providers of trainings to remove partisan considerations and establish collaborative relationships between elected officials and the CSO community; ▪ implement programs along similar lines for high-ranking career public officials at all levels of government, as they have the power to act as knowledgeable gatekeepers and are generally tasked with drafting and implementing consequential policy-reform regulations; and ▪ support permanent establishment of a national advocacy institute charged with disseminating across Nigeria’s large and diverse NGO/CSO community the SACE experience, as well as other future advocacy modalities that may become available to promote policy reforms across all sectors of the Nigerian polity. 22 | SACE FINAL EVALUATION REPORT EVALUATION QUESTION 2 FINDINGS EVALUATION QUESTION 2 - How have SACE partners’ financial and technical advocacy skills (Capacity 1.0 and 2.0) influenced their ability to engage with government on reforms? What has worked and what has not worked? Respondents were generally eager to share their CSO’s experiences with the SACE advocacy tools with members of the ET. It was seldom necessary to probe for recall, as the vast majority of key informants and FGD participants spontaneously retrieved the names of the various tools. Equally telling was the ease with which the informants recalled how the tools had been used while the project was being implemented and the extent to which they continue to be used (or not) after close-out. Have Systems for Transformation and Results NETWORK (STARNET, now PANDO), Outcome Harvester, Policy Tracker, and other advocacy tools been effective? Overall, SACE advocacy tools were effective for anchors and clusters. Some anchor/clusters took far greater advantage of the SACE tools than others, since some tools proved to be more usable and user￾friendly and, in the end, more resilient. These continue to be used more often than others. Respondents agreed the anchor/cluster model and SACE-related tools helped to discourage uncoordinated messages and allowed CSOs to target actionable, specific, and concrete messages. Through the SACE tools, a coordinated, nuanced, informed, and flexible approach could be followed, attuned to the evolving socio￾political and economic environment. They were also sensitive to the role played by different stakeholders, be they intended beneficiaries, the media, influencers, or policy-makers. As one key informant observed: “…our work along the anchor/cluster model had started long before SACE as we had been mentoring other CSOs. What SACE did was help us improve our efficiency and diversified the means whereby we could achieve our objectives.” The power of collective advocacy and speaking with one voice also emerged during our interviews as a recurring theme, as was the notion of “leading from behind,” another built-in strength of the SACE advocacy model. In most evaluation sites, informants noted that the anchor/cluster model was being replicated in other non-SACE-related advocacy efforts. As noted by a SACE anchor, “We have been able to utilize the cluster model in other related campaigns within our organization and continue to use the model since the SACE project ended.” The sustainability of the model and continued use by Nigerian CSOs seems assured. Another feature of the model that garnered repeated praise was its training component. The objective of the SACE design is not only to upgrade the policy-reform advocacy capacity of cluster members, but also to strengthen organizational capacity. Key informants found the objective of gradually diffusing through the cluster the advocacy knowledge first imparted to anchors as particularly meritorious, as the latter eventually began mentoring the former. 23 | SACE FINAL EVALUATION REPORT To what extent are the SACE partners using the tools and learning from the SACE activity to inform program design and implementation, and are they prepared to continue doing so beyond the evaluation period? While the existing anchor/cluster organizations continue in operation in most sites visited, in several they became less active since the end of 2018, when the SACE project ended. An important factor noted by informants in practically every Nigerian area visited was the distraction caused by the early 2019 general elections. This drew attention away from all other advocacy activities in which governance-related CSOs were involved. As one key informant related, the focus of the CSO community turned to the significance of electing the right candidates willing to endorse civil society reform objectives. In other instances, however, the lack of SACE funding created a vacuum that has eroded the anchor/cluster relationship, although anchors and individual CSOs may loosely continue focusing on their areas of interest. Regarding reliance and continued use of the SACE tools, the impressions collected by the ET are more mixed. Some informants praised STARNET, while others were far more reserved in their assessments. A CSO anchor representative, for instance, described STARNET as “the best tool that came out of our participation in the SACE project.” Yet, this same informant indicated some CSOs in his cluster no longer rely on STARNET and depend instead on alternative partnering methods to achieve similar goals. What this comment implies is that while the Internet-based tool may not be used, the purposes for which it was designed continue to be at the forefront of CSOs adopting the SACE advocacy approach. It appears that past and continued use of STARNET is mediated by several factors. One is SACE’s insisting that participating institutions rely on STARNET while the project was underway. STARNET’s usefulness and applicability was emphasized during the various training and learning events, with clusters rating how well they used the system. Following close-out, this focus seems to have waned. For instance, in the presence of two members of the ET the Executive Director of a prominent Abuja anchor reminded a staff member that they needed to update their STARNET data as they had not done so since December 2018. With the end of SACE funding, accessing STARNET among some anchors has become an afterthought— perhaps because the number of new initiatives has lessened, as noted by a participant in a Niger Delta focus group discussion, or perhaps because SACE staff members are no longer prodding anchors to use STARNET regularly. Yet, other key informants emphasized that despite the fact that fewer advocacy activities are being pursued, STARNET continues to be a crucial advocacy tool as during the heyday of SACE implementation. As one informant remarked, the usefulness of STARNET is particularly evident when working across state lines, as it helps CSOs identify partners they would otherwise not know about, a reason why this anchor still relies on the system. Technological factors appear to be a major hurdle conspiring against continued reliance on STARNET. In areas of Nigeria such as Lagos and Abuja, where Internet connectivity is best, former anchor and cluster members may find it easier to rely on the system. According to STARNET users, poor Internet service plagues current and potential users. As one informant remarked, “STARNET didn’t actually work for us, it was too complex and didn’t work well because the driver wasn’t user-friendly.” Another reason behind STARNET’s limited usefulness was limited educational attainment and connectivity rates among low income, marginalized populations. For example, cluster CSOs constituted by small subsistence women farmers had neither access to Internet networks nor the knowledge on how to use them. Without SACE staff prodding, many CSOs have apparently decided the difficulties confronted when accessing the system are not worth the trouble. Still other anchors, convinced of the networking advantages, have moved 24 | SACE FINAL EVALUATION REPORT forward and devised their own “low-tech” STARNET systems to overcome continuous connectivity or cost issues. An anchor in Northern Nigeria described to the ET how they “have improvised and created an offline/analog manual version of STARNET.” Unlike the varied response to STARNET, informants agreed other SACE advocacy tools were useful. Often cited as most useful and frequently used were Cluster Coaching, Policy Tracker, and Outcome Harvester. These three tools were invariably viewed as exceptionally useful to monitor and fine-tune advocacy approaches. Almost without exception, they were said to be in continued use, as they had become a CSO advocacy standard. Flip charts observed by ET members at various locations indicated how CSOs had internalized the fundamentals of these tools. Equal allusion was made to the usefulness of the collective impact model and relevance of continuously engaging in political economic analysis to adapt the advocacy strategy to changing circumstances and opportunities. One Niger Delta key informant, while most complimentary of all SACE tools, was most effusive about the benefits derived from the Cluster Coaching model describing it as; “very useful, really helped us, kept us heading in the right direction and aligned away from distraction while ensuring cluster collaboration.” While the findings described suggest continued SACE tool use, frequency of use will likely depend on the value assigned to each tool and level of difficulty associated with its use. Again, STARNET was consistently mentioned as the tool least likely to be used in the future, mostly due to of accessibility issues or because, as one informant noted, for STARNET to be used effectively “a constant capacity-building process is required as it was provided by SACE.” What appears certain is that the largest and more sophisticated anchor CSOs seem committed to continue relying upon STARNET to identify future partners as they venture into new advocacy reform areas. This is probably because they are in a better position financially and technologically to benefit from its use. Whether or not STARNET will remain as useful as it once was remains to be seen, as technological barriers have kept many CSOs from using it. Regarding the future use of other SACE tools, the evaluation findings track well with those reported for previous use in the Successes, Challenges, and Sustainability Issues Survey. Again, Cluster Coaching, Outcome Harvester, and Policy Tracker appear to be the preferred tools; these tools are described by one informant as instruments of continued use upon which his CSO will continue relying well into the future. Other informants mentioned that these particular tools had already begun to be applied in new projects even before SACE ended and that undoubtedly their use will continue. Some anchors were said to be already using the complete array of SACE tools to train newly established clusters organized to pursue new reforms. Among these were YAF and CENSOJ, the latter indicating it depended on the Cluster Coaching and other tools to train new CSO inductees in the newly established Government Reform Audit cluster. A JONAPWD spokesperson made a similar claim, emphasizing that the organization was planning to use Cluster Coaching and Outcome Harvester, among other SACE tools, to train members of clusters established in association with the national organization for the blind and advocate on behalf of disabled women farmers. Parallel plans to continue relying on the SACE methodology were reported by YAF, HDI, WARDC, LAPDO, ANEEJ, Citizens Centre for Integrated Development and Social Rights (CCIDESOR), and ESCFAL, among other SACE anchors. Former cluster members also plan to extend the SACE approach to reach into other advocacy areas, as Centre for Community Empowerment and Poverty Eradication (CCEPE) is already doing in Ilorin, Kwara State, through a recently established cluster to convince the state government to endorse the OGP, i.e. a goal parallel to that pursued by other clusters 25 | SACE FINAL EVALUATION REPORT in the Niger Delta under the “Open Niger Delta” initiative. As observed by a former SACE leader, one of the strongest indicators of the model’s sustainability is that even before the project ended, many clusters had already started to use the SACE tool in unrelated activities. A quantitative perspective: A complementary, triangulating perspective regarding continued SACE tool use after close-out was sought by the ET through the survey questionnaire sent to all anchor and cluster members (the Successes, Challenges, and Sustainability Survey) via email. Its aim was to generate information on CSO activities, including past successes, challenges, and sustainability strategies. The survey included items regarding continued use of SACE tools. The survey yielded a disappointingly low response rate (21 percent, or 24 of 115 CSOs to whom the questionnaire was sent responded). This was partly because many cluster CSOs were reticent to respond unless instructed to do so by their former anchors. By and large, survey results concur with KII and FDG data. The survey questionnaire is presented in Annex 8 of the work plan and the survey results in Annex VII of this evaluation report Continued advocacy after close-out: Slightly above 90 percent (22 out of 24) of respondents indicated they continued actively engaged in advocacy activities after SACE close-out. Most respondents reported engaging in more than one advocacy effort since close-out, with some involved in as many as five distinctive initiatives. Similarly, the same percentage (22 out of 24) were somewhat successful in their advocacy engagements, although an almost equal number (18 out of 24) or 75 percent, described the process as challenging. Continued SACE tool use: Nine out of ten respondents claimed to be still using some of the SACE advocacy tools, Cluster Coaching being the most frequently used, followed by Policy Tracker (45 percent) and Outcome Harvester (40 percent), as shown in Figure 1 below. Continued STARNET use was reported as low; even though it is not a preferred tool, over half of respondents indicated the STARNET tool had been useful in improving their advocacy skills. Fewer former users may be using STARNET, as indicated previously, as there are other easier (and cheaper) alternatives, such as WhatsApp. In order to access STARNET, users must login to an Internet platform, whereas WhatsApp is directly connected to a phone number. Also, for a variety of reasons, at least one anchor, LAPDO, was unable to access STARNET, perhaps because it joined SACE during the last year of the Activity’s implementation. In other cases, potential users lack the required level of education to access the system. Some anchors continue to use variants of these teaching fora for continued training or to introduce new cluster members to the SACE methodology. Potential future STARNET use seems assured as Root Change has already agreed to allow the Nigerian Network of NGOs in Ibadan to access and use the database developed under SACE, although the underlying software is proprietary rather than open source. Whether or not Root Change will be willing at a future date to share its source code is unclear at this time and may depend on future contracting options. Access to the SACE documentation, including that related to STARNET, should not be problematic as it has been stored by USAID. 26 | SACE FINAL EVALUATION REPORT Figure 1. Use of SACE STAR Methodology and Tools Among the System for Transformation and Results (STAR) drivers, Stakeholder Engagement was favored by half of the respondents, followed by (in order) Alliance Building, Public Awareness, Knowledge Exchange, and Adaptive Voice and Accountability. Regarding most useful learning events, nearly 50 percent of respondents chose Cluster Coaching, followed by the Sustainability Workshops, Issue Learning Forum, Social Media Week, and Innovation Design and Partners Workshops. Social media use: Among former SACE-supported CSOs, social media engagements slowed down since close-out as suggested by less activity in some #hashtags such as #mindset2.0, with 36 percent of CSOs claiming to have conducted between one and five engagements; another 22 percent from six to ten; nine percent from ten to 20; and 13 percent with more than 20 such engagements. While Mindset 2.0 is not as active as when SACE was underway, some clusters continue to use it. The ET is aware that CENSOJ continues to rely on Mindset 2.0 to facilitate stakeholder engagements and knowledge exchanges among members of its Audit Reform cluster, while YAF depends on it to conduct learning exchanges under its DFID-funded Youth Partnership Anti-Corruption project. Relationship with Government: A plurality of CSOs (37 percent) reported that the SACE interventions had been most helpful in building sustainable relationships with government officials, followed by 27 percent claiming they had been helpful, 23 percent stating they had been moderately helpful, and five percent not helpful at all. Six of every ten respondents indicated the SACE project had been most helpful in advocating for policies in their target thematic area since engagement. However, only two out of ten attributed moderate success to SACE, the rest claiming no impact. Finally, about half of the CSOs claimed to have successfully advocated for the approval of legislative initiatives, with two out of ten claiming their advocacy resulted in approval of a law after close-out. The remaining respondents, close to 40 percent, reported that they were still advocating in favor of other reform policies and legislative initiatives. Are the SACE tools accessible to potential future users? This issue came up several times while conducting KIIs. Opinions were divided; some respondents regarded the question as a non-issue, while others saw the matter as perhaps one of SACE’s primary failings—neglecting to ensure the availability of the SACE methodology for potential future users. Among the former were those, including some former SACE officials, who felt didactic materials about SACE processes and tools had been widely distributed during SACE-sponsored training sessions and amply shared with anchor CSO directors. These individuals, via thumb drives and other electronic venues, would then share those materials with cluster members. 27 | SACE FINAL EVALUATION REPORT Those who felt this dissemination approach was insufficient noted to the ET that SACE failed to leave behind a lasting repository of training materials, something inconceivable these days, as a dedicated website would have been sufficient to provide efficient and economical access to the SACE documentation to some of the thousands of Nigerian CSOs that someday may wish to learn about and apply the SACE methodology. The ET itself is puzzled as to why this was not done, although it understands the materials are archived in a USAID electronic storage facility. Are there additional SACE anchor partners ready to receive direct funding from USAID? The ET conducted a “Capacity 1.0” Organizational Capacity Assessment with the aim of assessing critical elements for effective organizational management. The capacity assessment conducted was an abridged version of the original Organizational Capacity Assessment Tool (that is, based on a four-point scale), using instead a three-point scale through which anchors were asked to rate their capabilities across multiple organizational, management, and financial categories (Basic, Moderate, or Robust). Once these ratings were obtained through responses provided to the electronic survey distributed through a KoboToolBox platform, the ET (in most cases while conducting the evaluation field work) validated the self￾assessments through personal interviews and visual verification of the evidence (e.g., availability of organizational manuals). Once these procedures were complete, individual CSO results were compared with the far more exhaustive August 2018 assessments conducted by Widows & Orphans Empowerment Organization (WEWE), a SACE sub-contractor, for the 16 SACE anchor sub-grantees. The purpose of the WEWE evaluation was to determine the relative effectiveness of the capacity-building interventions provided by SACE over a 12-month period. Within the limits of its available resources, the ET carefully studied the scales and definitional criteria at the heart of the WEWE assessment to determine the comparative suitability of both sets of criteria and scales for purposes of this evaluation. A limitation of the procedure is that it depended primarily on self-evaluation judgments that could be colored by the personal perspectives of specific informants and the relative organizational, administrative, and financial strengths of individual anchor CSOs. It was found that some of the strongest, better established and larger CSOs were prone to be more careful and tentative in their assessment than other anchor CSOs, who appeared to be more casual and decisive about their judgements. Nevertheless, the broad correspondence between the ET’s findings and those of WEWE suggests our methodology was sufficiently reliable to produce useful comparisons. The comparison between WEWE’s and the ET’s Organizational Capacity Assessments produced the results shown in Figure 2. The interpretation of the ET is that the relative capacity of the various anchor CSOs has changed little since the 2018 WEWE assessment. At this writing, seven CSOs appear to be qualified enough to receive direct USAID funding, with the caveat that two of them (Civil Society Legislative Advocacy Centre (CISLAC) and LAPDO) scored higher in the ET’s assessment than they did in the 2018 WEWE exercise. The ET was unable to rate three of the anchor CSOs, as they could not be reached while the ET engaged in field work. 28 | SACE FINAL EVALUATION REPORT WEWE GRADING (2017) CSO EVALUATION TEAM GRADING (2019) CSO Very Strong None N/A N/A Strong ANEEJ, CENSOJ, HDI, LITE AFRICA, WARDC, YAF Robust ANEEJ, CENSOJ, CISLAC, HDI, LAPDO, WARDC, YAF Moderate CISLAC, CITAD, HERFON, JONAPWD, LAPDO, NCDF, SDIC Moderate CITAD, HERFON, NNEW, SDIC, JONAPWD Weak NNEW, UPFFA, ESCFAL Basic CCIDESOR, ESCFAL • CCIDESOR and Disability Rights Advocacy Center (DRAC) were not evaluated by WEWE • LITE Africa, Ugboroke Progressive Fish Farmers Association (UPFFA), Nembe City Development Foundation (NCDF) were not assessed because their sites were not visited. • DRAC information arrived too late and data not validated • The ET used three categories, while WEWE used four. Figure 2. Comparative Results of the 2018 WEWE and 2019 Evaluation Team Organizational Capacity Assessments CONCLUSIONS Among the anchor/clusters contacted, the SACE methodology has been largely internalized; informants were familiar with the various tools and their uses. The qualitative and quantitative data also suggest that the SACE approach continues to be operative, as it is evident that all former SACE-associated CSOs are still, to some extent, using the SACE tools. Some, like Cluster Coaching and Outcome Harvester, continue to be favored, while others, primarily STARNET, less so. The SACE tools have played a useful role in enhancing the advocacy capacities of the partner CSOs; these skills and related tools are likely to continue to be used in policy-reform advocacy. With respect to organizational improvements, gains in general have been substantial, with about half of SACE anchors currently being capable of directly managing USAID grants. In fact, partly based on this achievement, some anchors have already been awarded grants by donors such as DFID and the Friedrich Ebert Stiftung. Those anchors failing to qualify continue to suffer from organizational deficiencies. RECOMMENDATIONS On the basis of the ET’s findings, including suggestions by key informants, it is recommended that any follow-on civil society advocacy activity should: ▪ Impart intensive training regarding financial and fund-raising skills, including proposal writing; ▪ Rely, to the extent possible, on more flexible administrative procedures to allow partner CSOs to take advantage of windows of opportunity and implement flexible work plans with more alacrity; ▪ Evaluate program approaches to gradually improve organizational capacities among Nigerian CSOs independently of whether or not they receive USAID support; 29 | SACE FINAL EVALUATION REPORT ▪ Provide support for the establishment, maintenance, and update of a clearly identifiable, user￾friendly website to inform potential CSO users about the availability of the SACE methodology. This should be an easily implemented recommendation as USAID preserved the SACE documentation in an off-line cloud-based archive; and ▪ Support permanent establishment of a national advocacy institute charged with disseminating the SACE experience across Nigeria’s large and diverse NGO/CSO community, as well as other future advocacy modalities that may become available to promote policy reforms across all sectors of the Nigerian polity. EVALUATION QUESTION 3 FINDINGS EVALUATION QUESTION 3 – What impact has SACE had in influencing government policy development and implementation regarding vulnerable populations such as women, youth, and persons with disabilities within the six months post-SACE? SACE accomplished its project design objective of having clusters include marginalized populations such as women, youth, and PWD in their policy-reform agendas. Some clusters were actually assembled to address the policy-reform priorities of such constituencies, while under SACE’s tutelage others embedded at least some of that policy priority in their agendas. Among the organizations with focused vulnerable population mandates are JONAPWD, YAF, LAPDO, and WARDC. Other anchors, such as CENSOJ and HDI, included in their policy-reform agenda the directive of serving marginalized populations, whether by advocating for inclusive practices in the assignation and execution of more inclusive maternal and newborn health budgets, or by promoting inclusive educational practices. Annexes VIII, IX, and X of this report provide an anchor-by-anchor summary of continuing post-close-out efforts on behalf of marginalized populations, including ISWF efforts along these lines. The ET’s field research findings, along with some of the survey results, corroborate that these achievements have been sustained and, in fact, may even be increasing after close-out. The efforts of JONAPWD are worthy of attention, as its cluster continues expanding the policy areas requiring reform if PWD needs are to be satisfied. The ongoing work of LAPDO and WARDC advocating on behalf of marginalized women farmers, or of NNEW seeking to improve working and living conditions of female stall sellers, continues to be strongly pursued even with exiguous funding sources, as the women themselves have become empowered and have developed the confidence to advocate on their own behalf. This is a largely unheralded, more subtle SACE achievement that the ET observed, as its advocacy interventions have served to generate attitudinal changes that can only expand in years to come; more and more members of marginalized communities are coming to appreciate more effective advocacy approaches and bring about beneficial policy reforms. Other sustained SACE interventions on behalf of marginalized populations include those initially advocated by ISWF partners, whether by proposing and implementing concepts specifically designed to improve the plight of marginalized populations, or by explicitly including such populations within reform-oriented innovations with broader policy objectives. Specific examples may be found in the Evaluation Question 1 section discussing the current activities of former ISWF recipients. The inclusion of marginalized populations was also addressed by adding several items regarding this topic to the Successes, Challenges, and Sustainability Issues Survey questionnaire (see Annex VII). As would 30 | SACE FINAL EVALUATION REPORT have been expected, nine out of ten respondents indicated having included marginalized populations as an agenda policy priority, while the remainder said their agendas did not prioritize marginalized groups. About half reported prioritizing and engaging women in their advocacy tasks, close to 40 percent promoting PWD, and 20 percent engaging with youth issues. Nine out of ten respondents indicated their engagement with SACE made them advocate with government on behalf of marginalized populations. Finally, the vast majority of the CSOs, close to 90 percent, reported that the SACE intervention strengthened their organization’s advocacy capacity to represent marginalized groups, with six out of ten claiming to have initiated, or being in the process of, passing legislation on behalf of marginalized populations. These findings provide an unexceptional buttress to those derived from the qualitative information. What are the documented policy milestones for which SACE can claim attribution? SACE, during its five-year implementation period, was able to register numerous accomplishments. These ranged from assisting partner anchor CSOs to increase their technical organizational capabilities to helping instill in them (and their clusters) a more sophisticated culture of policy-reform advocacy. After the SACE close-out, most anchor/clusters have repurposed their advocacy strategies to some extent, while continuing to foster sustainable relationships with policy-makers, building on policy achievement milestones at least partly attributable to SACE interventions. The ET’s field research findings show that, in some instances, inter-cluster collaboration is assuming a life of its own. LAPDO, a cluster anchor in Ondo State, was able to build some of its achievements on the Open Budget Partnerships spearheaded by another anchor, ANEEJ, to influence the Ondo State Ministry of Budget and Economic Planning to insert a budgetary allocation for small-holder women farmers (described below). CSOs like NNEW and WARDC were able to empower market women and women farmers, respectively, by advocating for more responsive regulatory environments leading to more stable and reduced market stall prices and more gender-equitable agricultural policies. Since the SACE close-out, a “no street-hawking” law has been approved to relocate women hawkers into markets, while WARDC continues insisting on the need to provide women farmers with their own agricultural land and gender-specific extension services. Other organizations, like CENSOJ, after having achieved partial but significant policy-reform successes with SACE support, continue their advocacy efforts to realize the promise of such achievements. Such is the case with YAF’s continued insistence that the Akwa Ibom State Youth Inclusion Bill be finally signed by the governor. Project achievements were documented in numerous ways by SACE staff. Detailed quarterly, annual and final reports list accomplishments according to the indicators and intermediate results required under the Mission’s Country Development Cooperative Strategy, subsumed under USAID/Nigeria’s Development Objective 3, specifically Intermediate Result 3.3, “Increased civic advocacy, monitoring, and engagement,” and their corresponding indicators. These sources could be consulted for overviews of the policy-reform achievements partially attributed to or claimed by SACE. The remainder of this section summarizes further post-SACE close-out policy-reform achievements by individual anchors or selected cluster members including, when appropriate, new grant fund awards reported to the ET. More detailed accounts are shown in Annex IX. One of SACE’s predictors of eventual advocacy success was the ability of anchor CSOs to secure funding from institutions other than USAID. Achievements in this regard have been significant, as the number of such awards has been on the rise; several former anchors have diversified their funding sources significantly by winning one or more grants. Such has been the case with ANEEJ, CENSOJ, CISLAC. HDI, and JONAPW. Major achievements could 31 | SACE FINAL EVALUATION REPORT be counted in terms other than monetary as several of these former anchors (as well as others) have been instrumental in furthering additional policy reforms or influential in achieving significant benefits and concessions for their marginalized community constituencies since close-out. At the risk of being redundant, it is worth mentioning again the policy-reform successes of WARDC and LAPDO on behalf of women farmers and of NNEW in favor of female informal sector traders. Despite internal leadership issues, JONAPWD continues shattering traditional barriers through its consistent policy victories defending PWD rights. CITAD and YAF each in its way advocate for youth and educational reforms, just as HERFON and CENSOJ continue battling for improved maternal child health programs. Last but not least, several governance-oriented anchors, such as CISLAC, CENSOJ, ANEEJ, and CCIDESOR—from Abuja and in the Niger Delta—persevere in their efforts to promote transparency and accountability in Nigeria’s notoriously opaque public sector. These initiatives persist despite SACE’s end; they are a tribute to what the activity managed to accomplish. This conclusion should not negate the fact that absent USAID institutional and financial support, the momentum associated with some of the reform initiatives begun under SACE may have flagged for reasons discussed earlier in this report. This momentum might be sustained if CSOs embrace the Open Society Partnership as an instrument for reform. In particular, what has been the influence on viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Mindset series, Twitter, live streaming, and films? Mindset 2.0: The Mindset 2.0 series started in 2016 with exchanges at various events that brought stakeholders together to consider how to tackle priority policy issues, including how to achieve reform objectives. Another purpose was ascertaining what each stakeholder was doing and how their activities would contribute to the country’s development. A primary objective was discussing communication innovations and how to bring other institutional partners into the fold. The Mindset 2.0 events linked social media and traditional media to expand their reach to other CSOs. Home Vida, through its participation in the Mindset 2.0 series, managed to bring the importance of expanding video platforms to the attention of others into the governance realm. The intended attitudinal change objective was partially achieved given that participating CSOs unambiguously indicated having embraced novel and generally successful advocacy techniques. Mindset, along with other SACE-inspired interventions, were jointly responsible for this achievement. While these sessions were underway, many CSOs shared information live on Twitter, while learning new advocacy techniques. Traditional Media: While engaged in this process, SACE collaborated with various media outlets to promote behavioral changes by incorporating in films priority-reform issues pertinent to society. A strong print, radio, and TV post-SACE presence continues at the national and local levels. Radio programs for audience participation, for example, serve as effective vehicles to advance open budget advocacy initiatives, as that which Social Action in Port Harcourt, Rivers State, is doing. YAF, ANEEJ, and NNEW utilize media partnerships with various radio stations to advance their agendas. CCIDESOR currently engages the traditional media to encourage it to provide a platform to engage the public in favor of policy reforms. In this and other clusters, media outlets are included in the membership. In addition, current partners embrace print media such as The Authority, Punch, Vanguard, Sunrise, Guardian, News Point, Heartland Radio, and online platforms such as Nigerian Voice, The Educator, and The Trumpeter, among others. Similarly, the involvement of print media journalists in some clusters assures SACE partners of advocacy support as they engage policy-makers on key democratic governance issues on a pro bono basis. These journalists, by regularly publishing newspaper feature articles, help drive home pressing advocacy issues. After the conclusion of the project, YAF sponsored an integrity awards competition to positively reinforce good practices in the Nigerian Public Service and is also conducting media capacity-building sessions to improve 32 | SACE FINAL EVALUATION REPORT the substance and delivery of advocacy messages to more effectively relay the intent of reforms for which it is proposing to win public support. Social Media: An unanticipated development reported In the Niger Delta region, as well as by respondents in other parts of the country, is that for a variety of reasons discussed elsewhere in this report, many CSOs are no longer relying on STARNET for many of their coordinating functions; they have substituted this SACE tool with WhatsApp. More concerning and a challenge to the sustainability of one of SACE’s achievements is that some of the hashtags corresponding to the various advocacy platforms launched under its sponsorship are relatively inactive following the activity’s close-out. Even though most of the clusters retain a presence in social media, there is a disconnect between the continuing participation in SACE advocacy initiatives. The most successful hashtag was #mindset2.0, but it was last tweeted on September 12, 2018, when the last Mindset 2.0 SACE event was held. The social media momentum may be lost since the SACE close-out, as most have been unable to muster staying power other than in WhatsApp, on which all anchor and cluster members remain active. In summary, the media sustainability record of SACE appears to be mixed. While across all sites where the activity was active, a fruitful CSO-conventional media relationship seems to have flourished and continues to be committed, the outcome with social media seems to be patchy. Social media has attained the distinction as a primary means for communication and coordination, even making up for functions previously played by SACE through its sharing and learning events. Some informants report that reliance on social media has increased as an alternative to some of the alliance-building services previously secured from STARNET. On the other hand, some social media applications have faltered, while others (such as ConsTrack) are poised to succeed far beyond initial expectations. The ET was unable to ascertain, however, if the SACE legacy on Nollywood and other such commercial mass media will be lasting in the absence of well-organized and orchestrated advocacy promotional efforts. CONCLUSIONS SACE accomplishments pertaining to policy reforms were in some cases profound and likely to have a transcendental impact on the beneficiaries they were intended to assist, among them: PWD, poor women farmers, female informal sector stall sellers, and youth. This largely followed from SACE intentional design that gave priority to advocacy initiatives intended to benefit vulnerable populations. While the mandate of some clusters zeroed in on such populations, such as JONAPWD and DRAC for PWD, YAF for youth, and LAPDO, WARDC, and CENSOJ for women, in other clusters this mandate was more indirect. Evaluating the long-term effectiveness of SACE’s reform agenda is a difficult endeavor. By all outward appearances, there have been some consequential achievements that may well prove to be irreversible, as CSO clusters contribute to chip away at political and bureaucratic intransigence. Some gains in public financial management, and in the adoption of fairer educational, health, and agricultural policies might prove (minimally in some states and levels of government), irreversible and likely to be strengthened. In other cases, the situation is not as clear-cut, as political promises do not necessarily translate into concrete results. They may be offered as mere rhetorical devises professed by policy-makers to appease civil society demands. In SACE’s experience, there has already been some policy backsliding, even though a more energized civil society keeps demanding that policy promises be kept, and reforms implemented. 33 | SACE FINAL EVALUATION REPORT RECOMMENDATIONS On the basis of the ET’s findings, including suggestions by key informants, it is recommended that any follow-on civil society advocacy activity should: ▪ Consider having USAID play a more decisive role in fomenting a Nigeria-wide advocacy policy￾reform initiative based on citizen demands that the transparency and accountability principles embodied in the Open Society Partnership and other international instruments be endorsed and implemented across all levels of government; ▪ Be sensitive to the unique concerns of the PWD community; its advocacy under SACE could have been more effective had they countered with advocacy materials suited to overcome limitations inherent to specific disabilities, such as Braille documentation for the blind or user￾friendly videos or media platforms for the deaf; ▪ Support a public-awareness campaign to educate the public as to why set-aside government programs on behalf of PWD should be endorsed, given that such individuals are among the most vulnerable and marginalized members of society. The campaign should include a component to pressure government at all levels to comply with the five percent requirement that all jobs in the public sector be reserved for PWD; and ▪ Evaluate hosting human development seminars or similar training focusing on app development to take advantage of contemporary technologies, as some ISWF recipients have done. 34 | SACE FINAL EVALUATION REPORT EVALUATION QUESTION 4 FINDINGS EVALUATION QUESTION 4 – How has SACE’s support for Niger Delta implementing partners contributed to their advocacy for economic reforms/growth in that region? Respondents in the field generally confirmed that the nature of the relationship and engagement between CSOs and government have evolved over the years. Before SACE, there was a general antagonism in the relationship, an attitude of “us” versus “them”. As one CSO Executive Director expressed it, CSOs were regarded as stone-throwing organizations who did not believe they should sit with government. According to an anchor in Rivers State, the orientation was based on “name and shame”. In the previous era, virtually every government document, including budget, was regarded as secret and to be kept away from CSOs. With greater capacity-building and advocacy skills through the assistance of SACE, the situation has improved to one of more cordiality, mutual respect, and amicable discussion. As one respondent put it, “the battles have given way to diplomacy” and negotiation based on SACE training. In one KII, a policy-maker in Imo State proudly pointed to stacks of bond copies of the current budget as an illustration of the thawing relationship—this was not previously available to the public. He spoke of how his office benefits from the CSO Monitoring and Evaluation (M&E) exercise and how he intends “to engage them to train my staff here in terms of budgeting and budget-monitoring, which they even do better than us.” In Edo State, the SACE anchor is a leading CSO who has worked with oil-producing development commissions that previously did not have websites, but now have “well and functioning websites running.’’ The CSO anchor worked with cluster members, including one dedicated to PWD, to bring more openness and transparency. In Akwa Ibom State, the anchor CSO leader stated that through their work, they have been recognized as youth sector leaders, who have linked young business start-ups nationally and internationally, as well as trained youths in various skills. The CSO is invited regularly by government to contribute to issues about youth and also invited to annual Children’s Day celebration (May 27); it is also regularly patronized by the media. This is an indication of awareness and engagement with the media, which in turn suggests that SACE has attained some of its objectives. In a similar manner, respondents in Ondo State mentioned how through their engagements, the budget office voted in N2 million for women farmers for the first time. In the next budget, the figure was increased to N60 million—short of the N100 million the CSOs requested, but still substantial. In Rivers State, the challenge was how to get touts and other “unofficial” revenue collectors off the backs of market women; the former forced double-taxation on the women. Through engagement in SACE, the women realized that violence produced little results and they had to become diplomatic. Through their engagement, the government stopped this double taxation and created a gender desk in the Obio-Akpo LGA, to which women can go to report harassment. In addition, they now have greater ease of access to market stalls than previously. According to a policy-maker in one KII, this “reduced the street trading drastically.’’ This view was the general opinion of NNEW and was noted in their policy brief. This assistance to market women illustrates SACE’s commitment to the marginalized in society; the project enabled market women to find their voice when demanding justice. 35 | SACE FINAL EVALUATION REPORT In Rivers State, another CSO anchor focused on open budget in the Niger Delta states of Akwa Ibom, Bayelsa, Delta, Edo, and Rivers with emphasis on openness, accessibility, and citizen participation. The outcome of their engagement in these five states is mixed. In Akwa Ibom, the budget was available for two years and later withdrawn; in Bayelsa, a statement of monthly income is published online but without the expenditure; in Delta State, the budget is open; the same is true of Edo. In interaction with Social Action, it was revealed that Rivers State; “published budgets of the State on the State website for accessibility of the citizenry.” This is an indication that with more mobilization, information previously withheld from the public could become available. In Imo State, the CSO anchor and cluster members have engaged government on issues of abandoned property and were able to make government complete 25 out 225 cases; they have also expanded the elected politicians’ campaign promises for open budgeting to include the electorate’s input, and held a Summit on open government for the entire Southeast region in April 2019. Experiences in the state thus show that the SACE project enabled the facilitation to compel government to return to abandoned property; its impact has continued as illustrated by the Enugu Summit and further engagement on the budget post-close-out. There is a general indication that the relationship between CSOs and government is now more cordial and there are regular engagements. There are also indications that the House of Assembly in the States are more accessible and easier to engage with than the Executive branch. This explains, at least in part, why—as detailed below—there are so many unsigned bills. A general understanding has emerged that SACE technical capacity and skills have contributed to this improved relationship. One anchor cluster program manager described it as a situation in which most CSOs were operating with inadequate experience in some areas, and SACE helped develop new skills. “The SACE project was an eye opener,” he said. “We thought we knew what monitoring and evaluation was but [the] SACE project M&E starts from the onset of the project [and goes] throughout the life of the project.” The new skills enabled CSOs to engage government in ways that yielded more results, especially with the realization that “CSOs alone can’t do it.” There were contributions by PIND to CSO capacity-building in the region as well. As part of its core mandate, PIND builds the capacity of what it calls “capable organizations.” As a PIND functionary explained in an interview with the ET, it partnered with ANEEJ, CCIDESOR, YAF, LITE Africa, ESCFAL, and UPPFA. ESCFAL’s primary role during SACE was to coordinate farmers in engagement with government to develop an agricultural policy. Its coordinator stated that the policy had been formulated but provided no concrete evidence. The ET finds ESCFAL’s commitment somewhat suspect, as during the meeting held at its offices only representatives from two of its cluster members were in attendance, the grounds around the locale were overgrown with weeds, and its appearance indicated inactivity. It appeared that the office had just been opened to receive the ET. PIND also worked with NNEW and SDIC on the issues of access to market stalls and illegal fees extorted from the women traders, and with ESCFAL and UPFFA as Niger Delta BMOs. These relationships have continued after the close-out; indeed, PIND created a “beyond-SACE grant” which was awarded to LITE Africa to continue mentoring four cluster partners. In the interview, it was revealed that further M&E training was planned for the week of September 9; some SACE CSOs are included. In this way, technical capacity continues to be available to some SACE partners. Milestones Achieved Several milestones have been achieved by SACE partners in the Niger Delta. One of these is the Oil Producing Areas Bill. Rendered as “OPADEC,” the organization harnesses revenue from oil 36 | SACE FINAL EVALUATION REPORT to enable the government to meet the needs of oil-producing areas in the states as well as revenue targets. A major activity here includes mobilization for greater popular input into the management of oil revenue, an issue which resonated in all the Niger Delta States. ANEEJ has worked with other CSOs in the region and the oil-commissions bill has been passed in six of the nine states in the region. In addition to the general issues of oil production, the oil-development commission harnesses and channels the 13 percent revenue to oil-producing states from the federal government. Until the oil-producing areas bill was established, the 13 percent revenue was conflated with other funds and got “lost.” The establishment of the oil commissions in the six states has facilitated more transparency in channeling oil revenue into areas of need as identified by beneficiaries in various localities. Already, there are several commissions: Abia State Oil Producing Areas Development Commission (ASOPADEC); Delta State Oil Producing Areas Development Commission (DESOPADEC); Edo State Oil and Gas Producing Areas Development Commission (EDSOPADEC); Imo State Oil Producing Areas Development Commission (ISOPADEC); and Ondo State Oil Producing Areas Development Commission (ONDOPADEC). Attempts to establish similar commissions in Rivers and Bayelsa States did not succeed because, as explained by the Acting Executive Director of ANEEJ, the ubiquity of oil in the two states make such commissions unnecessary. In our engagements with CSO representatives in Akwa Ibom, it was learned that in spite of pressures, the state has not inaugurated an oil-development commission. There is a Niger Delta Development Commission and the Federal Ministry of Niger Delta Affairs of the federal government, which both have strong influence over oil enterprise. As a former Permanent Secretary in EDOSOPADEC expressed it, “the impact was felt because when I came in, the attention shifted from those other projects like building wells to constructing roads in each of the local (oil) government areas. The 13 percent for oil-producing states (from the federal government) is now more judiciously used, and that’s why now you don’t have issue of youth restiveness in the state like what we [had] before this present administration came in.” Through the SACE project’s campaign for transparency and accountability, “The Niger Delta Development Commission for the first time published income and expenditure of the institution,” said a Niger Delta anchor CSO representative. In addition, CSOs efforts have led to more transparency and openness in government agencies. According to a policy-maker in Edo, CSOs are now part of the state oil commission to monitor project before approval for payment. In other states in the region, there are also milestones; in Akwa Ibom, the anchor CSO YAF led a coalition to pass the Akwa Ibom Youth Development Fund bill in February 2019. This bill, which seeks to commit N3 billion annually to youth development, has been passed by the State House of Assembly, but not yet signed into law. Other unsigned bills in the state include compulsory education (which empowers law enforcement to arrest any child hawking or loitering during school hours); bills on protection of PWD; a new bill on rape; and a bill on state health insurance. In Edo, an anchor CSO worked with government to draw up an Edo State agricultural policy which before was non-existent; in mid-August of 2019 another anchor led a coalition of government and non-government actors to draw up a state economic action plan. As the Executive Director of a CSO said, “What we are doing in the Niger Delta is to have a project we call ‘Open Niger Delta’ and it is targeted at bringing CSOs and government to a table to co-create policies and 37 | SACE FINAL EVALUATION REPORT programs that the government will guide and work with. We have been advocating with them to sign open government partnership, we succeeded in Edo.” The 2017 verification of stories confirmed that the SACE project managed to influence the adoption of policy reforms, improve citizens’ engagement, and increase transparency in budgeting. There is therefore evidence that the SACE project has contributed to economic development in the Niger Delta through the creation of oil-development commissions and overall policy which can bring more development to the region. All of these are consistent with SACE’s objective of getting citizens to engage with government. To what extent do achieved results continue to be relevant to the needs of the Niger Delta implementing partners? The foregoing indicates that areas of CSO engagement have pressing economic challenges of poverty, management of oil and other revenues, youth employment, and enactment of necessary policy and legislation. All these are issues which cannot be resolved “once and for all”; they are a moving target for which continuous adjustment is required. The situation suggests that the results achieved are relevant now and in the foreseeable future. Many more youths are completing school and will need employment; others will need the necessary skills to earn a living. The issue of OGP is a continuous one, as are the challenges of taxation, budget allocation, and transparency. It is in this regard that the achievements of SACE can be viewed as some form of relay race (part of the unfinished agenda discussed under Evaluation Question 1). The question, then, is who receives the baton next. In this way, achievements constitute starting points to build the groundwork for future engagements for CSOs and the government. Both sides have learned lessons on what collaboration and dialogue can achieve and should therefore be better placed now and in the future. There are also important issues that have been broached in some states but not in others. As shown above, Ondo State commits some budget to women farmers; this has not happened in other states, yet women remain the backbone of agriculture. In a similar vein, school enrollment is crucial to education and keeping children from hawking during school hours is required. In all the Niger Delta States, there is some “understanding” of this, but no law has passed on compulsory school enrollment. The other issue of transparency is financial management, making the budget public; again, nothing has been signed into law. The relevance of current achievements becomes more questionable when considering that most of the agreements reached have not been signed into law, save only the oil-producing areas bill. Others, such as the Akwa Ibom Youth Development Fund bill, the bill on compulsory school enrollment, the OGP in Edo, Bayelsa, River and other states, have not been signed into law. The government may have only shown political correctness (crucial in an election period and general elections were in held in February and March 2019); without signing the bills, there is little room to challenge actions and inactions. In effect, the draft bills are promissory notes—with attendant fears of retraction. There is, however, hope based on the dominant opinion with discussants in the states where fieldwork was conducted; many of the policy-reform advocacy issues raised while SACE was implemented continue to be pursued. What is more, the anchor/cluster SACE model and skill sets acquired continue to be used. This may be because some anchor CSOs have secured additional sources of funding, while others do so on their own or as part of other endeavors. In Akwa Ibom, for instance, the cluster members proudly assert their own capability based on the SACE toolkit and relate a story of how community members contact them directly on issues relating to 38 | SACE FINAL EVALUATION REPORT government construction projects. Furthermore, they started budget tracking, using “kene budget,” which means “follow the budget.” Through the SACE project, much has been attained in the general area of transparency, openness, and participation. It must, however, be acknowledged that it can be difficult to distinguish attribution from contribution. This is especially true in cases where many CSOs, often from multiple sources of support, engaged in the same issues. It seems, though, that steps taken have made some difference in the Niger Delta and beyond. The challenge therefore becomes one of sustaining achievements and building on them. CONCLUSIONS The conclusion reached regarding Evaluation Question 3 applies to the Niger Delta, where despite some major policy reforms achieved through the USAID/PIND partnership, many others are pending. Efforts to breach the state governments’ secretive and inequitable practices must be continued to yield lasting results. The region’s recent history demonstrates that real reforms, rather than illusions of reform, must be followed by concrete and sustained actions if forward movement is to be maintained. The establishment of OPADECs in all Niger Delta states was a major, consequential step in this direction. In addition, other transparency and accountability initiatives have been taken up, such as the cessation of multiple taxation on market women and making the public budget—in theory a public document anyway—available to the public. There are some obvious gaps, such as bills, including the Akwa Ibom Youth Development Fund, which has overcome the legislative hurdle but to become law must still be signed by the Governor. Far more will need to be done in terms of policy reforms to improve governance and promote economic growth. As this report concludes, such achievements will require sustained efforts for many years if they are to eventually succeed. Continued civil society advocacy and its engagement with the policy-making community should not cease if further policy and economic reforms are to produce desired results. RECOMMENDATIONS On the basis of the ET’s findings, including suggestions by key informants, it is recommended that any follow-on civil society advocacy activity should: ▪ stay the course, while enlarging the reach of its programmatic activities, given that SACE activities in the Delta Region were achieving the desired objectives; ▪ consider maintaining the mutually supportive relationship with PIND; and ▪ expand, to the extent possible, collaborative links with other international donors in the Niger Delta along the same lines as it has done with PIND. 39 | SACE FINAL EVALUATION REPORT ANNEXES ANNEX I EVALUATION STATEMENT OF WORK STRENGTHENING ADVOCACY AND CIVIC ENGAGEMENT (SACE) STATEMENT OF WORK (SOW): SUSTAINABILITY EVALUATION I. ACTIVITY IDENTIFICATION DATA Development Objective Activity Title Strengthened Good Governance Strengthening Advocacy and Civic Engagement (SACE) Award Number Award Dates AID-620-C-14-00001 January 2, 2014 – January 1, 2019 Type of Contract Total Estimated Cost Cost Plus Fixed Fee $19,280,704 Contract Officer’s Representative (COR) Implementing Partner (IP) Olufunke Baruwa Chemonics International Inc. (Chemonics) II. ACTIVITY BACKGROUND In December 2013, USAID awarded a contract to Chemonics to strengthen civil society’s ability to influence the development and implementation of key democratic reforms at the national, state, and local levels. The effective award date for this five-year activity was January 2, 2014 and the award ended successfully on January 1, 2019. The activity supported increased engagement and efficacy of civil society to influence public institutions whose function it is to serve citizens’ interests. This was achieved through strategic partnerships between civil society coalitions and networks (formal or informal) on the one hand, and other critical stakeholders in Nigerian society on the other. The latter included the media, reform-minded government institutions or representatives, private sector actors, influential public figures, grassroots organizations, professional associations, and youth groups. The activity also explicitly engaged marginalized populations, such as women, youth, and persons with disabilities in the process, and it emphasized the importance of leadership and innovation. Program Goals and Objectives 1. Strengthened institutional, organizational, and technical capacity of targeted CSO coalitions and networks to advance targeted democracy and good governance initiatives; 2. Strengthened partnerships between CSO-led coalitions and networks and targeted government institutions and key stakeholders to advocate for and monitor select democratic reforms aimed at strengthening transparency, accountability, and responsiveness of government institutions; 40 | SACE FINAL EVALUATION REPORT 3. Strengthened public awareness, discourse and support for key democratic governance issues such as transparency, accountability and good governance; and, 4. Strengthened capacity of partner business membership organizations (BMOs) and CSOs in the Niger Delta to advocate for inclusive economic reforms and equitable economic growth. Please note: The Chevron-affiliated Foundation for Partnership Initiatives in the Niger Delta (PIND) is USAID’s strategic partner in the Niger Delta. SACE complemented PIND’s other activities in the region under the fourth component. SACE supported the USAID Forward initiative and its goal of increasing the number of procurements directly with local organizations by building the capacity of some of those organizations to be ready to receive direct USAID funding. In its first three years, SACE worked with 16 anchor civil society sub-grantees and their cluster members on agriculture, health, education, public finance, extractive industry transparency, gender, social inclusiveness, conflict resolution, and transparency and accountability in governance. Approach and Implementation The SACE approach was expected to achieve its objectives by: 1. Supporting six national issue clusters/coalitions/networks, each with an anchor CSO that served as a foundation for collective action on a core advocacy issue and facilitates the efforts of participating organizations; 2. Supporting six Niger Delta CSOs and their clusters on issues of economic development and growth in the Niger Delta; and, 3. Directly engaging and supporting initiatives which strengthen and improve the enabling environment for CSOs to operate effectively, with a focus on political economy analysis and identification/response to windows of opportunity, the legislative space, innovation and new technologies as a driver of change. Existing Data The evaluation team will have access to the vital documents relevant to conducting this evaluation. These documents will include the technical proposal (statement of work), performance monitoring plans (PMPs), annual work plans, quarterly and annual reports, the PIND and USAID midterm assessment reports, final reports and any other relevant materials documenting the management, implementation process and results for the SACE activity permitted by the Office of Acquisition and Assistance. I.III. EVALUATION PURPOSE The contractor shall conduct a post-activity sustainability evaluation and provide a report of the findings of that evaluation of the SACE activity implemented by Chemonics. This exercise is intended to ascertain the progress made by SACE partners in sustaining the project’s goals and objectives beyond USAID’s funding. USAID also intends to incorporate the evaluation findings into the design and/or implementation of a new civil society activity. Specifically, this external sustainability evaluation will: 1. Develop key evaluation questions and a methodology that will answer these questions: o To what extent has the implementation of SACE been successful in promoting achievement of the activity’s goals and objectives post close out? 41 | SACE FINAL EVALUATION REPORT o Has the SACE’s anchor/cluster model worked effectively without/beyond USAID support? How have the Innovation Spread the Word (ISWF) grantees fared? o How have SACE partners financial and technical advocacy skills (capacity 1.0 and 2.0) influenced their capacity to engage with government on reforms? What has worked and what has not worked? o Has Systems for Transformation and Results (STAR) now PANDO, Outcome Harvester, Policy Tracker and other tools have been an effective tool in that regard? o Are there additional SACE anchor partners ready to receive direct funding from USAID? o What impact have SACE partners had in influencing government policy development and implementation regarding vulnerable populations such as women, youth and persons with disabilities within the last six months post SACE? What documented policy mileages can SACE claim attribution for? o In particular, what has been the influence on viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Mindset Series, Twitter, live streaming, and films? o How has SACE’s support for Niger Delta implementing partners contributed to their advocacy for economic reforms/growth in that region? 2. Evaluate progress made by partners within six months after the end of the activity and that partners are still using the tools and learnings from the SACE activity to inform program design and implementation and are open to continue doing so beyond the evaluation period; 3. Analyze SACE’s approach to determine if it has strengthened civil society’s technical and financial capacity to advocate for increased participation of vulnerable groups (women, youth and persons with disabilities) in decision-making, support government to institutionalize transparent and accountable governance, and to advocate for economic growth in the Niger Delta region; and, 4. Make recommendations to USAID/Nigeria regarding the design and/or implementation of a new civil society activity. Suggested Evaluation Questions Evaluation Question 1: To what extent has the implementation of SACE been successful in promoting achievement of the activity’s goals and objectives post close out? o Has the SACE’s anchor/cluster model worked effectively without/beyond USAID support? How are they funded or sustained? o How have the Innovation Spread the Word (ISWF) grantees fared? Evaluation Question 2: How have SACE partners financial and technical advocacy skills (capacity 1.0 and 2.0) influenced their capacity to engage with government on reforms? What has worked and what has not worked? o Has Systems for Transformation and Results (STAR) now PANDO, Outcome Harvester, Policy Tracker and other tools have been an effective tool in that regard? o To what extent are the SACE partners using the tools and learning from the SACE activity to inform program design and implementation and are open to continue doing so beyond the evaluation period? o Are there additional SACE anchor partners ready to receive direct funding from USAID? Evaluation Question 3: What impact have SACE partners had in influencing government policy development and implementation regarding vulnerable populations such as women, youth and persons with disabilities within the last six months post SACE? 42 | SACE FINAL EVALUATION REPORT o What documented policy mileages can SACE claim attribution for? o In particular, what has been the influence on viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Mindset Series, Twitter, live streaming, and films? Evaluation Question 4: How has SACE’s support for Niger Delta implementing partners contributed to their advocacy for economic reforms/growth in that region? o To what extent do achieved results continue to be relevant to the needs of Niger Delta implementing partners? Evaluation Question 5: Make recommendations to USAID/Nigeria regarding the design and/or implementation of a new civil society activity. Audience and Intended Users The primary users of the evaluation findings are the USAID/Nigeria Peace and Democratic G Office, other Mission colleagues, and other U. S. Government (USG) officials. Other audiences include non-USG donor organizations, the Government of Nigeria (GON), and CSOs, all of whom can use the findings to improve democracy and governance and human rights programming in Nigeria. The report will also serve as a reference material for program design, implementation, monitoring, evaluation and management of future PDG activities. IV. EVALUATION METHOD AND METHODOLOGY Evaluation Design This evaluation will use mainly a qualitative approach to gain insight into the evaluation questions that will be designed by the evaluation team. The methodology will broadly consist of personal testimony of staff members of the project’s cluster civil society organizations and their beneficiaries, key government officials and other civil society organizations (CSOs) and backed up by a thorough desk study of all relevant documentation. It is expected that 16 cluster anchors will form the evaluation population. SACE partners are in 24 states and a sample of the following states in 5 zones is proposed: Kano, Abuja, Lagos, Nasarawa, Imo, Edo, Rivers, Delta, Bayelsa and Akwa Ibom will provide that coverage. The evaluation work plan will include draft data collection instruments such as key informant interview protocols, structured survey questions, and focus group discussion guides and protocols. If necessary, the MEL Activity will apply for an Internal Review Board (IRB) review of protocols to demonstrate the evaluation is carried out ethically. Data Collection Methods The key approaches that will be used to collect and analyze data for the evaluation, as well as the questions, will be proposed by DevTech and then discussed and possibly modified through an iterative process with USAID/Nigeria. This evaluation will use key informants, focus groups, and outcome harvesting to map how outcomes were achieved to determine what interventions might be replicable. Table 1. Evaluation Questions and Approach Question Approach Data Collection Needed o Has the SACE’s anchor/cluster model The evaluation team will survey all CSOs that received SACE Survey CSOs (activity), document review, conversation 43 | SACE FINAL EVALUATION REPORT worked effectively without/beyond USAID support? Which activity/network continued and how are they funded? o How have the Innovation Spread the Word (ISWF) grantees fared? funding. CSOs will be identified from project documents or mid￾line evaluations. These results will be compared to what CSOs are currently doing. Discuss with key SACE stakeholders if there is a difference and how CSO activities are currently being funded. with CSO heads (KIIs), conversations with SACE and PIND staff (FGDs and KIIs). o Has Systems for Transformation and Results (STAR) now PANDO, Outcome Harvester, Policy Tracker and other tools have been an effective tool in influencing anchor and cluster members capacity to engage with government on reforms? o To what extent are the SACE partners using the tools and learning from the SACE activity to inform program design and implementation and are open to continue doing so beyond the evaluation period? Survey all CSOs that received SACE funding, observe/ask if equipment or people trained by the project still exist/work at the CSOs. Collect CSO representatives’ opinions on the after-effects of the project, and what results are missing/ have not been completed. The effects will be confirmed by selected CSO beneficiaries, for instance, is there a difference between CSO activities now vs. during the project? Survey CSOs (activity), conversation with CSO leaders (KIIs), FDGs with individuals targeted/benefiting from CSOs, KIIs with the external experts (GoN, USAID), conversations with SACE and PIND staff (FDGs and KIIs) o Are there additional SACE anchor partners ready to receive direct funding from USAID? Conduct a formal capacity assessment of anchor CSOs, experiment with how CSOs will act receiving USAID money (virtual experiment), conduct in￾depth case study of the anchor CSOs. Capacity assessment for anchor CSOs, game/test activity with CSO management and key actors, direct observations o What documented policy mileages can SACE claim attribution for? o What has been the influence on viewing, listening, and reading audiences of SACE social and traditional media, such as the 2.0 Mindset Series, Twitter, live streaming, and films? Interview SACE partners, government officials, policy￾makers and review media publications to identify policy outcomes that the SACE initiative contributed to. Survey the 2.0 Mindset Series audience to determine specific actions they have taken as a result of the knowledge or skills acquired from listening to the series. Interview CSO anchors and cluster members, policy-makers, document review. Survey the 2.0 Mind 2 series audience The Nigeria Monitoring, Evaluations and Learning (MEL) Activity will: 44 | SACE FINAL EVALUATION REPORT 1. Perform a systematic desk review of critical documents related to SACE including SACE programmatic documents: cluster and anchor activities, the technical proposal (statement of work), performance monitoring plans (PMPs), annual work plans, quarterly and annual reports, the PIND and USAID midterm evaluation reports, final reports and any other relevant materials documenting the management, implementation process and results for the SACE activity permitted by the Office of Acquisition and Assistance. The evaluation team will also review secondary sources such as the Nigerian government (GON) policies/strategies on civil engagement, CSO policies, and reports, and other donor assessments (if needed), to provide greater context of Nigerian civic engagement and SACE program results. 2. Collect qualitative data by conducting KIIs and FDGs with of SACE staff members, cluster CSO and anchor CSOs and their beneficiaries, key government officials, and USAID officials in order to inform a final report and recommendations on improving civic engagement in Nigeria. 3. Conduct a capacity assessment among anchor CSOs to observe management of direct USAID funding and conduct a case study of the results based on the CSO. 4. Provide specific recommendations to USAID/Nigeria, the Office of Peace Democracy and Governance, key CSO stakeholders and the Government of Nigeria on opportunities to improve future USAID PDG programmatic activities targeting civic engagement. To carry out the work, MEL will form a core team comprised of two local and one (or two) international consultant to work on the evaluation team. The team will consist of: 1. Team Lead: A Team Leader/International Senior Evaluation Specialist with strong methodological background and experience in evaluating peace, democratic governance and human rights activities in developing countries. The team leader will finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to USAID/Nigeria. 2. Evaluation Specialist: An International (or local) Evaluation Specialist with good methodological background in managing, implementing or evaluating peace, democratic governance and human rights activities in developing countries. 3. Local Evaluators (2): Two local evaluators with good knowledge of gender issues and social inclusiveness are required. A fair knowledge of the operations of both social and traditional media in developing countries will be an advantage. It is expected that 16 cluster anchors will form the evaluation population. SACE partners are in 24 states and a sample of the following 9 states in 5 zones is proposed: ● Northwest: Kano ● North Central: FCT, Nasarawa ● Southeast: Imo ● Southwest: Lagos ● South-South: Akwa Ibom, Bayelsa, Rivers, Delta, Edo Workplan: The evaluation will be conducted in six phases over a period of four months. 45 | SACE FINAL EVALUATION REPORT Phase One: Preparation for the evaluation will start immediately after notification from the COR that MEL will implement this task. This phase will include identification and recruitment of the evaluation team, attending contractual matters and initial logistics. Duration of this phase is up to four weeks. Phase Two: Preparatory work will start as soon as the evaluation team is contracted. The MEL DCOP, COP, MEL Specialists and the evaluation team (TL/SES, mid-level evaluation specialist and local evaluators) will have a kick-off phone or video conference with USAID. The TL/SES and the evaluation team members, as they are recruited, will begin the document review immediately after the kick-off and will begin refining the evaluation workplan (EWP). The evaluation team will submit the draft EWP to the mission within three weeks after the kick-off call. The final EWP will be submitted to the Mission within a week after receiving comments on the draft. The EWP will include draft data collection instruments such as key informant interview protocols, structured survey questions, and focus group discussion guides and protocols. If necessary, the MEL Activity will apply for an Internal Review Board (IRB) review of protocols to demonstrate the evaluation is carried out ethically. Duration of this phase is up to four weeks. Phase Three: Field work in Abuja will start immediately after the EWP is approved in this phase. The evaluation team will be further split into sub-teams for the data collection, referred to as Team A and Team B, with Team A visiting five states and Team B visiting four states as well as concluding the interviews and FGDs in Abuja. Duration of this phase is up to three weeks. Phase Four: Analysis and preparation of the draft report starts immediately after the team returns to Abuja. The team analyzes the data, drafts report, and submits report to MEL. MEL transmits draft report to Mission within three weeks after end of field work. The team will also prepare for review a PowerPoint presentation and two-pager. This phase will consist of the review of the report by USAID and return to the team with comments. The evaluation team will address the comments within two weeks of receiving them and submit a final report to USAID. The team will present the PowerPoint and participate in a debrief as well. The expected duration of this phase is three weeks. Phase Five: Two members of the evaluation team (one mid-level evaluation specialist and one local Evaluation Specialist will attend the dissemination meeting with USAID after the final report is approved. Both will develop the final report PPT and share with USAID three days before the dissemination. The expected duration of this phase is two weeks, including travel time for the evaluation team. Data Collection Methods Systematic Desk Review MEL will conduct document reviews to gain a greater understanding of SACE deliverables, results and literature related to civic engagement in Nigeria. The review will be used to inform the design of the data collection tools and to provide information about the target population (CSO clusters, anchors and their beneficiaries) and SACE program results. Document reviews will also provide extensive information on specific challenges and opportunities in engaging with Nigeria’s civil society that will inform future programming. The initial document and literature review will be conducted during the first week of the evaluation by the evaluation team. It will be based on the technical proposal (statement of work), performance monitoring plans (PMPs), annual work plans, quarterly and annual reports, the PIND and SACE midterm evaluation reports, final reports, CSO reports/policies and any other relevant materials documenting the management, implementation process and results for the SACE activity permitted by the Office of Acquisition and Assistance. Key Informant Interviews (KII) 46 | SACE FINAL EVALUATION REPORT Key informant interviews (KII), the principal data collection method for this evaluation, will be used to deepen our understanding of the challenges within the Nigerian CSO environment, the trends and expectations of different USAID, government and civil society stakeholders, as well as beneficiaries from targeted groups (beneficiaries from SACE cluster CSOs). The KIIs will help to understand the viewpoints and contributions of SACE activities, if activities are still running, how the activities are being funded, and their assessment of the challenges, successes and unintended consequences or benefits of program implementation or results, as well as viewpoints on post-program sustainability. There are four major groups of key informants that will be interviewed for this evaluation: (1) USAID PDG staff; (2) SACE activity and affiliated staff (Chemonics, Partnership Initiatives in the Niger Delta (PIND)); (3) Anchor CSOs (Center for Information Technology and Development, Civil Society Legislative Advocacy Center, Women’s Advocates Research and Documentation Center, Disability Rights Advocacy Center, Human Development initiatives, Center for Social Justice and Health Reform Foundation of Nigeria, African Network for Environment and Economic Justice, Edo State Cooperative Farmers Agency Ltd., Leadership Initiative for Transformation and Empowerment, Ugboroke Progressive Fish Farmers Association, Nembe City Development Foundation, Citizens Center for integrated Development on Social Rights, and Youth Alive Foundation) and non-anchor CSOs located in each state the evaluation is carried out; (4) Cluster organization implementing activities in the areas of Health, Education, Transparency and Accountability; and (5) GoN key stakeholders. The evaluation team will be looking for appropriate key informants at the state level, which may include program implementers/partners (SACE and PIND), local government officials, CSOs, CSO beneficiaries, or representatives of USAID. The initial list of key informants will be developed based on information provided by USAID and CSOs. The assessors will also use “snowball” techniques to identify additional relevant key informants during initial interviews. MEL specialists calculate that about 20 KIIs will be conducted in Abuja, and about six maximum KIIs in each of the rest of the nine states with anchor CSOs (two anchors per state, two GON stakeholders, two cluster CSOs) - about 74 KIIs in all. Interviews will be guided by separate open-ended interview schedules for each group of informants. This will include targeted inquiries to collect precise information to answer specific evaluation questions, as well as more open-ended questions aimed at better understanding the how and why of SACE and PIND activities, and the impact of the initiatives. DevTech will obtain all necessary approvals and adjust to changes as requested by the COR and other key stakeholders. Interviews will be designed to be completed within one hour. KIIs will commence as soon as the evaluation methodology is approved by the COR. Key informants will be contacted in advance to schedule interviews and will be provided a copy of the KII schedule and questionnaire, if required. If any of the planned KIIs are not possible in-person due to accessibility, security or availability issues, the team will make a follow-up attempt to interview the key informant in-person or over the phone. To assure accuracy and efficiency, the interviews will be conducted by two-person teams working in parallel. Each team will consist of the Team Lead/SES or a Mid-Level M&E expert, and one Nigerian junior team member. KII information will be analyzed as soon as possible after the interviews are completed. Analysts will identify the most typical responses to the questions and will tabulate them. A separate list of “outlier” responses will be created for further investigation by the evaluation team. Follow-up interviews or emails may be conducted if clarification is needed. All data collection and reporting will abide by the ethical principles of informed consent, respect, sensitivity, do no harm, non-discrimination, and confidentiality. Interviewers will read a brief statement about the purpose of the study and voluntary participation in the interview and will answer respondents’ 47 | SACE FINAL EVALUATION REPORT questions before proceeding. In addition, if needed IRB approval will be obtained to protect interviewee sensitive information. Focus Group Discussions (FGD) The exact number of KIIs and FDGs conducted in each state will depend on the number of anchor and non￾anchor CSOs are located in that state, their beneficiaries, and by the number of objectives the CSOs’ cover. The evaluation team will conduct about 16 FGDs for the anchor CSOs and about 15 FGDs for cluster members that focus on improving Nigerian civil engagement from a variety of clusters: health, education, agriculture, governance transparency/accountability, public financial management, extractive industry transparency, social inclusiveness, conflict resolution and gender among others and their beneficiaries. FGDs will ascertain the most important issues facing Nigerian civic engagement, including viewpoints on the SACE anchor/cluster model, SACE results, SACE activity sustainability, CSO organizational capacity, benefits/consequences of SACE funded CSO activities. These FGDs will provide insight into challenges or successes to civic engagement for SACE staff, CSOs and CSO beneficiary groups or potential sustainability opportunities or challenges. CSO beneficiaries will be divided into focus groups based on cluster and demographic if necessary (women, youth, persons with disabilities). The team will also conduct FGDs with SACE and PIND, to understand differences in how activities functioned before and after the project ended. The team will conduct focus groups at the CSO’s anchor or non-anchor location. Ideally, there will be 10 to 15 participants in each FGD, balanced between male and female participants. If necessary, however, depending on cultural norms, FGDs will be segregated by gender or age. Each FGD will last from one and a half to two hours. The FGDs will be carried out by one core evaluation team member, supported by a MEL specialist and a local evaluator. The core team (with the team lead) will visit six states (Team B), the second team will visit three states (Team A). The results of FGDs will be tabulated and analyzed for most common themes and issues. Surveys Structured electronic surveys will be sent out to each anchor and non-anchor CSO’s office to complete. The structured surveys will provide the evaluation team with insight on what each CSO does for SACE, and the activity challenges, successes and life-after-activity sustainability successes and challenges. Surveys will be sent via email to a designated point person, as specified by SACE activity staff, at the CSO. If the CSO cannot to complete the electronic version, a paper version will be provided to the CSO when one the evaluation teams are at the CSO office, either in Abuja or in one of the nine states. Capacity Assessment To gain a better understanding of the final evaluation question, “do anchor CSOs have the capacity (primarily managerial and financial) to be recipients of direct USAID funding”, the evaluation team will conduct a capacity assessment of anchor CSO staff. The assessment will include the evaluation team asking a series of questions and hands-on activities, such as developing workplans, preparing budgets, creating M&E plans to understand how well CSOs are prepared to receive USAID funding. The evaluation team will then conduct a case study on the capacity outcomes for each anchor CSO. In addition, the assessment will include the evaluation team reviewing CSO policies/protocols to see if the CSO has made positive organizational (financial or managerial) developments. 48 | SACE FINAL EVALUATION REPORT Estimated Level of Effort (LOE) Based on the proposed budget and implementation schedule, DevTech estimates the following LOE for team members: Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 Total Team Leader (TL)/Senior Evaluation Specialist 0 13 18 25 0 56 Mid-Level Evaluation Specialist 0 13 18 23 0 54 Local Evaluation Experts (2) 0 16 34 28 10 88 Home Office PM 5 0 2 2 9 TOTAL 5 42 70 78 12 207 Travel KIIs and FDGs will be conducted in two teams, Team A will travel to four states: Kano, Lagos, Nasarawa and Imo; Team B will travel to five states: Edo, Rivers, Delta, Bayelsa, and AkwaIbom. It is anticipated that in week one, both teams will conduct stakeholder interviews and FGDs in Abuja, in week two, Team A will continue with the field work in Abuja while team B proceeds to the states for data collection. After the second week of data collection in Abuja, team A will then proceed to the other four states. Deliverables The following deliverables will be submitted to USAID/Nigeria. The timeline for submission of deliverables will be finalized and agreed upon during the team planning meeting: 1. Evaluation work plan and timeline 2. Questionnaire/guidelines for conducting key informant interviews/focus group discussions: These documents will be prepared during the team planning meeting and submitted to USAID/Nigeria for review and approval prior to the initiation of key informant interviews and site visits. 3. List of proposed sites and list of respondents: Kano, Abuja, Lagos, Nasarawa, Imo, Edo, Rivers, Delta, Bayelsa and Akwa Ibom are suggested as sites for the evaluation team to consider. However, USAID/Nigeria will support the process of producing a list of sites to be visited and a list of respondents to be interviewed as key informants. Activities of three to five clusters (for example in Health, Education, Transparency and Accountability) in the states and their progress post-SACE will form a critical part of this evaluation. 4. Interview notes and completed surveys: All interview notes and completed survey instruments used for the evaluation will be submitted to USAID/Nigeria. The data used for analysis will also be submitted to USAID/Nigeria. 5. Debriefing(s): The Team Leader will regularly debrief USAID/Nigeria on the progress made during fieldwork. At the end of fieldwork and prior to departing Nigeria, the team will debrief USAID/Nigeria on its findings, conclusions and recommendations. 6. Draft Evaluation Report: The team will provide USAID/Nigeria with a draft report that includes all the components of the final evaluation report prior to their departure from Nigeria. 49 | SACE FINAL EVALUATION REPORT USAID/Nigeria will provide written comments on the draft report to the team within ten working days of receiving the draft report. 7. Final Evaluation Report: The final report will address the comments provided by USAID/Nigeria and other stakeholders on the draft report. The Evaluation Team Leader will revise the draft report and deliver a final revised version to USAID/Nigeria within three calendar weeks of receiving USAID feedback. The final report in both hard and electronic format will be submitted to USAID/Nigeria and approval given before submission to the Development Experience Clearinghouse (DEC). Reporting Guidelines USAID’s evaluation policy requires that all evaluation SOWs include USAID’s Criteria to Ensure the Quality of the Evaluation Report (see USAID Evaluation Policy, Appendix I). The policy also indicates that the report will outline in detail, any additional expectations USAID has regarding a report’s structure, format, and length. The report should be between 25 and 30 pages with an executive summary. Draft reports will be submitted in two hard copies and one electronic copy. The final report will be submitted in five hard copies and one electronic copy. All reports will be in English. The report must: ● Distinguish clearly between findings, conclusions (based strictly on findings) and recommendations (based clearly on the evaluation findings and conclusions); ● Comply with USAID’s Evaluation Policy ● Ensure submission to the DEC after finalization. 50 | SACE FINAL EVALUATION REPORT Dissemination Plan It is expected that USAID/Nigeria will plan a dissemination session with other technical units within the mission, implementing partners, donor community as well as other stakeholders. VI. TEAM COMPOSITION Team Qualifications USAID/Nigeria will engage the services of a team of two local and one (or two) international consultant to work on the evaluation team. The team will require: An International Senior Evaluation Specialist/Team Leader (IQC Labor Category – Program Development, Senior) with strong methodological background and experience in evaluating peace, democratic governance and human rights activities in developing countries. The team leader will finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to USAID/Nigeria. At least eight (8) years of experience in evaluation management is required. Experience in conducting evaluations or assessments of peace and democratic governance as well as human rights programming in developing countries are required. Knowledge of civil society strengthening programming is desired. Ability to produce a high￾quality evaluation report in English is essential. An International (or local) Evaluation Specialist – Mid (IQC Labor Category – Social Scientist) with good methodological background in managing, implementing or evaluating peace, democratic governance and human rights activities in developing countries. At least six (6) years of experience in evaluation is required. Experience preferred in assessing capacity-building efforts in nascent democracies; the role of civil society in strengthening citizen participation and involvement in holding governments to account. African/regional experience is desired. Ability to conduct interviews and discussions and write well in English is essential. Two local evaluators with good knowledge of gender issues and social inclusiveness are required. A fair knowledge of the operations of both social and traditional media in developing countries will be an advantage. A Nigerian with adequate knowledge of the way the national and state assemblies work will act as Sector Specialist, and one of the two local evaluators could be a former Nigerian government official with experience interacting with civil society, given the focus of SACE on that relationship. Team member experience should include graduate level economics, social science, law, and/or public finance training. In￾depth knowledge of issues relating to Nigerian legislative functions and process improvement programs, improving governmental institutional capacity, and fostering civil society oversight of government institutions in developing and/or transitional economies and democracies is required. Some experience in conducting evaluations or assessments is expected. Ability to conduct interviews and discussions and write well in English is essential. Overall, the team will need expertise in USAID practices and expectations in program evaluation; program design and analysis; quantitative and qualitative data collection and analysis; survey design and analysis; program issues, innovations and challenges in building civil society capacity (including financial and personnel management, including value for money); and USAID practices and requirements in program performance measurement. Conflicts of Interest 51 | SACE FINAL EVALUATION REPORT All evaluation team members will provide signed statements attesting to a lack of conflicts of interest or describing an existing conflict of interest relative to the activity being evaluated. USAID/Nigeria will provide the conflict of interest forms. VII. SCHEDULING AND LOGISTICS The proposed evaluation will be funded and implemented by the Mission through The Learning Program/DevTech. DevTech will be responsible for all offshore and in-country administrative and logistical support, including identification and fielding appropriate consultants. DevTech will arrange and schedule meetings, international and local travels, hotel bookings, working/office spaces, computers, printing and photocopying. DevTech will make all logistical arrangements, including the vehicles for travels around Abuja and other cities and should not expect any logistical support from the Mission. DevTech will also make their own arrangements about space for team meetings and equipment support for producing the report. TIMELINE The evaluation team will present a draft timeline with level of effort for the consultants and products to USAID/Nigeria. While it is expected that the final evaluation should be completed by no later than August 2019, in order to inform the design of a follow-on civil society activity, it is expected that the timeline that will be jointly discussed and agreed to between DevTech and USAID/Nigeria will include regular debriefs that will be used to inform the design process prior to the final report submission by September 2019. USAID/Nigeria will provide the following support: ● Supervise the monitoring and evaluation (M&E) unit and ensure that the M&E advisor complies with the job requirements ● Ensure that confidentiality of information is maintained and that files are well kept ● Ensure that copies of relevant documents are available to the evaluation team ● Create an enabling environment for using and learning from activity data and for promoting data￾and information-sharing among activity stakeholders and to interested external audiences ● Oversee and support the above and troubleshoot as needed or requested. Tab 1 – SACE Clusters and Anchor CSOs 1. The Accountability in Education in the Northeast Cluster, anchored by the Centre for Information Technology and Development (CITAD), supported the creation of a State House Assembly Committee in Yobe State to monitor education projects, and secured a commitment from the Bauchi State government to increase the education budget. 2. The National Assembly passed the Petroleum Industry Governance Bill, with pressure from the Civil Society Legislative Advocacy Centre (CISLAC) and the Accountability in Extractives Cluster. The bill, which seeks to provide a governance and institutional framework for Nigeria’s petroleum industry, restructures the Nigerian National Petroleum Corporation’s management and allows new taxation legislation to foster a more attractive investment environment. The Disability Rights Advocacy Center and the Access to Health for Women with Disabilities Cluster made progress toward disability mainstreaming by providing technical support and sustained advocacy pressure for the approval of Nigeria’s National Policy on Sexual and Reproductive Health and Rights of Women and Girls with Disabilities. 3. The Center for Social Justice (CENSOJ) and the Public Finance Management in Maternal, Newborn, and Child Health Cluster helped secure (in collaboration with the Health Reform 52 | SACE FINAL EVALUATION REPORT Foundation of Nigeria, or HERFON) 1 percent of the Consolidated Revenue Funds for a Basic Healthcare Provision Package in the 2018 federal budget. 4. Katsina and Kaduna States passed the Primary Healthcare Under One Roof Bill into law, following sustained advocacy from HERFON and the Quality of Maternal, Newborn, and Child Health Services Cluster, with strong collaboration from CENSOJ and the Public Finance Management in Maternal, Newborn, and Child Health Cluster. 5. The Joint National Association of Persons with Disabilities (JONAPWD) and the Social Inclusion in Basic Education Cluster worked with the Federal Capital Territory’s Universal Basic Education Office to push for adoption of their Accessible and Inclusive Schools Environment Model and to establish an inclusive education policy. They also made significant progress in this advocacy work in Akwa Ibom and Kwara states. 6. The Accountability in Education Cluster, led by Human Development Initiatives (HDI), advocated for budget transparency in education funding and engaged several states to provide universal basic education plans with joint CSO-government oversight of implementation mechanisms. The cluster also supported states to leverage millions of naira in counterpart support from the federal government. 7. Women Advocates Research and Documentation Centre (WARDC) and the Women Lead Agriculture Cluster drafted gender-focused agricultural policies and helped secure the inauguration of the first female executive member of the All Farmers Association of Nigeria in Osun State to amplify women’s voices on agriculture issues. 8. The Youth Advocacy Cluster, with leadership from Youth Alive Foundation (YAF), drafted the Akwa Ibom Youth Development Fund Bill. The bill, which addresses youth unemployment and poverty, has garnered public support and passed two readings and a public hearing in Akwa Ibom’s State Assembly. 9. The Life and Peace Development Organization and the Inclusive Agriculture Governance Cluster successfully advocated for Ondo State to include the Agro Women Initiative in its 2018 budget. They also garnered support from Ondo State’s first lady, Arabinrin Betty Akeredolu, to serve as the advocate champion for a gender-friendly agriculture policy. 10. NECA’s Network of Entrepreneurial Women and the Women in Business Cluster successfully advocated for more transparent engagement regarding local taxation for informal female traders. They also pushed for effective reporting and handling of harassment cases for market women and more equitable allocation of markets stalls to women. 11. The Effective Aquaculture Policy Cluster, led by the Ugboroke Progressive Fish Farmers Association, successfully advocated for developing a clear agriculture policy with a funded aquaculture component in Delta State. 12. The Edo State Cooperative Farmers Agency and the Effective Agriculture Policy Cluster advocated for drafting Edo State’s new agriculture policy. 13. The Africa Network for Environment and Economic Justice (ANEEJ) and Leadership Initiative for Transformation and Empowerment (LITE)-Africa, anchors of the Accountability in Resource Management of Niger Delta Institutions (NDIs) Clusters, engaged the Niger Delta Development Commission to push for greater civic participation and transparency, supporting the launch of the Project Monitoring Platform to track completion of abandoned projects. 14. The Social Development Integrated Centre and the Open Budget Cluster opened the door for citizens to take part in budget processes throughout the Niger Delta, securing a commitment for CSOs to work with the Edo State Budget and Planning Committee, and pressuring Akwa Ibom, Bayelsa, and River States to publicly publish annual budgets. 15. Following sustained advocacy efforts from the Citizens’ Centre for Integrated Development and Social Rights and the Open Budget in Health and Education Cluster, Imo State released the call 53 | SACE FINAL EVALUATION REPORT circular for the 2018 budget, and responded by completing previously abandoned capital projects, which improved basic service delivery. 16. The Nembe City Development Foundation (NCDF) and the Peace Advocacy Cluster developed the Central Peace Working Committee, with participation from the police, Operation Delta Safe team, media, and others, to use an early-warning system to avert conflict and promote peace. 54 | SACE FINAL EVALUATION REPORT ANNEX II EVALUATION METHODS AND LIMITATIONS In an evaluation design of this type, respondent selection bias could always creep in, although the ET was unable to control these biases. Although the ET was provided a list of potential respondents it had no control over who would be available to contact. Difficulties reaching respondents, now that the activity is over, became a concern as some CSO representatives who previously had been readily available, were either occupied with other activities or simply unreachable. Policy-makers, often times, were notoriously difficult to interview and uninterested. A related but inevitable bias, given the evaluation design, is that the vast majority of respondents, except policy-makers and respondents to the Share-out Survey, had had a previous direct association with the SACE Activity; that may or may not have colored their responses in unknowable ways. For example, respondents might have been predisposed to give favorable assessment with the expectation of receiving further USAID funding. A potential bias was also that of assumed source of error due recall bias. The SOW did not require the evaluation team to review government documentation other than what is documented by the project and the CSO partner. The examined reports that were provided by SACE as highlighted in the references of this report. Advocacy materials produced by the CSOs were also reviewed as listed in the references of this report Response rates to the surveys could also influence the solidity with which conclusions reached on the basis of triangulation of results were arrived at. Particularly disappointing was the low response rate of only 21% (24 out of 115 potential respondents) received from those to whom the Successes, Challenges and Sustainability Issues Survey was sent, despite repeated rejoinders. As the ET eventually learned, many of these potential respondents were reticent to comply unless they were told to do so by their respective anchors, something that apparently did not happen. The response rate of 46% for the Share-out Survey was much better, 23 out of 51 potential respondents returning completed questionnaires. 55 | SACE FINAL EVALUATION REPORT ANNEX III DATA COLLECTION INSTRUMENTS #1 KII GUIDE – NATIONAL ANCHOR/CLUSTER CSOs 1. Briefly tell us about your engagement with SACE Activity. How has the SACE project impacted your organization’s current activities after SACE Close-out? 2. Did the SACE project influence your advocacy work? How did it change? 3. What can you tell us about the collaborative nature of the cluster model? 4. What is current status of the partnerships between CSOs in the Cluster model since the SACE Close-out? 5. What can you tell us about the SACE tools such as STAR methodology (PANDO); (ii) Cluster/anchor model and collective impact; (iii) Strategy Matrix/outcome Harvester; (iv) Outcome Harvester; and (v) Policy Tracker? 6. Are you still using the SACE tools and learning from the SACE activity to inform program design and implementation? 7. What would be required to make your organization advance its agenda? Please provide some examples. 8. How is your organization currently funded or sustained after SACE close-out? 9. What has been your engagement with efforts to improve government’s responsiveness to your advocacy and/or economic policy reform in the past few years? 10. How receptive has your organization been on promoting and implementing policies regarding vulnerable populations such as women, youth and persons with disabilities within the last six months post SACE? #2 KII – NIGER DELTA ANCHOR / CLUSTER CSOs 1. Have the SACE interventions had concrete impact on policy-makers/economic activities in the Niger Delta region? If so, could you site a specific example? How has it contributed to the enhancement of social welfare? 2. Briefly tell us about your engagement with SACE Activity. How has the SACE project impacted your organization’s current activities post SACE Close-out? 3. Did the SACE project influence your advocacy work? How did it change? 4. What can you tell us about the collaborative nature of the cluster model? 5. What is the current status of the partnerships between CSOs and BMOs in the Cluster model since the SACE Close-out? 6. What can you tell us about the SACE tools such as STAR methodology (PANDO) (ii) Cluster/anchor model and collective impact (iii) Strategy Matrix/outcome Harvester (iv) Outcome Harvester and (v) Policy Tracker? 7. What would be required to make your organization advance its agenda? Please provide some examples. 8. How receptive has your organization been on promoting and implementing policies regarding vulnerable populations such as women, youth and persons with disabilities within the last six months post SACE? 9. How is your organization currently funded or sustained after SACE close-out? 10. What has been your engagement with efforts to improve government’s responsiveness to your advocacy and/or economic policy reform in the Niger Delta in the past few years? Do the reforms continue to be relevant to the needs of the Niger-Delta implementing partners? 56 | SACE FINAL EVALUATION REPORT #3 KII GUIDE – Policy-makers 1. Were you aware of the SACE project? 2. (Niger-Delta Exclusively) Are you aware of USAID/PIND/SACE-supported CSOs groups’ advocacy/public awareness activities in the state? Have these in your opinion contributed to strengthening public discourse and support for key democratic governance issues? 3. Can you mention the policies that SACE-supported CSOs were able to influence (whether in the formulation, legislative consideration/ passage, or implementation in your State? (Probe to determine which policy or policies the policy maker feels are most important) 4. Have you noticed any changes in CSOs engagement with your Ministry, Department or Agency (MDA) in the past six months? Why? 5. Has there been recent policy priorities initiated by the Civil Society organization? Please explain. 6. How receptive has your organization been on promoting and implementing policies regarding vulnerable populations such as women, youth and persons with disabilities within the last six months post SACE? 7. Did the SACE advocacy work influence your decision making? How? 8. What advocacy approaches do you find most convincing when deciding to endorse or not a given democratic governance reform? What would you recommend to Civil Society to be more effective in engagement with policy-makers? 9. What has been the influence on policy-makers viewing, listening, and reading audiences of SACE social and traditional media messages #4 KII – ISWF (INNOVATION) 1. Has ISWF developed any innovative advocacy approaches? If so, could you describe it for us? Were the innovative advocacy approaches influenced by your engagement with SACE? How? Why? 2. Which innovative advocacy tools are you currently using and why? (Probe for STARNET, STAR Index, Outcome Harvester, Work Plan tracker) 3. How has the Innovation Spread the Word (ISWF) been sustained now that SACE has closed out? 4. Has there been new innovative approaches after the close out of the SACE Activity? 5. How effective / useful are these tools in developing innovative advocacy practices? 6. How are you engaging marginalized populations, such as women, youth, and persons with disabilities in your cluster activities / process? Has there been changes in your engagement since the closeout of the SACE Activity? 7. Has the innovation impacted how you engage with marginalized populations (women, youth, and Persons with Disabilities)? 8. To what extent has your innovation activities successfully influenced policy formulation and implementation? Explain why? 9. What lessons were learned from the SACE Activity? (Probe for what has worked well and how?) What has not worked and why? #5 KII – SACE/USAID/PIND Team 1. Do you think that SACE is successful in achieving its goals and objectives? 2. What were the major factors influencing the achievement or non-achievement of the objectives? 57 | SACE FINAL EVALUATION REPORT 3. What were some of the major challenges in SACE implementation and the deployment of its tools? 4. Within the context of the current operating environment, is the SACE Theory of Change still valid / applicable? Why or why not? 5. Do you think SACE supported reforms will be sustained by the government and CSO partners? 6. How do you assess the continuity of the SACE’s results? 7. What is your opinion about the capacities of CSOs that have received SACE support? Which cluster anchor are most outstanding and why? (probe for readiness to receive USAID funding/support) 8. How well were the issues of social inclusion and gender addressed by SACE? Do you think CSOs can sustain the level of engagement with PWD and Women? FGD GUIDES #1 FGD GUIDE – NATIONAL ANCHOR/CLUSTER CSOs 1. Briefly tell us about your engagement with SACE Activity. How has the SACE project impacted your organization’s current activities after SACE Close-out? 2. Did the SACE project influence your advocacy work? How did it change? 3. What can you tell us about the collaborative nature of the cluster model? 4. What is current status of the partnerships between CSOs in the Cluster model since the SACE Close-out? 5. What can you tell us about the SACE tools such as STAR methodology (PANDO) (ii) Cluster/anchor model and collective impact (iii) Strategy Matrix/outcome Harvester (iv) Outcome Harvester and (v) Policy Tracker? 6. Are you still using the SACE tools and learning from the SACE activity to inform program design and implementation? 7. What would be required to make your organization advance its agenda? Please provide some examples. 8. How is your organization currently funded or sustained after SACE close-out? 9. What has been your engagement with efforts to improve government’s responsiveness to your advocacy and/or economic policy reform in the past few years? 10. How receptive has your organization been on promoting and implementing policies regarding vulnerable populations such as women, youth and persons with disabilities within the last six months post SACE? #2 FGD GUIDE – NIGER DELTA ANCHOR / CLUSTER CSOS MEMBERS 1. Have the SACE interventions had concrete impact on policy-makers/economic activities in the Niger Delta region? If so, could you site a specific example? How has it contributed to the enhancement of social welfare? 2. Briefly tell us about your engagement with SACE Activity. How has the SACE project impacted your organization’s current activities post SACE Close-out? 3. Did the SACE project influence your advocacy work? How did it change? 58 | SACE FINAL EVALUATION REPORT 4. What can you tell us about the collaborative nature of the cluster model? 5. What is current status of the partnerships between CSOs and BMOs in the Cluster model since the SACE Close-out? 6. What can you tell us about the SACE tools such as STAR methodology (PANDO) (ii) Cluster/anchor model and collective impact (iii) Strategy Matrix/outcome Harvester (iv) Outcome Harvester, and (v) Policy Tracker? 7. What would be required to make your organization advance its agenda? Please provide some examples. 8. How receptive has your organization been on promoting and implementing policies regarding vulnerable populations such as women, youth and persons with disabilities within the last six months post SACE? 9. How is your organization currently funded or sustained, after SACE close-out? 10. What has been your engagement with efforts to improve government’s responsiveness to your advocacy and/or economic policy reform in the Niger Delta in the past few years? Do the reforms continue to be relevant to the needs of the Niger-Delta implementing partners? 59 | SACE FINAL EVALUATION REPORT ANNEX IV ABRIDGED ORGANIZATIONAL CAPACITY ASSESSMENT TOOL (OCAT) Proposed capacity assessment questionnaire to evaluate organizational capacity of the Anchor CSOs (NB: This is an abridged version, based on the standard USAID Organizational Capacity Assessment Tool). 60 | SACE FINAL EVALUATION REPORT ANNEX V LIST OF DOCUMENTS REVIEWED 1. Centre for Information Technology and Development, (2017) Outcome Mapping for Accountability in Education in the Northeast. 2. Disability Rights Advocacy Center, (2017) Outcome Mapping for Social inclusion in Health Services. 3. Citizens Centre for Integrated Development and Social Rights, (2017) Outcome Mapping for open budgeting for Government. 4. Africa Network for Environment and Economic Justice, (2017) Outcome Mapping for Accountability of Resource Management in Niger Delta Institutions. 5. Center for Social Justice (2019) Outcome Mapping for open budgeting for Health. 6. Civil Society Legislative Advocacy Center (2017) Outcome Mapping for Accountability in Extractives. 7. Strengthening Advocacy and Civic Engagement, Annual Report Oct 1, 2018-Sept 30, 2018. USAID Nigeria 8. Strengthening Advocacy and Civic Engagement, Quarterly Report July 1 –Sept 30, 2018. USAID Nigeria 9. Strengthening Advocacy and Civic Engagement, Performance Management Plan 2016-2019, USAID Nigeria 10. Strengthening Advocacy and Civic Engagement, Annual report Jan 2- December 31, 2014 11. Strengthening Advocacy and Civic Engagement, combined Q4 and Year 5 Annual report, October 2017- September 2018, ROOT CHANGE 12. System Analysis for Civic Engagement and Advocacy Initiatives, PANDO 13. Strengthening Advocacy and Civic Engagement Quarterly report, April 1- June 30, 2017 14. Strengthening Advocacy and Civic Engagement, Annual and Quarterly report 2015, USAID Nigeria 15. Report of Mid-term Evaluation of the Strengthening Advocacy and Civic Engagement (SACE) project in the Niger Delta, November 2016. USAID/PIND 16. Centre for Information Technology and Development, (2017) Understanding Community Resilience in the CONTEXT of Boko Haram Insurgency in Nigeria. 17. Centre for Information Technology and Development, (2018) Faith and language as Domains of Hate Speech Mobilization in Nigeria. 61 | SACE FINAL EVALUATION REPORT 18. Centre for Information Technology and Development, (2018) Challenging the Fabrics of our national unity, Annual hate speech report. 19. Centre for Information Technology and Development, (2018) Annual Report. 20. NECA’s Network of Entrepreneurial Women, (August 2018) a policy brief on Equitable and affordable allocation of market space for traders in rivers state. 21. NECA’s Network of Entrepreneurial Women, (August 2018) a policy brief on the Harassment of informal sector Business women in their business premises in Rivers state. 22. NECA’s Network of Entrepreneurial Women, (August 2018) a policy brief on promoting clear cut local government Tax policy in the Informal Sector business in Rivers state. 23. Y. Z. Ya’u, (2017) Midnight Monologue. 24. Y, Z, Ya’u, (2019) Politicians Fake Facebook Profiles and their possibility of being un-writing hate speech purveyors. 25. Chris M. A Kwaja Ph.D., (August 2019) Policy brief on Corruption in the Context of Response to Conflicts and Insurgency in Nigeria. 26. Centre for Information Technology and Development, (October 2018) Towards a gender digital inclusion agenda for Nigeria. 27. Centre for Information Technology and Development (December 2018) Hate speech and the campaigns for the 2019 Election. 28. Centre for Information Technology and Development (December 2018) Curbing hate speech, A Policy Brief. 29. Civil Society Legislative Advocacy Center, (2018) a policy brief on Role of the Legislature in Implementing the National Anti-corruption Strategy. 30. Human Development Initiative (2016, 2017) Report of Monitoring the implementation of 2016 and 2017 Lagos SUBEB Actions Plan in selected LGAs. 31. Center for Social Justice (2019) Recommendations on the line items of frivolous, Inappropriate, unclear and wasteful estimates in the 2019 federal appropriation bill. 32. Civil Society Legislative Advocacy Center, (2018–2023) Advocating for an Accessible Responsive and Accountable Government to Citizens. 33. Civil Society Legislative Advocacy Center, (2019) An Assessment of Nigeria’s beneficial ownership transparency legal framework. 34. Women Advocates Research and Documentation Center, (2019) the smart action demand. 35. Africa Network for Environment and Economic Justice (2019) Report of the Niger Delta Institutions Transparency and Accountability Index. 62 | SACE FINAL EVALUATION REPORT 36. Africa Network for Environment and Economic Justice (December 2017) A report of Independent CSOs Monitoring of NDIs projects in 3 Senatorial Districts in each of the 9 Niger Delta states. 37. Africa Network for Environment and Economic Justice (2018) Institutionalizing Transparency and Accountability at Sub-national level. 38. Africa Network for Environment and Economic Justice (September 2018). Role of Citizens in the Implementation of Conditional Cash Transfer Programme. 39. Africa Network for Environment and Economic Justice (October 2018) Assessment of Behavioral Change as an option for tackling Corruption in Nigeria. 40. Citizens Centre for Integrated Development and Social Rights, (2015) Deeping Democracy through Realistic Campaign Promises. 41. Citizens Centre for Integrated Development and Social Rights, (2015) Deeping Democracy through Profiling of Elected Representatives. 42. Citizens Centre for Integrated Development and Social Rights, (2018) Abandoned Projects, Factors of delayed development in southeast Nigeria. 43. Felix. Obaanubi. (August 9 –October 31, 2017) Conducting Verification of stories and beneficiary feedback for the SACE project. 44. Africa Network for Environment and Economic Justice (June 2017). Policy brief of UNCAC implementation and the 2nd review cycle in Nigeria. 45. Citizens Centre for Integrated Development and Social Rights, (2018) What Delays Democracy Dividends in Nigeria. 63 | SACE FINAL EVALUATION REPORT ANNEX VI DISCLOSURE OF ANY CONFLICTS OF INTEREST 64 | SACE FINAL EVALUATION REPORT 65 | SACE FINAL EVALUATION REPORT 66 | SACE FINAL EVALUATION REPORT Name Ogoh Title Alubo Professor Organization University Evaluation Position? of Jos. Team Leader × Team Evaluation Award Number (contract member or USAID otherProject(s) instrumentEvaluated ) (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following No facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated orthe implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvementin the project design or previousiterations ofthe project. 4. Current or previous work experience or seeking employment with theUSAIDoperating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives ofthe particular projects and organizations being I certify evaluated (1) that tha could I havebiascompleted the evaluation. this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. 67 | SACE FINAL EVALUATION REPORT ANNEX VII DATA SUMMARIES 5.1(a): Organizational Capacity Assessment Summary Results (2019) Compared to WEWE (2017) WEWE GRADING (2017) CSO EVALUATION TEAM GRADING (2019) CSO Very Strong None N/A N/A Strong ANEEJ, CENSOJ, HDI, LITE AFRICA, WARDC, YAF Robust ANEEJ, CENSOJ, CISLAC, HDI, LAPDO, WARDC, YAF Moderate CISLAC, CITAD, HERFON, JONAPWD, LAPDO, NCDF, SDIC Moderate CITAD, HERFON, NNEW, SDIC, JONAPWD Weak NNEW, UPFFA, ESCFAL Basic CCIDESOR, ESCFAL • CCIDESOR and Disability Rights Advocacy Center (DRAC) were not evaluated by WEWE • LITE Africa, Ugboroke Progressive Fish Farmers Association (UPFFA), Nembe City Development Foundation (NCDF) were not assessed because their sites were not visited. • DRAC information arrived too late and data not validated • The Team used three categories, while WEWE used four. 68 | SACE FINAL EVALUATION REPORT 5.1(b): Organization Capacity Assessment Tool Summary Results (2019) Compared to WEWE (2017) Leadership and Governance Administratio n Human Resources Financial Management Organization Management Program Management M&E and Quality Assurance FINAL SCORE CCIDESO R 1 1 1 1 1 1 1 1.0 CITAD 3 2 2 2 2 2 2 2.1 WARDC 3 3 3 2 3 2 3 2.5 SDIC 3 3 2 2 2 2 2 2.3 ESCFAL 2 2 2 2 2 2 2 1.9 NNEW 3 2 2 2 2 2 1 2.2 YAF 3 3 2 3 2 2 3 2.6 CENSOJ 3 3 2 3 3 3 2 2.6 HERFON 3 3 2 2 2 2 1 2.2 CISLAC 3 3 2 3 3 3 3 2.7 HDI 3 3 2 3 3 3 3 2.6 LAPDO 3 3 3 3 3 3 3 3.0 ANEEJ 3 3 2 3 3 2 3 2.6 JONAPWD 2 3 2 2 2 3 2 2.3 69 | SACE FINAL EVALUATION REPORT 5.2: CSO Successes, Challenges and Sustainability Issues Survey Results 70 | SACE FINAL EVALUATION REPORT 71 | SACE FINAL EVALUATION REPORT 72 | SACE FINAL EVALUATION REPORT 5.3: Share-out Survey Results 73 | SACE FINAL EVALUATION REPORT 74 | SACE FINAL EVALUATION REPORT ANNEX VIII INNOVATION SPREAD THE WORD UPDATES Organization Name: Order Paper Innovation Name: ConsTrack About the Innovation: ConsTrack is technology application that works with tracking constituency projects. This enables citizens to track their legislator’s constituency project and gauge what they were to accomplish in the process. The app can be downloaded immediately on Google Play Store and currently (after the SACE project) on iOS. Successes: ▪ The development of the ConsTrack application and the relationship-building process of various stakeholders such as the media, members of parliament, and other agencies. ▪ The ConsTrack tool has enabled Order Paper to better advocate for the tracking of constituency projects. ▪ There is potential to scale the application to the 36 states in Nigeria. ▪ It has been able to hold legislators accountable for constituency projects and meet with their constituents to advocate for what is actually needed. ▪ The technology underlying the app has been upgraded to reach all parts of the country after the SACE project. Sustainability: Order Paper’s sustainability plan is to identify more donors to further expand the application to other states and constituencies. Based on the Team’s conversation with Ofordile Tony-Okeke, Head of Operations, Order Paper, some National Assembly delegates and officials at the Ministry of Budget and National Planning have become receptive to the application. The Ministry’s Director of Monitoring and Evaluation has discussed with Order Paper the possibility of transforming ConsTrack into a national tracking tool, thus extending its reach beyond the three original states were the technology has been deployed. As a result, Order Paper was requested to submit to the Ministry a Concept Note to include ConsTrack as a line item in a budget support request to be submitted to a donor. The national chapter of Nigeria Union of Journalists (NUJ), an inclusive union of journalists and media professionals, on the 8th of July indicated willingness to collaborate with us on Constituency Project Monitoring using ConsTrack. There is an MOU in place which both parties should sign in late October of 2019. The implication of this is that they will have the reach to properly track projects around the nooks and crannies of Nigeria using NUJ-accredited community journalists who are knowledgeable about their locality. These tracked projects would also be given the publicity they deserve. Secondly, the ConsTrack app is to be integrated into the Open Budget - Improved Service Delivery Commitment of the Fiscal Transparency Working Group of the 2nd National Action Plan (NAP) of the OGP in Nigeria. The app, among others, would be used to monitor citizens’ satisfaction with the services and responsiveness of the ministries, departments, and agencies. 75 | SACE FINAL EVALUATION REPORT Servicom, a federal government initiative with the mission to improve citizen satisfaction by promoting service excellence in public services, has indicated their desire to collaborate with ConsTrack on project delivery. This was stated in a response to a letter seeking partnership. The ConsTrack app has been demonstrated to the Ministry of Budget and National Planning and also the Budget Office of the Federation. Both organizations have indicated willingness to adopt the ConsTrack app as a tool for project monitoring and getting feedback from Nigerians on government projects. However, they have asked Order Paper to speak to donors willing to finance the logistics that would be involved. Order Paper has also engaged with lot of CSOs on how they can use the ConsTrack app to monitor and track projects. This engagement is yielding benefits as a CSO, Social Action, has indicated interest in using ConsTrack to track constituency projects in Akwa-Ibom State. Organization Name: African Network for Environment and Economic Justice (ANEEJ) About the Innovation: Propati Tracka is a website (www.propatitracka.com) that tracks the properties of politically-exposed persons and senior civil servants. This platform has relationships with agencies such as Economic and Financial Crimes Commission (EFCC) and Independent Corrupt Practices and Other Related Offences Commission (ICPC) to ensure the information they release is verifiable. The website has become a reference point for media engagements with the public. Individuals and organizations refer to Propati Tracka when identifying properties of politically exposed persons and senior civil servants. Sustainability: After the SACE project ended, ANEEJ renewed the website subscription. It is still running and keeping track of more than 90 confiscated properties. Organization Name: Community Life Project (CLP) About the Innovation: The Oshodi Business Connect is a technology app that enables artisans to remit their taxes digitally to enable transparency and reduce duplication. In addition, it enables the Oshodi local government to collect data on additional potential taxpayers to raise revenues. The Oshodi Business connect can be found on Google Play Store for immediate download. Sustainability: CLP has a Memorandum of Agreement with the LGA management. In this agreement, CLP will be set up in the LGA office to sensitize the artisans as they register into the Oshodi Business Connect. There are challenges associated with the signing of this document, since the tax collection agents asked CLP to pay for the collection of data from local artisans in the market. However, the LGA office will manage the billing and tax collection process. This agreement will ensure sustainability with a new administration. In addition, there is legislation related to prevent cash transactions in tax collection. Organization Name: Spaces for Change (S4C) About the Innovation: Digital Storytelling is an advocacy platform based on the Story Works model for young people that, through the filming and presentation of youth-produced videos, S4C sponsors community workshops can shed light on stories of abuse in Lago’s slums. Their goal is to influence public perceptions, promote dialogue among citizens and policy-makers, and bring about policy reforms. Often, the videos reach a wider audience as they are broadcast by the regular media. Among other achievements, this innovation led to the review of the Lagos State Eviction Guidelines. A capacity-building component is a major factor behind the success of this initiative. 76 | SACE FINAL EVALUATION REPORT Sustainability: S4C’s sustainability plan is to continue identifying opportunities for donor funding. The SACE-sponsored initiative helped empower youth to tell stories about their own communities and is likely to be sustained and broadened under other projects. One of the mandates of S4C is good urban governance; the results of this innovation are encouraging. Aside from current efforts to further revise the Lagos State Eviction Guidelines, S4C is in conversations with various stakeholders to fairly and equitably amend the Land Use Act. A major problem with this Act is that when individuals lack a Certificate of Occupancy document, they face quick demolitions and forced evictions. The stories collected in this project are helping change public perceptions of slum residents, which in turn are contributing to a broader dialogue about urban governance in the country’s largest city. The model could eventually be replicated in other urban areas to address the inequities of the poor in Nigeria. Organization Name: Stakeholder Democracy Network (SDN) About the Innovation: The Electricity Distribution Tracker mobile is a technology app developed to promote transparency between electricity customers and distributors. Many issues affect access electricity access in Port Harcourt and Bayelsa State; this platform seeks to address them from the varied perspectives of customers, distributors, and regulators to ensure adequate power in the region. An unresolved challenge is that electricity regulators have yet to come on board to engage other stakeholders in this application, a key for the innovation to achieve its potential in giving consumers a voice that could be heard. Sustainability: Although the Nigerian Electricity Distribution Group initially agreed to be part of this venture, it reneged on its commitment. This has not deterred SDN from continued advocacy to make the app a reality. SDN is working with Niger Delta CSOs and other stakeholders, such as BANGOF (Bayelsa Non-Governmental Forum), CPC (Consumer Protection Council), and the PHEDC (Port Harcourt Electricity Distribution Companies) to bring the concept to fruition. Some technological issues also remain to be overcome, in part because the SACE-provided budget was insufficient to address connectivity challenges in the Delta’s underdeveloped environment. SDN continues pushing forward with the expectation that a more modest app today could gradually be upgraded to eventually attain its initial objectives. The current, more modest initiative is being embedded in existing platforms managed by the Rivers State Civil Society Organization and BANGOF. Organization Name: Youth Alive Foundation (YAF) About the Innovation: “Your Excellency Mobile Gaming App” is an interactive gaming platform aimed to educate youth on civic engagement. The app is a rendition of “Temple Run,” a game developed abroad and customized to the Nigerian landscape and culture. It is intended to relate to and educate Nigerian Youth about the governance process. The app can be downloaded from Google Play Store. Sustainability: YAF has a sustainability plan for the “Your Excellency” gaming app. The team talked about expanding the options of the game and having “in-app purchases” to continue updating the game. They would also like to expand the scope of the game to include more content for civic engagement. In addition, Microsoft Philanthropy will be providing funding based on the number of downloads for the game. They are also registering “Your Excellency” as a social enterprise, to ensure they are able to make a profit while sustaining the gaming platform. Organization Name: The Nigerian Women’s Trust Fund (NWTF) About the Innovation: Ideathon NG is a competition geared towards engaging young women in civic engagement. The target demographic is the National Youth Service Corps (NYSC) members that provide 77 | SACE FINAL EVALUATION REPORT concepts on innovative ways to engage women in civic engagement and political processes. Winners are selected to implement innovations in their states, with the selected locations being Lagos and Osun States. This innovation enables students in secondary schools to understand democracy and civic engagement through experiential learning. Sustainability: The Nigerian Women’s Trust Fund remains committed to support its mission of engaging women in the political process. They have utilized various models, including Ideathon, to ensure that happens. The idea behind this innovation is to use another medium to meet their core mission to “Engage women in the political process”. To sustain this effort, the NWTF will continue to work on various other related projects to advance their vision and mission to engage women. In pursuit of this objective, the NWTF also customized the experiential learning module they had developed, along the lines of the Ideathon, in partnership with the Stakeholder Democracy Network in the Niger Delta sponsored by the Open Society Initiative, West Africa (OSIWA). 78 | SACE FINAL EVALUATION REPORT ANNEX IX POLICY MILESTONES POST-SACE ACTIVITY Policy Milestones ANEEJ Located in Benin in the Niger Delta, ANEEJ has two primary advocacy focuses: anti-corruption and environmental preservation and recovery. Since SACE ended in 2018, ANEEJ has continued battling corruption, been energized by a 1.5 million pounds grant from DFID. Through the MANTRA accountability project, ANEEJ will engage in a campaign for the recovery of looted funds, primarily be seeking the repatriation of monies stolen by former public officials. ANEEJ’s Open Niger Delta initiative, also with an emphasis on transparency and accountability in the region, is being supported by a 650,000 Euros Bread for the World grant which will run through April 2020. ANJEE’s reputation as a pre-eminent good governance CSO advocate transcends Nigeria. The organization has been invited to make policy presentations at Chatham House, a British think tank, The World Bank and at an international conference in Fiji. With increasing international recognition and a growing basket of grant support, ANEEJ should continue adding to its past policy-reform successes. Engaging in policy-reform advocacy in Kwara State has not been easy as the government was not well predisposed to interact with civil society. Nevertheless, CCEPE joined the WARDC-led cluster to work in the state (as part of a four-state effort) to advocate on behalf of women farmers. In this environment, CCEPE set out to reach to individual women farmers and began providing them training to encourage them to start advocating collectively. During this process, two local women farmer champions were identified who were invited to attend the various SACE trainings. These women helped mobilize others and through their collective efforts, managed to extract some concessions from the state government regarding agricultural policies specifically geared to the needs of women farmers. The original organizing impetus provided by SACE continues as the minor gains achieved early on are to be expanded as a new State government elected in 2019 is more receptive to civil society reform demands and willing to contemplate more comprehensive reforms. CCIDESOR CCIDESOR, located in Owerri, Imo State, in the Southeast geopolitical region, but now classified as part of the Niger Delta, has undertaken, as part of its portfolio, two major projects: the abandoned projects initiative and the open budget campaign. Since the SACE close-out, CCIDESOR participated in the monitoring of the February/March 2019 general elections and has continued advocating for its two main long-standing projects. In April 2019, with funding from a 650,000 DFID Euro grant, CCIDESOR organized the Open Government Enugu Summit for organizations representing the entire South-east region. Funding from this grant, approved through April 2020, is also being used to monitor the compliance of politicians with promises made during their electoral campaigns. CENSOJ CENSOJ remains very active as it continues to pursue reforms in maternal/child health public financing. Ongoing efforts focus on implementation of the 1% maternal/child health funding requirement through a broad health sector coalition in which the CENSOJ-led cluster is involved. The CENSOJ cluster is also following closely and advocating for passage of the proposed Health Insurance Law. Since SACE came to an end, CENSOJ has established several new clusters, most notably the Government Audit Reform cluster. To promote the more effective allocation and use of government resources not only in health, but across all government sectors. In 2019, new grants to CENSOJ were awarded by the North East Regional 79 | SACE FINAL EVALUATION REPORT Initiative, a Creative Associates-managed USAID activity; the European Union for an Audit Reform project; and by Foster for an International Bench-Marking Extractive Budget study. In addition, CENSOJ has been shortlisted for a UNDP Spot-light grant. New clusters established since the SACE close-out include: The CENSOJ-led Audit Reform Cluster, formed in July 2019 involves CSOs from all of Nigeria’s 36 states (two CSOs per state). This initiative is being conducted with support from the European Union Rule of Law and Accountability project. WARDC is also relying on the cluster model in the Strengthening Citizens Resistance against Prevalence of Corruption (SCRAP-C project), a DIFD initiative. To conduct this effort, WARDC identified women farmers in seven states other than those it had worked in under SACE. In total, six clusters groups are involved, some drawn from women farmers’ associations, women in religious settings, and women leaders in private and public organizations. Some former SACE-cluster CSOs are also participating in this activity. SCRAP-C CSOs rely on the Outcome Harvester SACE tool, while meeting quarterly to harvest outcomes and evaluate successes and lessons learned. In addition, the Team learned that other former SACE anchors, such as JONAPWD, have been considering, but may not have yet formalized, the establishment of new advocacy clusters. JONAPWD is considering doing so in two instances: with the National Organization for the Blind and with the Disabled Women Farmers. CISLAC’s cluster is going strong in the post-SACE environment as it is currently leading, among other efforts, an advocacy campaign dealing with tax justice and governance sponsored by OXFAM. The work involves partners in 16 states, with CISLAC leading from behind as called for under the anchor/cluster model. CISLAC is also managing another new project funded by the Friedrich Ebert Stiftung (FES) and has a pending proposal with the Ford Foundation. As part of its Transparency in Extractives cluster strategy, CISLAC continues advocating in favor of infrastructure projects to help minimize high youth unemployment, while not giving up on the policy objectives sought in the Petroleum Industry Bill, despite the fact that President Buhari refused to endorse it. Upon joining SACE, CITAD and its cluster focused on the findings of an education survey in the Northeast, which revealed multiple deficits. As a result, CITAD embarked on an advocacy campaign that yielded a State of Emergency declaration in the sector, resulting in 18 percent increases in the education budgets for the States of Yobe, Bauchi, and Adamawa, and a corresponding 12 percent increase in Borno State. Building on these achievements, since the SACE close-out the CITAD cluster has been organizing a North East Book Education and Exhibition Conference scheduled for November 2019, as collaborative links between CSOs and government continue to be strengthened. HDI representatives claim that the organizational capabilities acquired under SACE have helped the CSO access other funding sources, including a recent MacArthur Foundation grant to continue working on the same educational reform issues. A current advocacy activity includes working with Action Aid on the Breaking Barriers in Child Education project, in collaboration with the Education Secretary and the local UBEC. Advocacy capability improvements are also leading other CSOs to seek HDI input to help formulate strategies in favor of reforms to provide services directly to schools, including EDU Planner, which has proposed a partnership to HDI. In a recent funding proposal, HDI proposed to employ the anchor/cluster model to combat sexual violence against women students in Kano, Lagos, and FCT. As SACE was closing, HERFON and its cluster members participated in a SACE-convened organization development meeting where the decision was made to establish a broader coalition under the aegis of the 80 | SACE FINAL EVALUATION REPORT Community Health and Research Initiative (CHR). Subsequent engagements with the Kaduna State government led to the appointment of two commissioners drawn from the CSO community; their main responsibility will be serving as civil society champions within government. The engagement with the then former Commissioner for Health (now at the State Ministry of Education) yielded further dividends, as it led to the decision to increase the number of employees within the Primary Health Care System from 3,000 to 5,000 to improve service delivery. HERFON also managed to increase the state budget family￾planning line item as it continues expanding its reach into other thematic areas. The Kaduna State media regularly provides free air time in support of HERFON’s advocacy efforts. Finally, the cluster continues to play an oversight function by connecting Ministry of Health, other ministry departments and agencies, CSOs, and the media through #OpenKadunaHealthSector, thus facilitating the review of how government projects are funded and implemented. Prior to the relationship with SACE, JONAPWD had never had a contract with a donor, only receiving in-kind support which limited its capacity to impact the PWD space. Thanks to SACE support, JONAPWD was able to build its administrative system, which has helped the anchor draw support from donors other than USAID. Since the end of SACE, the cluster partnership has solidified and continues to be very active; currently, the various CSOs are acting in concert—not only advocating for policy reforms, but also for their implementation. The cluster approach was so productive that JONAPWD currently has six active sub-clusters engaged in different advocacy issues, including one being conducted in concert with the National Organization for the Blind. The most pressing revolves about enforcement of the disability-rights legislation signed by President Buhari in 2019 that includes the government-wide mandate than 5 percent of new civil service employment be reserved for PWDs. JONAPWD also lobbies to raise the requirement to as much as 20 percent. Even though legislation on inclusive education was a major breakthrough, JONAPWD is not resting in its laurels, as it sees what has already been achieved as initial components of a far more ambitious multi-sector national reform agenda. A member of the ANEEJ Niger Delta cluster, Komunity Action Initiative (KAI)’s engagements in Ondo State are an excellent example of how SACE’s collective impact approach is bringing about economic improvements in this region. A governance-reform-oriented CSO, KAI has been collaborating with LAPDO to promote greater budget transparency and accountability in Ondo State. It has joined forces with the LAPDO cluster to secure the allocation of the budgetary line item in the state budget on behalf of women farmers. This initiative ultimately arose from ANEEJ’s budget transparency reform advocacy in the Niger Delta generally, and in Ondo State more particularly. KAI also collaborates with the LAPDO￾led cluster in several community development initiatives. KAI currently participates in PIND’s sponsored needs assessments, including those related to how buildings constructed as schools are being used for other purposes; it also functioned as an election overseer in 2019. While advocating as a SACE anchor in association with other stakeholders in Nigeria’s Southwestern region, LAPDO successfully convinced the authorities to include a gender-inclusionary line item for women in agriculture in the Ondo State budget. Building on this achievement, LAPDO has entered into a partnership with the state government to improve the livelihood of these women. One result was that the initial 2018 N2 billion budget allocation, which in 2019 was raised to 60BN following negotiations between the State Ministry of Economic Planning and Budget and LAPDO-cluster representatives. This government investment in extension services and agricultural inputs will help raise living standards for women farmers. In addition, the Ondo State Ministry of Agriculture is engaged in consultations with LAPDO as it seeks input for developing a long-term action plan as part of the Innovation Platforms initiative to enhance rural value chains attuned to women farmers concerns. A less tangible, but perhaps more consequential, milestone that may be at least partly attributed to LAPDO’s involvement with SACE 81 | SACE FINAL EVALUATION REPORT is the self-confidence some women farmers have gained. They no longer wait for CSO intervention to advocate on their own behalf: they go directly to the Ministry of Agriculture, a major attitudinal change. Organizational capability improvements gained through to SACE, according to LAPDO’s leadership, was a major factor behind a DFID award to LAPDO in connection with the 2019 general election. PIND has also become a LAPDO donor. Before joining SACE, NNEW was only working on three policy arenas; today, its advocacy extends to 19 policy-reform objectives. Since the SACE close-out, NNEW decided to participate in online business management courses offered by Facebook. Even though the electoral period distracted NNEW from its core advocacy issues, the cluster has continued pursuing its flagship reform agenda, leading to the passage in 2019 of the “No Street Trading” bill, pending approval by the Governor. The law’s purpose is reallocating street traders into market sites sponsored by localities, while increasing the number of market stalls allocated to women and PWD. Since the end of SACE in late 2018, the WARDC cluster has expanded to other states as it tackles the challenge of curbing corruption in the distribution of subsidized agricultural inputs. This builds on its previous achievement when, while associated with SACE, it successfully led an advocacy campaign that resulted in the approval of a budget line item in 2016 in favor of smallholder women farmers in Kwara State. Some observers find it remarkable that women farmers, belonging to the WARDC cluster, have acquired the ability to advocate on their own behalf and to do so effectively. This is one of the reasons the cluster continues to thrive even after the SACE close-out. Another benefit arising from the former SACE relationship is the cascading training role disseminated by WARDC to some of its key cluster members in other states; these are acting as sub-cluster leaders as the WARDC cluster expands the geographical reach of its advocacy in favor of a broader woman-farmer constituency. WARDC’s recent focus has been on two main topics: corruption in the agricultural sector (as it is generally known that government officials sell fertilizers subsidized for small farmers at commercial prices) and preventing violence against women. WARDC was also involved in election-related activities in early 2019. An indication WARDC’s increased recognition was the recent designation of one of its former small women farmers’ association clusters to represent Nigerian women in international fora, including a United Nations Women’s Leadership Network meeting in Addis Ababa. YAF continues to be very active, expanding its scope of action by relying on the anchor/cluster model. Since the SACE close-out, YAF has encouraged local cluster members to assume ownership of the model in Akwa Ibom State, following approval by the State Assembly of the Youth Development Funding Bill. YAF is seeking to replicate the Akwa Ibom success in four other states, this time with DFID support. These efforts will include components to benefit young disabled people. The model has also served to establish a new anti-corruption cluster in Abuja, mostly with CSOs not part of the initial SACE-promoted cluster. Meetings among these cluster members are held regularly on a monthly basis. Innovation is another SACE legacy, as the digital-gaming initiative on governance launched by YAF with ISWF funding began to be used to sensitize youth about governance reforms just as SACE came to a close. Use of the mobile application is gaining momentum as it reaches more young gamers. Other tech-driven apps are under development with funding from the DFID award to YAF. 82 | SACE FINAL EVALUATION REPORT ANNEX X ENGAGEMENT OF MARGINALIZED GROUPS POST-SACE Cluster Anchor Name (National Cluster) Activity Engagement of Marginalized Groups Post￾SACE Activity JONAPWD JONAPWD was established to serve the PWD communities and had been doing so before becoming a SACE anchor. PWD’s reach is inclusive, covering the blind, deaf, albinism, and the physically and mentally disabled. Organizations representing all these communities come together in the JONAPWD cluster, which advocates for inclusive education reforms following the achievement of federal legislative accomplishments regarding the rights of the disabled. Yes WARDC WARDC focuses on the social and economic rights of women farmers, having facilitated the development of programs on their behalf in Borno, Akwa Ibom, Lagos, Kano, Kwara, and Kaduna States, as well as in the FCT. Advocacy efforts have included promotion of policies to give women better access to farmland and reduce corruption in the distribution of subsidized farming inputs. Yes CENSOJ CENSOJ’s Public Financial Management of Maternal/ Child health cluster prioritizes the provision of services to marginalized populations. CENSOJ has achieved major successes securing funding for such programs and is currently providing oversight to ensure legal mandates are implemented as intended. Yes CITAD CITAD’s programmatic focus is service delivery in education partly, but not exclusively, to improve service delivery and improve school attendance rates. Yes HERFON HERFON’s policy priority is the betterment of maternal and child health services through the provision of quality health care facilities and increased health care personnel. HERFON and CENSOJ have a close working collaborative relationship. Yes DRAC In some respects, DRAC’s mandate is similar to JONAPWD, except the former focuses exclusively on the needs of disabled women. Yes CISLAC CISLAC’s policy-reform emphasis of promoting Transparency in Extractives includes a related component to reduce youth unemployment in the Niger Delta, a persistent social problem affecting this marginalized age group. Yes HDI HDI advocacy efforts are largely intended to improve educational opportunities by impacting the policy Yes 83 | SACE FINAL EVALUATION REPORT decisions of UBEC and SUBECs. Other programs include the provision of educational scholarships for needy children and advocating for inclusionary school polices for children with disabilities. Cluster Anchor Name (Niger Delta) Activity Engagement of Marginalized Groups Post￾SACE Activity LAPDO LAPDO works with small-scale women farmers to ensure they have access to resources necessary to their work and to improve their socioeconomic living standards. Yes ANEEJ Since ANEEJ’s budget transparency initiatives are cross￾sectoral, they are of potential benefit to all marginalized populations. Yes, by implication Social Action Social Action transparency promotion activities, like those of ANEEJ in the Niger Delta, are likely to benefit every vulnerable population, although that is not an explicit aim of the organization and its cluster. Yes, directly and by implication ESCFAL ESCFAL’s focus on economic growth speaks to the betterment of marginalized groups, particularly in the Niger Delta, although its mandate makes no reference to marginalized population. Yes, by implication YAF YAF’s vision and mission centers on youth welfare, a SACE-defined vulnerable population. Yes UPFFA Since most farmers in Nigeria are women, the agricultural development objective of this organization ultimately is of benefit to a key vulnerable population. No NNEW Since their target demographic is market women, they were able to create a safe environment for them to conduct business. Yes Innovation Spread the Word Fund Organizations NWTF The objective of the Nigerian Women Trust Fund is to further engage women and girls in political participation. Yes Order Paper Order Paper’s emphasis on community grassroots constituency issues have a potentially direct bearing on the welfare of women and children, as well as that of other marginalized populations. Yes Spaces for Change The work of this CSO defending the rights of slum dwellers in Lagos and exposing the consequences of evictions speaks to the abuses vulnerable populations are exposed to. Yes Community Life Project This ISWF initiative was intended to assist low-income female members of hairdresser associations. Yes Stakeholder Democracy Network The Team was unable to reach anyone from this organization. N/A