JULY 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by NORC at the University of Chicago. DRG Learning, Evaluation, and Research Activity: Performance Evaluation of the Consortium for Elections and Political Party Strengthening (CEPPS III LWA) Performance Evaluation of the Consortium for Elections and Political Processes Strengthening III Leader Award (CEPPS III LWA) Final Report July 2016 Prepared under Contract No.: GS-10F-0033M/AID-OAA-M-13-00013 Submitted to: USAID/CEPPS/DRG-LER Submitted by: Katharine Mark Clifford Zinnes Russell Owen Alex Rigaux David Sipp Ellen Seats Jeffrey Swedberg Contractor: NORC at the University of Chicago Attention: Renée Hendley Bethesda, MD 20814 Tel: 301-634-9489; E-mail: Hendley-Renee@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii TABLE OF CONTENTS ACRONYMS.................................................................................................................................. v TABLES AND FIGURES ............................................................................................................. vi EXECUTIVE SUMMARY.............................................................................................................. 7 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS................................... 11 1.1 EVALUATION PURPOSE ........................................................................................................................... 11 1.2 EVALUATION QUESTIONS ..................................................................................................................... 11 2.0 PROGRAM BACKGROUND AND OBJECTIVES .......................................................... 12 2.1 PROGRAM CONTEXT............................................................................................................................... 12 2.3 PROGRAM COUNTRIES AND TYPES................................................................................................... 13 3.0 EVALUATION METHODOLOGY................................................................................... 14 3.1 EVALUATION MANAGEMENT............................................................................................................... 14 3.2 WEB SURVEY DESIGN AND IMPLEMENTATION............................................................................. 14 3.3 SUMMARY OF QUANTITATIVE METHODS....................................................................................... 15 3.4 CASE STUDY SELECTION AND IMPLEMENTATION...................................................................... 16 3.5 LIMITATIONS ................................................................................................................................................ 17 4.0 FINDINGS AND CONCLUSIONS .................................................................................. 20 4.2 EVALUATION QUESTION 2: DEGREE OF DEEPENING FROM CEPPS II TO CEPPS III........ 27 4.3 EVALUATION QUESTION 3: THE NATURE OF CEPPS III PARTNERSHIPS............................. 39 4.4 EVALUATION QUESTION 4: FACTORS LEADING TO CEPPS III SUCCESS............................ 49 4.5 EVALUATION QUESTION 5: ASSESSING CSO SUSTAINABILITY AND FINANCIAL MANAGEMENT DUE TO CEPPS III ASSISTANCE.................................................................................... 54 5.0 OVERALL RECOMMENDATIONS.................................................................................. 62 iv APPENDIX: CASE STUDY REPORTS Cambodia Georgia Guatemala Nigeria Tunisia Zambia ANNEXES Annex I. Tasking and Statement of Work Annex II.A. CEPPS Award Map Annex II.B. Country Classification Tables Annex III. Methodology Annex IV. Draft KII protocols Annex V. Data Collection Instruments Annex VI. Matrix of Research Questions and Data Sources Annex VII. Analysis Plan Annex VIII. Regression Tables Annex IX. Supplemental Tables and Analysis for Selected Results v ACRONYMS AOR Agreement Officer Representative CEPPS III Consortium for Elections and Political Processes III LWA Leader with Associates Award COP Chief of Party CSO Civil Society Organization DRG-LER Democracy, Human Rights, and Governance Learning, Evaluation, and Research Activity DRL Bureau for Democracy, Human Rights, and Labor (US Department of State) EQ Evaluation Question ET Evaluation Team IBTCI International Business & Technical Consultants Inc. IFES International Foundation for Electoral Systems IRI International Republican Institute KII Key-Informant Interview M&E Monitoring & Evaluation NDI National Democracy Institute for International Affairs NORC National Opinion Research Center NGO Non-Governmental Organization PMP Performance Management Plan PVT Parallel Vote Tabulation TA Technical Assistance USAID United States Agency for International Development USG United States Government vi TABLES AND FIGURES Table 2.1. Country Programs.......................................................................................................................14 Table 4.1.2. Average Number of CSOs per Country, 2009 and 2014...............................................23 Table 4.1.3. Number of CSOs between 2009 and 2014........................................................................24 Table 4.1.4. Average Life Expectancy of CSOs per Country, 2009 and 2014 ..................................26 Table 4.1.5. Average Number of CSOs with which CEPPS-affiliated and Non-CEPPS-affiliated CSOs Collaborated..................................................................................................................26 Table 4.2.1. Share of CSOs Receiving Organizational Capacity Building, by Implementer............31 Table 4.2.2. Share of CSOs Receiving Organizational Capacity Building, by Region.......................32 Table 4.2.3. Share of CSOs Receiving Technical Capacity Building, by Implementer.....................32 Table 4.2.4. Share of CSOs Receiving Technical Capacity Building, by Region ................................33 Table 4.2.5. AOR Opinions on the Importance of Selection Criteria................................................38 Table 4.2.6. Share of CSOs Reporting a Fair Procurement Process, by Implementer...................38 Table 4.3.1. Share of CSOs Reporting Implementer Decided on Programmatic Activities..........40 Table 4.3.2. CSO Involvement in Award Design and Management.....................................................40 Table 4.3.3. Purpose of Grants....................................................................................................................43 Table 4.3.4. Type of Organizational Capacity Building Most Needed and Received under CEPPS II and CEPPS III, according to CSOs.....................................................................................44 Table 4.3.5. CSO Opinion on Effectiveness of Capacity Building ........................................................45 Table 4.3.6. Building Capacity as an Explicit Part of the Program, Implementer Survey................45 Table 4.3.7. Degree that Program Involved Building CSO Capacity, Implementer Survey ...........46 Table 4.3.8. Organizational Capacity-building Integrated vs. Parallel Track, Implementer Survey..........................................................................................................................................46 Table 4.3.9. Share of CSOs Receiving Organizational Capacity Building through Joint Work, Total and by Region .................................................................................................................47 Table 4.3.10. Share of CSOs Receiving Technical Capacity Building through Joint Work, Total and by Region ............................................................................................................................47 Table 4.4.1. Implementer and CSO Opinion on Factors to CSO Success........................................52 Table 4.5.1. CSO Experience with Sustainability and Financial Management....................................55 Table 4.5.2. Helping CSOs to Obtain Donor Funding as a CEPPS Objective ..................................56 Table 4.5.3. Rate of Obtaining Donor Funding, and Level of Assistance Received .........................58 Table 4.5.4. Relationship between CEPPS III Assistance and CSOs Obtaining Funds.....................58 Table 4.5.5. Implementer and CSO Opinions on the Extent to which Implementers Built CSO Capacity to Apply for Larger Awards..................................................................................60 Table 4.5.6. Implementer Efforts to Increase CSO Sustainability........................................................61 7 EXECUTIVE SUMMARY USAID’s Consortium for Elections and Political Process Strengthening III, implemented by the International Republican Institute (IRI), National Democratic Institute (NDI), and International Foundations for Electoral Strengthening (IFES), is a global program with objectives including addressing the legal framework, building indigenous capacity to administer elections, supporting monitoring, promoting inclusion, ensuring competitive multi-party system, fostering smooth transitions of power. The program was implemented in 54 countries, with a total of 92 associate awards NORC was engaged to undertake a performance evaluation of CEPPS with the purpose of understanding how the Leader with Associates (LWA) program has contributed to the capacity of local organizations to support elections and political transitions on CEPPS countries. In particular, the focus was on Objective 10 of the LWA, which is to build “capable and sustainable local and regional civil society organizations engaged in election assistance and in providing technical assistance to political parties and political processes.” Five evaluation questions were identified, as follows: EQ1. Are there more CSOs active in elections assistance and political processes than five years ago? EQ2. How were programmatic strategies developed under CEPPS II altered or expanded under CEPPS III to address local capacity building goals under Objective 10? EQ3. To what degree did CEPPS involve local partners in the design, management, M&E process, and reporting of programs with which they were affiliated? EQ4. Which factors contribute to the successful elevation of local organizations into sustainable providers of technical assistance related to elections and political processes? EQ5. What efforts on behalf of CEPPS have been made to support local organization opportunities for becoming direct recipients of donor funding? METHODOLOGY The mixed-methods evaluation was designed to include Document review, Key Informant Interviews (KIIs) of implementers at HQ, Case study visits to six program countries, and web￾based surveys of CEPPS implementers, USAID’s Agreement Officer Representatives (AORs), and CSOs across the entire program. Case Studies Case studies were conducted in six countries – Cambodia, Guatemala, Georgia, Nigeria, Tunisia, and Zambia – selected to represent five different regions, include several programs implemented by each of the three implementers, and cover a variety of approaches. The case studies were conducted by three subject matter experts with experience in electoral/political 8 processes as well as evaluation. In each country interviewees included representatives from USAID and the respective implementers of the country programs, as well as a range of different CSOs, some who had participated in CEPPS III programs, and some who had not. Web survey Survey sampling relied on email addresses obtained from USAID and implementers, which was fairly straightforward for the implementers and AORs, except for older programs. Obtaining a robust survey of CSOs was much more challenging. An initial set of addresses was obtained from implementers, although these varied considerably in quality and completeness, as they depended on record keeping across different implementers and programs. To augment the sample, a “snowball” approach was used, in which CSOs as well as implementers and AORs were asked to name other CSOs active in each country. The sample included all CSOs active in the field of electoral or political party strengthen and not limited to those participating in CEPPS activities. Cross section data from the three web surveys were analyzed using a variety of statistical methods (varying by research question), including cross-tab frequencies, t-tests and Chi￾Squared tests of means, and logit regression (with and without country fixed-effects). Evaluation Caveats Several limitations in the evaluation must be noted. Sample sizes often too small to yield statistically significant estimates or comparisons, and model specifications may be subject to missing-variable bias due to limited CEPPS administrative data. The CSO sampling frame, despite efforts made, is likely missing both active and defunct CSOs, and may not capture precisely which implementers they have worked with. Finally and importantly, it must be noted that many different factors beyond the control of the CEPPS program are affecting program outcomes. It may not be possible to attribute results to specific program activities. FINDINGS Evaluation Question 1: Number of Active Organizations The number of CSOs active in elections assistance and political processes appears to have increased over the past six years, although the evidence should be interpreted with caution. CSOs and implementers mostly believed the number of CSOs had increased over the CEPPS III period, and case study data suggests there was a growth trend in countries with reasonably permissible environments. There was statistically significant evidence associating an increase in CEPPS III CSOs encaging in only political or only electoral processes. However, observers cautioned this increase in quantity does not necessarily imply a stronger or more vibrant civil society. Evidence is mixed on increases in the life expectancy of CSOs under CEPPS III. 9 Evaluation Question 2: Degree of Deepening from CEPPS II to CEPPS III Technical capacity building saw the largest increases between CEPPS II and CEPPS III, with organizational capacity building showing a positive but not statistically significant trend. There was a downward trend in CSOs not receiving ether organizational or technical support, although the trend was only significant for NDI-supported countries. Asia had the only statistically significant trend for all 3 categories (technical capacity building, organizational capacity building, or both), and Eurasia was almost static between CEPPS II and CEPPS III. CSOs reported receiving a greater share of their awards in the form of contracts, and a lesser share in the form of grants, although results were only statistically significant for IFES. Asia, Eurasia, and LAC follow the global trends of rising rates of contracts and falling rates for grants. Africa and MENA show the opposite: rising shares of grants and falling shares of contracts. Overall, there were few changes in competitive award shares in the CEPPS III period, with considerable regional variation with only Asia and MENA showing statistically significant substantial changes over the period. Key informant interviews with CEPPS implementers revealed that Objective 10 was warmly received although some do not believe it was prioritized by USAID. Interviews with implementers showed some differences between NDI, IRI, and IFES on the selection of CSOs, ranging from informal conversations with USAID and local partners to competitive selection. When asked about the balance between maintaining relationships and competitive selection, most AORs believed there was an optimal balance. Only IFES-supported countries were more likely to use competitive procurement, but this distinction disappeared when the bidder was a CEPPS II awardee. Both the AOR opinions and the CSO data seem to be aligned and indicate evidence that having a CEPPS II grant increased the probability that a CSO received a CEPPS III grant. Competitive selection is used less than a third of the time, but the mode of selection varies according to implementer, country context, and the capacity of the CSO sector, among other factors. Evaluation Question 3: The Nature of CEPPS III Partnerships Findings suggest implementers mostly involve local partners in the design, awards management, M&E processes, and reporting of programs, with 42% of CSOs reporting being involved to a great extent with the design of programmatic activities and only 14% reporting the implementer made these decisions for them. There were differences in perceptions about whether CEPPS incorporated CSO organizational capacity building as an integral part of collaboration, with implementers being more likely than AORs to report that their grants were given for capacity building. Capacity building is most usually delivered through implementer and local partners working jointly, although there is a rise in the share of capacity building provide in other ways. More than half of CSOs reported that capacity building provided by CEPPS III was useful and 78% said that some or most of the practices presented have been adopted and used. 10 Evaluation Question 4: Factors Leading To CEPPS III Success Findings show the following factors contribute to the successful elevation of local organizations into sustainable providers: consistent donor-approved overhead rate; the number of local awards above regional average; the number of international awards above regional average; having large number of organizational good practices. Respondents from case study interviews cited a clear mission statement, a vision of the future, financial accountability, and a predictable management structure as key factors for sustainability. Main factors for success according to implementer and CSO opinions were funding, CSO collaboration, and capacity-building. Though there are consistent statistically significant relationships between sustainability inputs and our sustainability indices, being a CEPPS III CSO did not affect the strength of this relationship. Evaluation Question 5: Assessing CSO Sustainability and Financial Management Other CEPPS efforts to support opportunities for local organizations for becoming direct recipients of local funding include the level of effort of implementers, involving the CSO in the project design, awards management, or M&E process, a CEPPS III implementer introducing a CSO to a funder, helping the CSO prepare materials for gaining funds, and identifying activities for the CSO to find new funding. However, almost a third of implementers indicated that helping CSOs obtain direct funding was an objective of the program. Case study interviews yielded some examples of explicit support to CSOs to become direct recipients, although many also reported preparation for direct funding was not an explicit goal. Implementers have a more-positive opinion than CSOs on the extent to which they built CSO capacity to apply for larger awards, independent of which implementer gave the support. The level of effort an implementer expended in helping a CSO obtain funds from other donors, as reported by the CSO, was positively correlated to the CSO actually obtaining those funds. The most common activity to support sustainability, as reported by implementers, was working with the CSO to identify activities they could provide that would be of interest to donors. The second most common was introducing the CSOs to potential funders. Finally, an implementer’s involving CSOs in the reporting, design, awards management, and M&E processes of CEPPS III activities had no effect on the CSO’s sustainability. Case study interviews identified that established CSOs are often more favored to become prepared to receive larger and more complicated awards. Some Missions were also skeptical of direct funding and saw it as a bigger management burden that would be better suited for a small Mission. 11 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE USAID seeks to better understand how the CEPPS III Leader with Associates (LWA) has contributed to the capacity of local organizations to support elections and political transitions in the countries where CEPPS is active. In particular, the evaluation will examine achievement of Objective 10 of the CEPPS III LWA, which is, namely “Building capable and sustainable local and regional civil society organizations engaged in election assistance and in providing technical assistance to political parties and political processes.” The evaluation will further seek to identify factors hindering or assisting those results, and will make recommendations for future program management and design. 1.2 EVALUATION QUESTIONS This evaluation is based on five evaluation questions developed by the Evaluation Team in collaboration with USAID: EQ1. Are there more CSOs active in elections assistance and political processes than five years ago? EQ2. How were programmatic strategies developed under CEPPS II altered or expanded under CEPPS III to address local capacity building goals under Objective 10? EQ3. To what degree did CEPPS involve local partners in the design, management, M&E process, and reporting of programs with which they were affiliated? EQ4. Which factors contribute to the successful elevation of local organizations into sustainable providers of technical assistance related to elections and political processes? EQ5. What efforts on behalf of CEPPS have been made to support local organization opportunities for becoming direct recipients of donor funding? 12 2.0 PROGRAM BACKGROUND AND OBJECTIVES 2.1 PROGRAM CONTEXT USAID’s Consortium for Elections and Political Process Strengthening III Leadership with Associates (CEPPS III LWA) is a program currently being implemented by the International Republican Institute (IRI), the National Democratic Institute (NDI), and the International Foundation for Electoral Strengthening (IFES). The CEPPS program, managed by USAID’s Office of Democracy, Human Rights, and Governance (DRG), builds on decades of work to support and strengthen electoral and political processes, as well as the civil-society organizations (CSOs) that promote these activities globally. There are 54 countries that have been CEPPS III LWA recipients, with 92 total CEPPS Associate Awards. These countries are presented on a program map in Annex II.A. Although there are a number of objectives of the CEPPS III LWA related to elections and political processes around the globe,1 this evaluation emphasizes Objective 10: Building capable and sustainable local and regional civil society organizations engaged in election assistance and in providing technical assistance to political parties and political processes. Objective 10 is unique to CEPPS III and represented a new priority in the democracy and governance sector. As included in the program description of the CEPPS III award, it states: USAID aims to strengthen the capacity of indigenous local/regional organizations in developing countries to: 1) conduct elections related activities; and 2) provide technical assistance and training to other local organizations on elections and political processes. The rationale is that strengthened election-oriented organizations in developing countries will: 1) broaden the community of democracy promoters and advocates abroad; 2) demonstrate that USAID assistance leads directly to sustainable change among local organizations, without dependence on intermediary organizations; and 3) promote replication. CEPPS III LWA predates – but is consistent with – the USAID Forward initiative for supporting local solutions, which grants awards directly to capable organizations in host countries to address development needs. The award incorporate two specific anticipated results: 1 This includes objectives such as: developing an impartial legal framework for elections and political parties; building a sustainable, indigenous capacity to effectively administer elections; informing and empowering all members of society to become active participants in the political process; supporting local, regional and international efforts to monitor electoral processes; ensuring inclusive political participation wherein a greater range of groups, including women and other historically excluded populations, influence and participate in political processes; promoting consensus-building for peaceful agreement on democratic reform, rules and framework through peaceful, broad-based participation in defining and negotiating changes to governing structures; strengthening political parties to ensure a competitive multi-party system that represents the diversity within a country; fostering the smooth transfer of power following elections; and strengthening governance capacity of elected leaders and bodies. Most of these objectives are virtually identical to objectives of the CEPPS II LWA. 13  Improved technical and organizational capacities of local civil society and regional organizations to implement and sustain election-related programs over the long term. This includes strengthened abilities to compete for, administer and account for resources from foundations, U.S. nonprofits, international organizations, and international financial institutions, and bilateral donors.  Local (non-U.S.) civil society organizations provide technical assistance and training to other local organizations on elections and political processes activities. [emphasis in original] To accomplish these objectives, the CEPPS III LWA offers the implementing consortium and its partners a variety of tools.  Under the Leader Award, global field support is granted in the areas of assessment and program design, state-of-the-art research in elections and political processes, evaluations, knowledge management and transfer of experience, and sub-grants to indigenous local/regional organizations. The Leader Award also contains a “rapid response” function to provide quickly key support in suddenly emerging country contexts unanticipated in the annual work plan when normal programming processes are insufficient. In 2011, the Global Women’s Leadership Fund was established as part of this Leader Award with the dedicated purpose of improving women’s participation in consensus-building processes.  Through Associate Awards, CEPPS has provided technical assistance, sub-grants, international exchanges, and other means to involve local and regional partners in achieving specific results of each award. Associate Awards have also included capacity building of local organizations as a specific activity, although capacity building also may have occurred through other activities not specifically identified as such. Accordingly, it was important to include especially Associate Awards in NORC’s evaluation of CEPPS III LWA. 2.2 PROGRAM COUNTRIES AND TYPES The table below lists all the countries where CEPPS III programs have been implemented. See also a map of program countries in Annex IIA. The Evaluation Team (ET) prepared two tables classifying programs by intervention type in order to get a sense of the diversity of the programs and the extent to which building the capacity of CSOs was an explicit requirement. The two tables can be found in Annex II.B. The first of these tables identifies programs (by country) that had an explicit local capacity objective to improve the capacity of local CSOs. The table subdivides them by type of capacity building, both technical and organizational across different activities, such as support for promotion of civic education around elections, those that support capacity to monitor elections, and so forth. 14 Table 2.1: Country Programs Africa African Union, Angola, Burkina Faso, Burundi, CAR, Côte d’Ivoire, Guinea, Kenya, Liberia, Malawi, Mali, Niger, Nigeria, Sierra Leone, Somalia, Sudan, South Sudan, Togo, Uganda, Zambia, Zimbabwe Middle East & North Africa Egypt, Iraq, Jordan, Lebanon, Libya, Mauritania, Morocco, Syria, Tunisia, Yemen Asia Bangladesh, Bhutan, Burma, Cambodia, Fiji, Indonesia, Maldives, Nepal, Pakistan, Papua New Guinea, Sri Lanka LAC Colombia, Ecuador, Guatemala, Guyana, Haiti, Honduras, Mexico, Nicaragua Europe Bosnia & Herzegovina, Kosovo, Macedonia, Serbia Eurasia Armenia, Azerbaijan, Belarus, Georgia, Kazakhstan, Kyrgyzstan, Tajikistan, Ukraine, Uzbekistan The second table lists the programs with no clear objective of building CSO-related local capacities, noting their areas of activity across six categories such as conflict prevention and resolution, international observation of elections, political processes, and governing institutions, and promoting public awareness. Both tables also note the political environment in each country, through a composite Freedom House rating that measures whether countries are “Free,” “Partially Free,” or “Not Free.” 3.0 EVALUATION METHODOLOGY 3.1 EVALUATION MANAGEMENT The Evaluation Team (ET) consisted of a project manager, subject matter experts, evaluation/ performance management specialists, research analysts, and web survey specialists. 3.2 WEB SURVEY DESIGN AND IMPLEMENTATION The evaluation team conducted three separate web surveys to collect data on perceptions of the CEPPS III program from the intended beneficiaries of the program, and program implementers, and USAID staff country AORs. As such, each of the web surveys targeted a different respondent group: (1) civil-society organization (CSO) CEPPS grantees and applicants; (2) CEPPS III implementer Chiefs of Party (COPs) from each of the three implementers, IRI, NDI, and IFES; or (3) USAID country Agreement Officer Representatives (AORs). The ET formulated questions for web surveys to administer to the CEPPS beneficiaries, implementer COPs, and USAID AORs following a review of CEPPS III documents and input from USAID, IRI, NDI, and IFES staff. Once the questions were developed, they were vetted and tested by the entire ET The questionnaire development process was iterative, with all members of the ET providing feedback on each revision. Following USAID approval, the ET conducted two rounds of internal cognitive testing to ensure that survey questions were clear, followed a logical pattern, not cognitively difficult for participants, , and were culturally sensitive. 15 The final English-language version of the CSO survey instrument was then translated into Arabic, French, Spanish, and Russian, and another round of cognitive testing with native speakers took place. The evaluation team relied on USAID and the implementers’ (IRI, NDI, and IFES) support to provide sampling frames for each of the stakeholders. IRI, NDI, and IFES provided COP contact information for each CEPPS III project, and also provided contact information for all CSOs with which they have worked previously on a CEPPS III project. A rolling sample was also utilized, wherein we requested respondents to provide us with additional CSO respondents to be included in the sample. USAID provided the list of USAID AORs for each of the CEPPS III projects. The original CSO sample received from CEPPS consisted of 805 individuals (after removing invalid and inactive emails – see the Implementation section below), with an additional 213 respondents being obtained via the rolling sample during the survey launch period, bringing the total sample size for the CSO survey to 1,018. The implementer survey had a final sample size of 128 COPs from IRI, NDI, and IFES. The USAID staff survey has a sample size of 63 AORs from different CEPPS III projects. More detailed information on the methodology is provided in Annex III. 3.3 SUMMARY OF QUANTITATIVE METHODS NORC used quantitative methodologies to assess each of the individual evaluation questions (EQs) and sub-questions presented in this report. We used cross-sectional data from separate web-surveys of CSOs, AORs, and implementers. Methods used include cross-tab frequencies, statistical significance t-tests and Chi-Squared tests of means (where feasible), and logit regression (with and without country fixed effects). Quantitative results were then compared to qualitative analyses conducted by NORC’s team of subject-matter experts. Each evaluation research question (and associated sub-question) employed a slightly different set of quantitative methods appropriate for the issues raised. Since detailed step-by-step procedures utilized are presented in Annex VII, only a brief summary of quantitative methods is presented below.  Evaluation Question 1: The Evaluation Team (ET) team conducted a series of regressions and hypothesis tests examining the relationship between the presence of CEPPS III recipients in political or electoral processes, and how that number changed from CEPPS II and CEPPS III recipients from 2009-2014. In estimating these relationships, we controlled for other factors that would be associated with increases in the number of CSOs over time, such as the amount of funds spent in a country, the political environment (as measured by existing indices), GDP per-capita, and other variables described in the evaluation results section.  Evaluation Question 2: NORC compared opinions of implementers, CSOs, and AORs during CEPPS III to evaluate how CEPPS II programmatic strategies were altered or expanded under CEPPS III. The NORC team supplemented these descriptive results with statistical hypothesis tests and more attributable regression analyses. Questions of interest 16 included responses on organizational capacity-building, technical capacity-building, joint￾work assistance, and changes in the numbers of grants and contracts from CEPPS II to CEPPS III.  Evaluation Question 3: The NORC team used the surveys of CSOs and implementing staff to examine the degree to which CEPPS engaged local partners in awards management, M&E processes, and reporting of programs with which they were affiliated, the. NORC explored the degree to which CEPPS participants believed they involved CSOs in the above processes, while the CSO survey assessed the extent to which CSOs believed they were involved in the above tasks. Quantitative analysis further examined CSO responses by issue￾area across countries to test for differences by implementer, country, and issue-area.  Evaluation Question 4: To examine the factors associated with moving CEPPS partner CSOs towards sustainability as technical assistance providers, NORC developed custom sustainability indices to gauge how the sustainability of CSOs evolved between CEPPS II and CEPPS III. We then ran hypothesis tests and regressions on our sustainability measures, looking at the relationships between key inputs to sustainability, and how they were related to our sustainability indices.  Evaluation Question 5: To analyze efforts that have been made to support local organizations to develop so they can receive donor funding, NORC looked at the various inputs to CSO support, and how that affected the likelihood that they would receive funding from other sources. NORC examined the various activities and initiatives that implementers had available to encourage CSOs to seek larger and more complicated awards. We then ran a series of hypothesis tests and regressions to see how the inputs and activities were associated with the outcomes. 3.4 CASE STUDY SELECTION AND IMPLEMENTATION To complement the quantitative data obtained through the web surveys, the evaluation also included six case studies. NORC conducted a careful process to select the six country programs to be the object of the case studies. The objective was to select a varied set of programs that cover different regions, approaches, and subject matter (for example, of political versus electoral processes) in a variety of contexts. A primary goal was to be able to evaluate the work of all three implementers, IRI, NDI, and IFES. The intent was to explore both successful and less successful programs, as well as to find some countries where there would be fairly robust Civil Society Organizations. The selection process for case study countries involved asking the three implementers (IRI, NDI, and IFES) about recommendations to obtain a sample that ideally had one program per subject area (political processes and electoral processes). Then, the ET assessed the countries with a rating system to access the involved capacity building of CSOs, an environment hostile to CSOs, or an environment that was deemed problematic. These countries were then grouped by region, and the ET selected the following countries based on aggregate results: 17  Africa. Kenya, Malawi, and Nigeria were all considered strong candidates; Kenya and Nigeria had the advantage of having all three implementers. Nigeria seems more focused on CSOs, but had the advantage of being a large country. Zambia was an unusual case, with programming relying exclusively on one CSO and prioritizing its development, and provided the sole example of that approach to building sustainable capacity. Nigeria and Zambia were selected.  Middle East and North Africa. Tunisia was an example of a short and intensive program focused on building the capacity of CSOs in electoral processes and in internal organizations. It is also an example of a program that was implemented without an USAID AOR on the ground out of necessity. Jordan was a possibility, but only had one implementer present and only focused on two CSOs even though that was its focus. Tunisia was selected.  Asia. Bhutan had the highest score, but only one implementer and a very small budget. Cambodia and Burma both have three implementer projects, and active CSO programs. The environments in the two countries are very different, although both presented challenges. Cambodia was selected as a larger and more mature program.  Latin America and the Caribbean. It was decided that only one country would be selected for LAC given the smaller number of LAC CEPPS countries overall. Guatemala and Honduras had the highest scores. Guatemala was selected because the program there has all three implementers and both political and electoral programs.  Eastern Europe & Central Asia. Georgia had the highest score, programs involving all three implementers, and a current program. For a more detailed explanation of the case study selection process, see the Introduction to the Appendix with the Case Studies. 3.5 LIMITATIONS The ET acknowledges several limitations inherent to the design of this evaluation. Some of the more relevant limitations are listed below: Sample Size and Selection First, sample sizes were often too small to yield statistically significant estimates or comparisons, particularly given the intrinsic noisiness of categorical data. For the CSO-level analyses in Evaluation Research Questions 3, 4 and 5, we used a categorical variable which attempted to assign CSOs to whichever implementer – NDI, IRI, or IFES – that they had collaborated with during the CEPPS III period. We did this by matching the CSO respondents to the respective CEPPS III implementer sampling frame source we got the respondent from. The variable could either have been IRI, NDI, or IFES. There are several concerns with this method of sampling-frame construction. It is possible that even though the CSO contact was obtained from one CEPPS III implementer, the CSO might have also worked with another CEPPS III implementer. For instance, the sampling frame from the latter CEPPS III implementer 18 could have simply not included the CSO. NORC was explicitly told that the lists may not be comprehensive. Further, even in the case where we received duplicate respondents from multiple CEPPS III implementers, the CSO name or contact could have been spelled differently or been in a different language (English and the local tongue). It could be the case that, for example, both IRI and IFES provided a common respondent, but our “sample source” variable may have identified them as different CSOs. Third, while we sent email web-survey solicitations to all CSOs on the CEPPS III implementer lists, not all CSOs responded. NORC did not have the resources to determine whether the non-response was due to the CSO being defunct or the recipient being too busy (or uninterested) to send a return email demurring from participation. One consequence of this imperfect sampling construction is that all CSOs in the sample existed in 2014 and that no defunct CSOs were included in the sample. Secondly, in spite of the use of “snowball” sampling (a technique in which the initial sample of respondents are asked to provide contact information for another round of potential respondents)to compensate from incomplete lists of CSOs, the sampling frame is likely missing active CSOs as well as defunct CSOs. Due to limitations in the data, we were unable to directly observe changes in the number of active CSOs and the life expectancies of CSOs. The best approximation we could make on the number of CSOs was based on the number of responding CSOs from a given country, including those whose information was acquired through snowball sampling. However, since the survey took place in 2015, we were only able to collect data from currently operating CSOs. We did ask currently operating CSOs when they began their operations, and were able to identify which 2014-CSOs existed during or prior to 2009, and which formed between 2009 and 2015. However, any CSO that had disbanded prior to our survey would not have been captured in the sample. Therefore, the sample of CSOs that existed in 2009 is limited only to CSOs that were still operating in 2015. Likewise, surprisingly there were no post-2008-established CSOs included in the implementer lists upon which NORC based the CSO survey’s sampling frame. The CSO survey itself contains examples of such recent CSOs but the resources and time were insufficient to add the unique instantiations of these (i.e., overcoming spelling and language variations) into the implementer-based sampling frame. These are important caveats to the quantitative results in Section 4. Due to budget constraints, NORC targeted the web survey only to grantees and applicants of CEPPS awards, the implementers, and USAID staff in selected CEPPS countries. Therefore the web survey is not entirely representative of the global nature of the program. The evaluation team obtained respondent names for the web survey through IRI, NDI, IFES, and the organizations being evaluated.2 While this strategy was utilized to maximize the response rate and obtain a sample of CSO partners that were able to actually speak to their experience with CEPPS, several limitations must be noted. First, it raises the possibility of selection bias as, at least in principle, the implementers might choose to pass on information of only the organizations most positively disposed toward the program. In addition, because the sample of CSOs was provided by the 2 NORC considered seeking other sources of CSO names and contact information, such as CSO registries, or listings from other USAID projects. However, that would also have major disadvantages, such as resulting in many CSOs whose work is not germane to the CEPPS program, and potentially be time-consuming and costly. The evaluation team concluded that despite the limitations, the implementers remained the best source. 19 implementers, it is not representative of all CSOs in each country, or even necessarily of all the organizations with whom IRI, NDI, and/or IFES has worked. The sampling approach requesting additional CSO representatives from respondents increased the representativeness of the sample, but it still must be noted that the CSO samples are not representative of all relevant CSOs in each CEPPS country. Additionally, because of the web-based platform the respondent pool was also limited by CSO members with access to email and internet to complete the survey, and NORC received many outdated emails from the implementers. Difficulties in Obtaining Documents As mentioned above, we relied on IRI, NDI, and IFES to provide sample documents for both CSOs and their own staff. There was a challenge in obtaining sample documents from the implementers, particularly for IFES, which led to time delays and a potentially less robust sample than desirable. Further complicating analysis, it was very difficult to obtain verified accurate, complete, and up-to-date budget information for all CEPPS II and CEPPS III projects. Model specifications may have missing-variable bias given the scarcity of CEPPS administrative data, particularly budget data. Further, there are many factors or variables beyond CEPPS that may contribute to the results described in this report. Therefore, the reader should be careful not to infer causation from correlation. Where this is appropriate, a causal attribution is made explicit in the text. USAID Staff AOR Response Rate It is important to note that while the evaluation teams was able to obtain sufficient data for analysis, the response rate for the AOR survey was particularly low, especially given the assumption USAID staff should be the most willing to assist with the evaluation. Only 27 completed surveys were received out of a possible total of 63, despite sending out multiple automated reminders and a targeted reminder from the DRG lead. Evaluation Timeframe This kind of program may take many years to produce concrete results. Thus the ability of NORC’s evaluation team to determine sustainable results at this early stage was challenging. The above limitations, however, did not prevent the ET from gathering the information and data needed to produce findings, conclusions and recommendations (however tentative) for this particular performance evaluation. 20 4.0 FINDINGS AND CONCLUSIONS 4.1 EVALUATION QUESTION 1: NUMBER OF ACTIVE ORGANIZATIONS Q1.1A: Are there more CSOs active in elections assistance and political processes now than six years ago? CSO, Implementer, and AOR Perceptions of CSO Growth Trends The evaluation’s three web surveys asked CSOs, implementers and AORs if they believed that there are more CSOs operating now than in 2009. An average of 74% of CSOs, 57% of implementers, and 43% of AORs believed there were more CSOs today than in 2009. Summary statistics for this result can be found in the Annex under Table 4.1.1.3 Key informant interviews and case studies offer support to these findings. The case studies confirm a trend that the number of CSOs in a country with a reasonably permissive operating environment is likely to have grown over the past 6 years, generally, as well as in the specific elections and political processes sector. Where growth in the number of active civil society organizations working on issues related to elections and political processes was observed, it tended to follow a momentous occurrence in the political environment of that country. For example, more democratic elections in Nigeria seem to have spurred an increase in civil society activity since 2011. Similarly, prior to the Arab Spring Revolution in 2011, the stakeholders reported little if any CSO involvement in electoral processes; this changed starting in 2012 when CEPPS partners noted an increase in CSO engagement, particularly with the youth. Similarly in Guatemala, prior to the 2008 elections, monitoring and other electoral activities were done by fewer and less organized CSOs. The international prosecutors’ panel (CISIG) investigation implicating more than 150 government officials in a massive fraud and money laundering scheme in April 2015 prompted civil society 3 Throughout the Findings section, summary information or abbreviated tables may be provided. Annex IX contains a full set of the complete tables with all statistical information. Principal Findings  Most CSOs and implementers believe the number of CSOs increased over the CEPPS III period.  Case study data suggests a growth trend in countries with reasonably permissive environments, but observers caution that increase does not necessarily imply a stronger or more vibrant civil society.  There is statistically significant evidence associating an increase in CEPPS III CSOs engaging in political or electoral processes with an increase in non￾CEPPS III CSOs working in those areas 21 into action. Civil society organizations that had previously been relatively dormant, or had worked in sectors other than electoral processes were united in their demand more transparent and honest governance. In particular, organizations advocating on behalf of traditionally underserved segments of the population (i.e. youth, women, people with disabilities, and LGBT) are more likely to engage in electoral and political processes in the course of political upheaval; this was observed in Cambodia, Guatemala and Georgia. Surges in the number of CSOs engaged in electoral processes after political upheaval is likely attributable in part to a corresponding uptick in the availability of donor funding. In Georgia, for example, donor allocations for civil society tended to rise as successive governments became increasing truculent. In response, greater civil society engagement in electoral process was observed towards the end of President Shevardnadze’s tenure, in the advance of the Rose Revolution in 2003. Similarly, as concerns grew over the abuse of administrative resources and interference with the opposition’s ability to campaign at the end of President Saakashvili’s second term in 2012, more money was made available to civil society organizations to monitor and advocate for free and fair elections. In Nigeria, as donors increased their support to electoral activity, CEPPS partners observed an increase in the number of CSOs interested in electoral monitoring. Conversely, when donors pull out of a country, the number of CSOs engaged in electoral and political processes tends to drop. As observed in Tunisia, once the post Arab Spring elections were completed and USAID’s Office of Transitions Initiatives (OTI) program ended, many of the former CEPPS partner CSOs disappeared. This correlation was confirmed by the CSO interviewees themselves, who cited lack of funding as a primary obstacle to their subsistence, as well as by CEPPS partners. However, caution should be exercised before concluding that growth in the number of CSOs is tantamount to a stronger or more vibrant civil society. Although a substantial increase in number of registered CSOs was observed in Cambodia and Nigeria, this fact is quickly qualified by the subjective observation that the vast majority of those NGOs are inactive. Another phenomenon specific to elections and political processes was observed in Georgia where stakeholders noted that CSOs were “springing up” just before elections for the purpose of providing a parallel narrative to the conclusions in observation reports issued by the established monitoring organizations. While agreeing that there have been increases in the number of CSOs in the countries where they have worked, a number of implementers interviewed added another caveat against assuming that “more is better.” For example, one CEPPS staff member at headquarters said that while there has been “an explosion of civil society” in Libya, “I would not say that this equates to increased capacity; quantitative increase does not necessarily mean qualitative increase. Instead we see a lot of not very good CSOs and very few strong ones. Those that could speak English, had EU references, wound up sucking up a lot of the international assistance. ” Another interviewee remarked that “In Liberia, USAID said they did not want a sustainable coalition and directed NDI not to create an entity that expects to be funded in between elections. Instead, we wanted to help USAID create the space that would allow for elections observations six years from now in Liberia. In that regards, our efforts have been a success. It doesn’t matter if there is one, or multiple organizations; what counts is the capacity.” 22 And another CEPPS senior advisor said: “There are many more CSOs than before. More does not necessarily mean better. But more are using better methodologies, and are collecting evidence. There is a shift from a focus on a single national election observation organization, to a more inclusive focus on networks. They have applied these skills to apply to other activities that they are involved in….i.e. the PVT methodology can be applied to education testing and corruption, or medicines, media monitoring, etc. “ The opposite was also cited: one CEPPS home office staff member commented that “In Russia, [for example, there are] fewer organizations, but those that were active are largely still active and strong.” Is an Increase in Number of CSOs a CEPPS goal? One underlying question is whether CEPPS activities actively support an increase in the number of CSOs. Most interviewees at CEPPS headquarters did not see that as an objective for CEPPS nor as a development related to CEPPS’s activities. A key informant from IRI headquarters said: “I think that CEPPS influence has little to do with it. That is, in Libya, Burma and Afghanistan there are probably a lot more [CSOs] because they can form now, as before the environment was closed. Jordan, likely numbers are consistent. It comes down to the organization and what they are allowed to do.” Many others also ascribed the growth (or decrease) in the number of CSOs as being primarily related to the environment. One interviewee commented that in Libya there has been an “explosion” of civil society, post revolution; in Egypt, post Arab Spring there were more CSOs active, but that quickly constricted. The constriction persists today, and the civil society sector is now worse than in 2011. In Pakistan there has been a consistent and steady increase in its number and capacity of civil society actors; “…which I attribute to a general improvement in the democratic environment in a lot of ways. The country gets a lot of flak for its extreme elements, but politically it has opened up.” It also appears that working to increase the number of CSOs has rarely if ever been an explicit goal for CEPPS. A member of senior management at one of the CEPPS implementers said, “We haven’t really evaluated [the increase in the number of CSOs]. There was never a baseline assessment and we wouldn’t necessarily measure this unless there was something written into a PD. The growth of capacity and growth of space in which to operate is more what we were interested in.” They added: “Sometimes we are asked to find new CSOs that need capacity building, or to grow the CSO community (engage more actors), or to build networks/coalitions. For example, this type of guidance/directive could come up in a situation where USAID asks them to get more people involved in a particular sector, i.e. women’s issues. In this case, there might be a volume increase in CSO partners, though not necessarily NGOs.” Interestingly, at least one interviewee noted that there is somewhat of a contradiction between USAID Forward and a desire for “more CSOs” – to develop one strong local CSO would probably require working more intensively with a smaller number of CSOs to develop their organizational capacity. “We are big proponents of pushing USAID to not work with the same groups over and over again. USAID Forward, in theory, is pushing us to work with the same NGO groups, but a negative consequence is the donor community given the prominent NGOs 23 huge amounts of money, beyond which they can effectively handle. USAID also often says that they want to work in the regions, but the reality is that they prefer big urban NGOs that speak English.” Changes in the Number CSOs Engaged in Political and Electoral Processes To explore the change in the number of CSOs engaged in political and electoral processes in our quantitative analysis, we undertook two steps, first comparing the average number active in 2009 versus 2014 per country (and performing tests to determine if that change was statistically significant), and then running a series of regressions to try to determine whether the number of CSOs was affected by the CEPPS program. Although we were unable to directly observe changes in the number of active CSOs in our data (see Section 3.1 on caveats), our survey allowed us to approximate changes in the number of CSOs engaged in political versus electoral processes over time. We measured this by asking existing CSOs if they were or had been engaged in electoral or political processes. CSOs who confirmed participation in such processes were then asked in what year they began those activities. This enabled us to identify which existing CSOs were engaged in political or electoral processes in 2009, and which were not. Table 4.1.2 presents the results of difference-in-means t-tests on the number of active CSOs between 2009 and 2014. Table 4.1.2. Average Number of CSOs per Country, 2009 and 2014 CSO engaged in… 2009 2014 Change Electoral processes only 3.22 4.20 0.98 Political processes only 3.42 5.04 1.62* Both political and electoral processes 1.24 2.76 1.52 Notes: Sample size was 55 countries for both periods. * indicates statistical significance at the 10% level. For all three types of CSOs above, there are general upward trends in the average number of CSOs between 2009 and 2014 although the only statistically significant change was for the CSOs engaged in political processes, which increased by approximately 1.6 CSOs per country between 2009 and 2014. When interpreting the results in Table 4.1.2, it is important to keep in mind this simply compares group means between 2009 and 2014, and does not control for other factors that could have led to increases in the number of CSOs per country. For estimates of changes in the number of CSOs that might be more attributable to the CEPPS program, we ran a series of regressions with the number of electoral and political CSOs (whether or not engaged with CEPPS) per country as the dependent variable, dummy variables for 2009 and for 2014, a variable with the number of CEPPS III CSOs in electoral or political processes, and a vector of control variables to account for external factors that may be 24 correlated with the number of CSOs per country.4 (Country fixed-effects are a method of controlling for differences between countries over time.) We first ran the regressions with the full vector of control variables, then ran them again by removing the time-invariant characteristics and using country-level fixed-effects.5 The results of the regressions are presented in Table 4.1.3, showing results for the full set of control variables and for the fixed effects regressions in alternating rows. Table 4.1.3 Number of CSOs between 2009 and 20146 (1) General Trend in No. of CSOs (2) No. of CEPPS Electoral CSOs (3) No. of CEPPS Political CSOs (4) No. of CEPPS Grantees Only Electoral + + + + Only Electoral (FE) +++ 0 Only Political 0 + + + Only Political (FE) +++ + + + Electoral/Political - - - + + + Electoral/Political (FE) + + + + Note: Positive change: +++ p<0.01, ++ p<0.05, + p<0.1. Negative change: - - - p<0.01, - - p<0.1 For the table above, looking first at the first column for the general trends in the number of CSOs, a positive sign would indicate a positive increase in the average number of CSOs per country between 2009 and 2014. We see important differences between general results and results that included country fixed-effects (FE) in our regressions. For electoral and political CSOs, the former method shows a slightly positive trend, with only the electoral-process CSOs showing a statistically significant difference. However, under the country fixed-effects model, the average number of CSOs is highly significant. For the number of CSOs engaged in both electoral and political processes, the covariate-conditioned OLS shows a slightly negative (and highly statistically significant) trend between 2009 and 2014. However, under the country fixed effects model, the sign on the coefficient becomes positive and highly significant. 4 Specifically, the control variables were: GDP per-capita; Economist Intelligence Unit democracy index (free; not￾free; partially free); number of NGOs per country, per year; whether or not IFES, NDI, or IRI implemented in￾country; whether or not they had an election the previous year; average age of CSOs in-country. 5 Country fixed-effects require adding a dummy variable for each country into the regression, which allows us to control for “unobserved” differences across countries. For instance, if any cultural factors that could not be measured were impacting the outcome variable, using fixed-effects allows us to control for cultural differences in estimating impact. Time-invariant characteristics are removed because, in theory, country fixed-effects would already be controlling for time-invariant characteristics. The time-invariant variables removed were these: Economist Intelligence Unit democracy index (free; not-free; partially free); number of NGOs per country; and whether or not IFES, NDI, or IRI implemented in-country. 6 This table includes statistically significant results only, and has been simplified by removing coefficients and only leaving in the sign of the effects. The full table is included in Annex VIII. 25 To examine any signs that the CEPPS program itself affected the number of CSOs,, we can look at the sign of the coefficient on the CEPPS variables – namely, the number of CEPPS electoral CSOs, CEPPS political CSOs, and the total number of CEPPS grantees per country (see columns (2), (3), and (4)). The CEPPS variables could be interpreted as “intensity of treatment” variables, with larger numbers of CEPPS grantees as a proxy for higher intensity of CEPPS involvement or influence in a country.7 The signs of the CEPPS coefficients in Table 4.1.3 – all but one are positive – indicate that, on average, larger numbers of CSOs receiving CEPPS grants lead to larger numbers of CSOs operating in their respective processes. For instance, an increase in the number of CEPPS-grantee CSOs operating in electoral processes by one, is highly significant; however, this effect shrinks substantially under the country fixed-effects specification. For CSOs engaging in political process, this number also significant, and remains significant even under the country-level fixed effects. EQ1.1B: Has life expectancy of CSOs increased compared to five years ago? To estimate the life expectancy of CSOs, we asked each CSO respondent the year in which the CSO started operations. This allowed us to calculate the average CSO age per country. We noted earlier the limitation that only CSOs that existed in 2015 could report on their ages, so that observations on the average CSO age in 2009 are excluding any CSOs that disbanded before the 2015 survey period. With that important caveat in mind, we present below the results of the quantitative analysis on average CSO age. Changes in CSO Life Expectancy Though there is a general upward trend in the sample-wide average age of CSOs between 2009 and 2014, the difference is not statistically significant. At the regional level, only Latin America and the Caribbean (LAC) show a statistically significant increase in CSO age. As with the analysis above on the total number of CSOs, when interpreting the results in Table 4.1.4, it is important to keep in mind the t-test results are simply comparing group means between 2009 and 2014, and do not control for other factors that could have led to increases in the life expectancies of CSOs per country. For more rigorous estimates of changes in CSO life￾expectancy, refer to the next sub-section on regression results. 7 Note that year-to-year CEPPS project budgets were not available at the country level. Principal Findings  Simple age averages seem to increase for CEPPS III, but results are only statistically significant for Latin America and the Caribbean.  Once country-specific factors are taken into account (controlled for), the results become ambiguous (sensitive to model specification). 26 Table 4.1.4. Average Life Expectancy of CSOs per Country, 2009 and 2014 Region 2009 2014 Change N Mean N Mean Total Sample 50 11.5 54 13.5 2.0 Asia 18 12.2 18 14.6 2.4 Eurasia 10 9.2 11 11.2 2.0 LAC 11 9.7 11 13.2 3.5* MENA 6 19.6 6 22.2 2.6 Note: *** p<0.01, ** p<0.05, * p<0.1. EQ1.2: Do a greater share of CSOs interact with CEPPS implementers or their local partners in 2014 compared to five years back? In our survey, we asked respondents to estimate the number of other local CSOs their organizations had substantial collaborations with in 2009 and 2014. Table 4.1.5 presents summary statistics for the question, disaggregated by CEPPS grantees/non-CEPPS grantees, and region. Table 4.1.5 Average Number of CSOs with which CEPPS-affiliated and Non-CEPPS￾affiliated CSOs Collaborated Region CEPPS (Mean) Non-CEPPS (Mean) 2009 2014 Change 2009 2014 Change Africa 14.7 17.4 2.7 14.7 9.8 -4.9 Asia 12.6 22.2 9.6 24 17.8 -6.2 Eurasia 10.9 15.5 4.6 9.1 16.7 7.6 LAC 6.5 9.4 2.9 9.7 10.8 1.1 MENA 10.1 14.4 4.3 4.5 6.0 1.5 As is suggested from Table 4.1.5, on average the CEPPS grantees had increases in the number of CSOs they were interacting with (though the increase, in spite of its size, was not statistically significant). CEPPS grantees in Asia had the largest increase between 2009 and 2014, adding on Principal Findings  Possible (not statistically significant) increases in collaborations for CEPPS grantees in each region ranging from 15% (Africa) to 75% (Asia).  Possible (not statistically significant) trends in collaboration for non-CEPPS were mixed, with Africa and Asia falling by over 25% and the other regions increasing modestly, with the exception of an 80% increase in Eurasia. 27 average 9.6 CSOs to their networks (though, again, not statistically significant). The trend was not uniformly positive for the non-CEPPS CSOs. We see negative trends for Africa and Asia for collaboration among non-CEPPS CSOs, and positive trends for the other regions (though, again, not statistically significant). 4.2 EVALUATION QUESTION 2: DEGREE OF DEEPENING FROM CEPPS II TO CEPPS III EQ2.1: How were programmatic strategies developed under CEPPS altered or expanded for Objective 10? Emphasis on Objective 10 and Building CSO Capacity Long-time observers, managers, and implementers of CEPPS note that there was a deliberate strategy in CEPPS III, which was new and different from CEPPS II, which was to build the capacity of local partner organization as manifested in the RFA. 8 In what ways were the approaches used in CEPPS II altered and adapted to better support the greater focus on local partners? 8 The USAID Request for Applications (RFA) for CEPPS III included specific information about capacity-building. Capacity-building included assisting local organizations to diversify their funding sources; manage multiple streams Principal Findings  Objective 10 was largely welcomed by CEPPS implementers  CSO strengthening is listed as an objective in almost two-fifths of CEPPS PDs, but some implementers don’t feel it is prioritized by USAID  There were clear increases in the amount of technical and organizational capacity building delivered under CEPPS III compared to CEPPS II, although only some of the trends were statistically significant.  Globally, CEPPS technical capacity-building saw the largest increases  There were upward trends in organizational capacity building, but these were not statistically significant  There was a downward trend in the number of CSOs who received neither organizational nor technical support, though the trend was only significant for NDI-supported countries, and – when broken out geographically – only in Asia. Asia had the only statistically significant changes in all three of these categories, with big increases in the region in both organizational and technical support.  Although not statistically significant (except for IFES), there were general upward trends for reported shares of contracts, and downward trends for grants. 28 Most implementers interviewed expressed satisfaction at this shift, and felt it was an acknowledgement in support of work they already believes was important. One implementer remarked: “CEPPS was initially intended to solve the problem of how to develop a system to rapidly help when elections come up. The CEPPS II RFA had a little more structure. When CEPPS III RFA was released, the appearance of objective 10 was a pleasant surprise. We gladly wrote our response based upon some of the things we have already done.” A representative from NDI said about Objective 10 “When it began, we weren’t sure if it was in recognition of the fact that we were already doing that work or whether it was a desire to do more. At that time there were a dozen examples of organizations that NDI had helped to stand up. It is as much recognition of that as anything else. At the time we were institutionally examining partnership. We spent a lot of time analyzing what partnership means between us and local groups…. Our objective is to build organizations that are politically active and become part of the political landscape.” One interviewee said “In Guinea we had a seamless transition from CEPPS II to III. We had done capacity building in Guinea under CEPPS II, but having it explicitly spelled out (as in CEPPS III) is a good reminder.” On the other hand, in interviews with CEPPS implementers at headquarters, many respondents said they did not see a major change in USAID’s focus under CEPPS III. Many interviewees remarked on the lack of explicit guidance on capacity building from USAID; for example: “I don’t recall ever getting guidance on organizational development on CEPPS from USAID at the Washington level, but there probably has been at the individual mission level…. USAID has substantial involvement in terms of partner selection.” Many of those interviewed at the headquarters offices said they had been doing capacity building of CSOs earlier in any case. When asked about the inclusion of Objective 10 in CEPPS III, those interviewed were positive about it, but felt it was not a big priority for USAID. Another interviewee, when asked about guidance form USAID, said: “Very little. Hard time even thinking of any real guidance whatsoever, beyond the initial project description to which we respond. Certainly not in terms of ongoing guidance. We receive less guidance for civil society capacity building than in any of our other activities. USAID is more engaged in political party building, post-election government work, and in polling. Civil society mostly left up to us.” One informant from IRI said: “We are disappointed by the very small number of PDs that actually incorporated Objective 10. Few PDs called for organizations who can do training, or can serve the regional structures.” Another interviewee said “USAID provides little guidance….If they want us to do this, the PDs should provide more direction. The bulk of the money goes to other objectives. The priority of objective 10 is low. PDs are all that we have as guidance. There is sometimes a question and answer process around a PD, but it is never as clear as it should be.” of funding; improve accounting and financial management practices; expand their memberships, and; better articulate their missions, and that the USAID Request for Applications (RFA) for CEPPS III. 29 The ET conducted a review of 86 of the CEPPS PDs, which found that actually 33 of them (almost 40%) list capacity building as one of the explicit objectives. It should be noted, though, that almost all of those are focused on building technical capacity so that CSOs could carry out specific tasks such as monitoring elections, or citizen outreach, and not on organizational development. One of the few exceptions is the Civil Society Support Program in Malawi, for example, which listed one of three objectives as “to improve financial management and civic education capacity of CSOs”, and included illustrative activities such as “Provide technical assistance to improve CSOs’ proposal writing skills; provide technical assistance to improve CSOs’ financial management capacity to handle small grants.” But this is appears rarely in the PDs reviewed. Some interviews highlighted some exceptions in countries where there was in fact a very strong focus on CSO capacity building. “Afghanistan and Iraq are a bit of exceptions, and in both of those places we were asked to develop civil society and we did, among many other actors. And currently with both of those programs sunsetting for IRI, now not so much.” “In Syria we have had for a long time, and continue even now, an explicitly civil society focused project. There are components of a political bent, but the emphasis is on civil society. We are building something of a civil society there. Can’t say it is too successful due to the environment….IRI’s CSO work is more funded by DRL [US State Department Bureau of Democracy, Human Rights and Labor] than by CEPPS. Jordan has civil society components, but it hasn’t been a major focus of the work there.” The case study countries provided examples of a shift to greater focus on CSOs, although mostly due to circumstances. Under CEPPS III, there was initially no USAID objective, requirement, or request to build the capacity of local organizations in Cambodia, Georgia, and Nigeria. In Nigeria under CEPPS II, CSOs had been considered too weak and dispersed to include in programming but following a number of reforms made, USAID decided “to pull CSOs into the process” in an effort to change the democratic trajectory in Nigeria. In Cambodia or Tunisia (where no CEPPS II programming had taken place) CEPPS III was introduced in 2009 and 2011 respectively. Capacity-building activities began in earnest in Cambodia after a recommendation was made to do so in a USAID mid-term performance evaluation of the CEPPS III activities. In Georgia, program descriptions under CEPPS III called for civil society engagement, without specifically referencing an intent to build their organizational capacity. In Guatemala, some of the programming under CEPPS III included a greater emphasis on strengthening the capacity of CSOs to “prevent electoral violence”, to “promote and implement key [electoral] reforms”, and to “participate” in the electoral and political processes, but it was not prompted by the addition of the capacity-building Objective 10 in the CEPPS III LWA. In Tunisia, however, capacity-building activities, both integrated and conducted in parallel with program activities, commenced from the beginning of the program awards to the CEPPS III implementing partner organizations. Whether or not specifically addressed in country program descriptions under CEPPS III, as it was in Tunisia, it is evident that USAID appreciated the extent to which CEPPS partners had built the capacity of its partner CSOs and encourages capacity-building for local organizations. 30 In 2013 in Georgia for example, USAID began planning for local sourcing election observation and vote verification activities, and provided for funding to local organizations to conduct short￾and long-term electoral observation. USAID Georgia is also currently encouraging receiving funding applications from organizations that have not previously received direct funding. An interviewee from IRI responded: “CEPPS III may be a little more focused on the internal OD aspects, but it comes more directly from State and NED—and USAID reaps the benefits a little bit. IRI has developed modules on grants management, Women’s Democracy Networks, etc., but none of this was funded by USAID. Of course, USAID’s programming benefits from working with more capable CSO partners.” The speaker when on to describe a CSO scorecard IRI had developed to assess the organizational capacity of CSOs, that has been incorporated institute wide. “We have different training modules that we use for all of our funding mechanisms.” But they then noted that the training is largely developed with funding from NED. IRI uses it in reporting to USAID but have not received feedback on it or ways to build CSO capacity. Shifts in the Approach to CSO Capacity Building under CEPPS III When asked about the types of strategies used by CEPPS III compared to earlier CEPPS programs, NDI cited peer-to-peer programming. “More of a focus on peer to peer learning ‒ much more focus on lessons learned, and building solidarity,” said one staff member. Another said: “NDI uses CEPPS III to focus more on bringing people together in a peer/peer relationship. For instance, by bringing representatives of election monitoring groups to visit colleagues in other countries, NDI is able to create wider networks. NDI is able to use representatives from sophisticated civil societies in Chile or South Africa as resources for other in an earlier stage of capacity.” IRI described a conscious shift in strategy. “In places like Iraq and Afghanistan where there were very large CSO programs: start with small projects, or even before that training, to see what kind of capacities they need; those that proved capable/promising in the training would be considered for a seed grant and gradually increase support/engagement that way…. The best of our groups did become supported with core funding.” That changed under CEPPS III to narrow and focus support, as follows: “In Iraq and Afghanistan we started to constrict core funding, which was healthy for the CSO who had to moderate expenses—finding better cost/value office space, conference facilities, etc. It created a good dynamic where we started with capacity building and early engagement, then gave seed grants, then became very heavily engaged, and ultimately drawing down that support to be very minimally project supported.” One CEPPS partner noted that there was a “greater level of stability in CEPPS III” which made their work with CSOs easier than under CEPPS II. They described a lot of program modifications and no cost extensions under CEPPS II that made it hard to plan. Uncertainty in the funding streams caused the CEPPS partners to help CSO partners fundraise for small grants from other donors. In contrast, under CEPPS III, they felt that donor coordination was better and that rather than scrambling to find funding for their local partners, they were struggling to effectively program new influxes of money allocated immediately prior to hotly contested elections. 31 Local organizations working together, collaboratively, in networks and coalitions, were also encouraged to a greater extent during CEPPS III. For example, In Cambodia the recent elections-related activities provided an opportunity and political space for new organizations to become involved through networks and coalition efforts. Not only was there an increase over the last six years in the number of civil society organizations active in the election assistance and political processes space in Cambodia, but these organizations have shown increasing coordination over the time period. This coordination and forming of networks was encouraged by CEPPS. Recognizing the value of domestic networks to reach the rural regions, CEPPS partners in Guatemala and Nigeria sought out trusted civil society partners to reach out to smaller organizations to lead and coordinate networks. In Guatemala this was also a means to meet USAID’s goal of engaging more and different CSO partners while still accommodating the partners’ preference for working with known entities. Organizational and Technical Capacity Building A critical question is to what extent direct capacity building shifted between CEPPS II and III. Did the amount of support increase? How was the support divided between technical skills and organizational capacity? Drawing on quantitative evidence from the web surveys, the following tables summarize the share of CSOs that received organizational capacity building, technical capacity building, or neither, and how that share shifted between CEPPS II and CEPPS III. For each type of capacity building, we present summary statistics of the entire sample, and then disaggregated by implementer-supported countries, followed by a region-level analysis. Table 4.2.1. Share of CSOs Receiving Organizational Capacity Building, by Implementer Implementer CEPPS II CEPPS III Change (percentage N points) 9 Mean N Mean Total Sample 56 34% 133 36% 2 IRI 37 22% 87 34% 12 NDI 50 32% 121 35% 3 IFES 47 36% 110 37% 1 Note: Countries are typically supported by more than one CEPPS implementer at a time. Table 4.2.1 reveals increases between CEPPS II and CEPPS III but these were small, with the exception of countries with IRI presence (where capacity building increased from 22% to 34%); however, even this change is not statistically significant.10 9 We note again that the total number of CSOs under CEPPS II is systematically lower than for CEPPS III, due to sampling limitations (as explained in Chapter 3). Most analysis relies on means so as to mitigate this shortcoming. 10 Note that lack of statistical significance does not mean lack of change, only that there wasn’t enough data to detect a change given the extent of change and the variability of the data. 32 Table 4.2.2. Share of CSOs Receiving Organizational Capacity Building, by Region Region CEPPS II CEPPS III Change (percentage points) Pr > F (a) N Mean N Mean Africa 18 50% 48 38% -12 0.376 Asia 9 0% 28 29% 29 0.002** Eurasia 18 39% 34 38% -1 0.964 LAC 7 0% 11 9% 9 0.331 MENA 4 75% 12 67% -8 0.776 Note: (a) F-test to test if the proportions of CSOs are equal between CEPPS II and CEPPS III. When we examine the changes by region, we again see negligible changes in the share of CSOs reporting receiving organizational capacity building, with the only statistically significant change being, again, in Asia, which went from 0% to 29% between CEPPSII and CEPPSIII. This is a surprising result: it is notable that capacity building in organizational strengthening did not increase between II and III, except in Asia, despite the specific emphasis given in the CEPPS III award to “improved … organizational capacities” that are intended to result in “strengthened abilities to compete for, administer, and account” for international funds from donor institutions. Table 4.2.3. Share of CSOs Receiving Technical Capacity Building, by Implementer Country Source of Support CEPPS II CEPPS III Change (percentage points) Pr > F (a) N Mean N Mean Total Sample 56 50% 133 65% 15 0.066* 133 37 54% 87 67% 13 0.198 IRI 50 50% 121 65% 15 0.069* NDI 47 51% 110 67% 16 0.062* Note: Countries are typically being supported by more than one CEPPS implementer at a time. *Statistically significant at the 10-percent level. (a) F-test to test if the proportions of CSOs are equal between CEPPS II and CEPPS III. On the other hand, the percent receiving technical capacity building did increase significantly. Table 4.2.3 reports the share of CSOs receiving technical capacity building, by implementer, between CEPPS II and CEPPS III, and the changes between the two programs are much larger and more significant than for organizational capacity building. Across the total sample, the share of CSOs receiving technical capacity building increased by approximately 15 percentage points, from 50% to 65%, and the change was statistically significant. Both NDI and IFES saw similar increases in the share of CSOs receiving technical capacity building, with statistically significant increases of 15 and 16 percentage points, respectively, between CEPPS II and CEPPS III. Only 33 IRI-supported countries did not see statistically significant changes; although they did show an upward trend, from 54% to 67%. Breaking down technical capacity building by region, the trends are mixed. Latin America/Carib￾bean (LAC) and Eurasia seems to have experienced a decline while Africa, Asia, and the Middle East and North Africa (MENA) saw upward trends in the share of CSOs receiving technical capacity building; only Asia’s changes were statistically significant. Table 4.2.4. Share of CSOs Receiving Technical Capacity Building, by Region Region CEPPS II CEPPS III Change (percentage points) Pr > F (a) N Mean N Mean Africa 18 61% 48 75% 14 0.304 Asia 9 33% 28 79% 46 0.019** Eurasia 18 50% 34 47% -3 0.844 LAC 7 57% 11 36% -21 0.423 MENA 4 25% 12 67% 42 0.168 Note: (a) F-test to test if the proportions of CSOs are equal between CEPPS II and CEPPS III. **Statistically significant at the 5-percent level. Overall, the figures show a downward trend in the number of CSOs who received no capacity building (i.e., neither technical nor organizational). However, this was only statistically significant in NDI-supported countries, who saw the share of CSOs not receiving capacity building drop from 42% to 28%, and, when looking across regions, only in Asia, which showed a decrease from 67% for CEPPS II to 18% for CEPPS III. Average Shares of Contracts and Grants One question is whether there has been any systematic change in the share of contracts and grants between CEPPS II and CEPPS III. The motivation for this research question is the recognition that to be sustainable, it is likely that CSOs will have the capacity to compete for and manage contracts required by international donors. A greater share of contracts, consequently, might be considered a step towards building the skills that contracts may require such as stronger accounting skills, the ability to pre-finance activities, and, in fixed price contracts, the ability to take on the contract risk, for example. We conducted analyses summarizing the average share of grants versus contracts, for all awards, that CSOs reported between the CEPPS II and CEPPS III periods. Globally, there were general downward trends in the share of grants, per CSO, and upward trends in the share of contracts. In other words, CSOs reported receiving a greater share of their awards in the form of contracts, and lesser share in the form of grants, between the CEPPS II and CEPPS III periods. The only statistically significant difference was for IFES-supported countries, which saw shares of contracts per-CSO increase, on average, by 13 percentage points from 28% to 41%, and the average share of grants per CSO decrease from 71% to 59%. Summary statistics for by implementer and region can be found in the Annex in Tables 4.2.10 and 4.2.11, respectively. 34 EQ2.2: Was CEPPS’s process for selecting local partner organizations optimal in the sense that competitive selection processes were balanced against the need for establishing long relationships? Depending on USAID’s objectives, there is a trade-off between maximizing the number of CSOs who participate in CEPPS and focusing on a smaller number to establish longer term relationships and develop skills more deeply. In this section we consider that trade-off, examining the different approaches to selecting CSOs and trends for using competitive selection, and looking at the current mix of long and short term relationships between implementers and CSOs. All AORs thought there was a good balance between competition and selections favoring long term relationships, although only 24% percent found it to be “optimal.” This is in contrast to the implementers, where 41% believed the balance to be “optimal.” The implementers, however, were asked a slightly different question, in which those who did not say “optimal” could identify in which direction it was tilted: 44% of the respondents thought long term relationships were more likely to determine selection, compared to only 8% who thought the process was tilted towards competitive selection. (See Tables 4.2.14 and 4.2.16 in the Annex IX.) Competitive Selection of CSOs Although competitive selection processes were used in some programs from the case study countries, it seems to be far more common for CEPPS partners to work with CSOs with whom they had previous relationships. This emerged from case studies, interviews with implementers at headquarters, and the surveys; the survey of CSOs put the figure at 27%. The rationale for this varied somewhat, though it demonstrated a tendency to value the substantive activities in the sub-grants and subcontracts over a peripheral objective of building the capacity of the CSO sector more broadly. In countries where the civil society sector was less developed CEPPS partners tended to sole source their grants to the most competent and established NGOs. In Tunisia, for example, it was explained that all of the sub-awards were Principal Findings  The three implementers have different approaches to CSO selection.  Survey data and qualitative findings agree that competitive selection is not the primary method for identifying local CSO partners for CEPPS, with competitive selection used less than a third of the time.  Globally, there is a slight decrease in the number of competitive awards for CSOs, but this is only significant for IRI. There is considerable regional variation but only Asia (20% increase) and MENA (35% decrease) show statistically significant substantial changes over the period.  There is no statistically significant evidence of a relationship between being a CEPPS II grantee and participating in a competitive procurement process during CEPPS III. 35 directed to a handful of select CSOs given that the sector was nascent and most other organizations were not experienced in managing large financial awards. In Nigeria the need to work with proven CSOs was paramount for IRI, as it lacked the time required to train less experienced organizations on the technical and administrative aspects of grant management. Both case studies and interviewers with implementers at headquarters found that there was considerable flexibility between using sole source and competitive awards. The process for local partner selection appeared country tailored to meet the demands of the program descriptions as well as maximize efficiency. The different CEPPS partners utilized different methods of selecting sub-grantees, reflecting the variations in their respective objectives, level of experience in country, and country context. In many instances, sub-awards utilized under CEPPS III were sole-source awards, used to fit the country needs and context at the time. In Tunisia, sub￾awards utilized under CEPPS III were sole-source awards, and according to one CEPPS implementing partner in Nigeria, “our working relationships are with established organizations.” These are the organizations that “have shown they have the ability to work on their own. “ CEPPS implementing partners in Georgia have established long-standing relationships with the CSOs it worked with, building their skills and also their organizational capacity. CEPPS has worked with ISFED, for example, since its inception in 2005. KIIs with the implementers at headquarters revealed some general differences across the three implementers. One interviewee from NDI noted that partner selection usually follows informal conversation with USAID and local partners. Sometimes this results in a particular group being named in the PD, but not always. “The closer the date of the election, the more likely there will be a need for an experienced coalition or organization. However, you might still want to engage new members in a network, or exclude some that didn’t work well.” Another NDI representative said: “Process is not so free and open regarding election observation. It depends on which groups are on the ground, are in the best position, and have wherewithal to do it. Other cases may be slightly more competitive. We are looking for the groups that feel most passionate about the issues, and some track record of getting things done. It can be hard to screen for that in an open process. We sometimes use focus groups and identify community priorities and then find groups within those communities who do that work.” In some cases it was deemed necessary to work with the same organizations over a sustained period of time in order to build their skills in a particular area. NDI, for example, typically works exclusively with just one organization, building its capacity to conduct a parallel vote tabulation (PVT). This was observed in Guatemala, Georgia, Tunisia and Cambodia. NDI’s rationale is two-fold: (1) PVTs are highly precise instruments that require considerable experience and expertise; and (2) it is unnecessary, and potentially counter-productive to have more than one organization conducting PVTs in the same electoral cycle.11 Similarly, IFES in Georgia issued repeated grants to an organization with whom it was working on a unique methodology to monitor use of administrative resources in order to consolidate the investments made previously. In each of these instances, the NGOs who benefitted from 11 These conclusions are not necessarily shared universally but were consistently given by NDI to justify its practice of working with a single organization for many years. 36 successive grants demonstrated a greater degree of skill and organizational capacity, sometimes having qualified for direct donor funding already. Responses from several interviewees at IFES suggested that open solicitation is a common option. “Sometimes, when we know there are only a handful of organizations – we do a limited solicitation. Rarely will we sole source. PDs rarely dictate whom to partner with.” “We err on the side of more competition. At least three organizations should respond to the solicitation.” “We go through normal procurement process. We want them to be competitive. In extreme circumstances, where there is a dearth of qualified organizations, we might go sole source.” “In Guinea we kept working with the same CSOs. For initial selection, however, we try to be competitive. “ Competition seems to be less frequent in IRI projects. One interviewee at IRI said “Some of the grants are dictated in the PD; others are ones we have a longer relationship with, or more developed organizations. Lower capacity organizations require more handholding. Rarely do we do a fully open competitive process, though this will change with the upcoming regulations [which will require competition]. YCC in Cambodia is our youth partner that we have been working with for years. They are doing great work both in electoral assistance and political processes. In the proposals we put together we said we’d work with organizations like YCC, but will now have to do a competitive process. We can write an RFP that kind of favors them, but we have yet to see how these new regulations will affect our relationship with our long term partners.” Another interviewee at IRI said that “…occasionally USAID will incorporate the name of a particular group they want us to work with into the PD. More often, our proposal will state that we anticipate working with this or that group, or sometimes less explicit, describing the sector, rather than individual prospective partners. The procurement process can range from ‘we’ve been working with them forever and will continue’ to ‘we will seek new partners in x [field].’ It depends on the program goals and objectives.” Even where CEPPS partners engaged new CSOs they were often hand-selected as the competitive process was not always considered a reliable means of selecting the most capable or appropriate CSO. The reasons varied significantly from country to country. In Nigeria it appeared that CSOs hired consultants to prepare their bids as several competing proposals were substantially similar; consequently the competitive bid process provided few insights into the capacity of the bidding organizations. When CEPPS sought to engage the transgender community in Guatemala, they found that bitterness between some of the leaders prevented them from engaging CSOs from each of the networks, as they refused to work together. Finally, some CSOs became involved in CEPPS programs at their own initiative. In Georgia, for example, IRI was promoting youth engagement in politics, working with the youth arms of political parties in the regions. Youth CSOs who sought to participate in IRI’s activities were allowed as it was consistent with the project goal to promote youth engagement in politics. Another reason for showing preference to organizations that had previously worked with CEPPS related to the desire to promote networking among local NGOs. The competitive process was utilized by IFES and IRI in Georgia for some procurements. IFES issued calls for applications for CSOs representing minorities and underserved segments of the 37 population interested in promoting electoral participation in their constituencies. In these instances IFES found the competitive process gave CSOs the opportunity to present approaches that would be most effective for their respective populations. IRI took a slightly different approach in its competitive procurements. If the task involved was well-defined and outcome-oriented IRI would make an open call for proposals. For example, IRI in Georgia might engage a CSO to make the logistical arrangements for a conference, in accordance with particular specifications. In this way, IRI ensured that it was getting the best value for money by selecting the least cost alternative that met the technical requirements. The differences among the implementers borne out by data from the CSO survey, which shows IRI using competitive selection just 23% of the time, compared to figures of 27% and 30% for NDI and IFES respectively. Across both CEPPS II and CEPPS III the share of CSOs selected competitively for participation in the program has largely remained at just under a third. While there appear to be some variations across regions and implementers, the only significant changes took place in Asia (an increase of 20 percentage points in competition) and the MENA region (a decrease of 35 percentage points).12 Two tables (Tables 4.2.12 and 4.2.13) summarizing the average share of competitive awards that CSOs reported between the CEPPS II and CEPPS III periods can be found in Annex IX. They summarize statistics of the entire sample, and then disaggregated by implementer-supported countries, followed by a region-level analysis. To estimate more rigorously the propensity of CSOs to engage in the competitive procurement process we conducted logit regressions For a further exploration of the advantages of long term association, the evaluation asked whether being a participant in CEPPS II would make it more or less likely to be selected competitively for CEPPS III, with the hypothesis being that competitions are most likely used to select newcomers to the CEPPS program, while previous associations might lead to either longer term partnerships or non￾selection. The results were only statistically significant for IFES, meaning that CSOs who had held an award from IFES in CEPPS II were less likely to be involved in a competitive procurement in CEPPS III. (See further description of the analysis and the full results in Tables 4.2.17 and 4.2.18 in Annex IX.) The Competitive Process 12 Because of the small sample size for CEPPS II in these two regions (10 CSOs in Asia and 5 CSOs in MENA) suggests that it would be prudent not to attach too much significance to these figures. Principal Findings  AORs rated the quality of CSO management skills as the most important criterion in selecting CSOs.  The only statistically significant difference for share of CSOs reporting a fair (over an unfair) procurement process was in the IFES-supported countries. 38 When asked about the importance of selection criteria for CSOs, AORs gave by far the highest importance on the quality of technical skills and management followed by familiarity with the primary actors in the field. Past performance, age, and previous experience with USAID or CEPPS were rated considerably lower. Table 4.2.5. AOR Opinions on the Importance of Selection Criteria Important Not important Quality of CSO technical skills 90.5% 9.5% Quality of CSO management 95.2% 4.8% Past performance 76.2% 23.8% Length of existence 61.9% 38.1% Past experience with CEPPS or USAID 61.9% 38.1% Name recognition 61.9% 38.1% Familiarity with main actors in the field 85.7% 14.3% Cost 57.1% 42.9% Responses to “How important were each of the following in selecting a local CSO?” [N=21; AOR survey] Of all CSOs responding, just over a fifth reported losing a formal bidding competition for participation in CEPPS. Of those 71 CSOs, three-fourths thought the competition was fair, with those disagreeing backing different explanations for the unfair selection. Table 4.2.6. Share of CSOs Reporting a Fair Procurement Process, by Implementer N Percent A CSO with a previous contract or grant was given preference 5 26.3% A less qualified CSO was selected 3 15.8% A politically connected CSO was selected 3 15.8% The scope of work or the rules of procurement were not clear 2 10.5% Other, please specify: 5 26.3% Skipped 1 5.3% Responses to “Why do you feel that the selection process was not fair? Please select the statement below that most closely matches your view” [N=19; AOR survey] 39 EQ2.3: To what degree did receiving a CEPPS grant in 2009 change the probability of receiving an award in 2014, all else held equal? We collected AOR opinions on whether the selected CSOs during CEPPS III had been actively engaged with CEPPS programming in the past, and 71.4% of responding AORs responded in the affirmative. To determine if there was a statistically significant relationship between a CSO being active with CEPPS, and a CSO being selected in future implementation rounds, we conducted another series of logit regressions. The results show a statistically significant positive relationship between being a CEPPS II sub￾grantee and a CEPPS III sub-grantee. The results are robust to all model specifications, with and without country fixed-effects. When looking at individual implementers, CSOs in both NDI￾and IRI-supported countries showed positive relationships between being a CEPPS II CSO and a CEPPS III CSO. The regression tables and more detailed analysis can be found in Annex IX under EQ 2.3, and Tables 4.2.20 and 4.2.21. 4.3 EVALUATION QUESTION 3: THE NATURE OF CEPPS III PARTNERSHIPS EQ3.1: To what degree did CEPPS involve local partners in the design, awards management, M&E processes, and reporting of programs with which they were affiliated? We asked CSOs if the implementer decided which activities the CSO would undertake. We use this question as a proxy for the degree to which CEPPS involved its local partners in various programmatic processes. If a CSO answered that the implementer did not make programmatic decisions for them, it would indicate that there was a collaborative process Quantitative Findings  AOR opinions and CSO experience are aligned and indicate evidence that having a CEPPS II grant increased the probability that a CSO received a CEPPS III grant. Quantitative Findings  14% of CEPPS III CSOs reported that the implementer made the decisions for the CSOs on programmatic activities.  Asia is the only region in which the share of CSOs actively involved in all aspects of program involvement is lower (and this difference in all cases is statistically significant)  The programmatic area with the largest share of CSOs being involved “to a great extent” was for Reporting, at 44% (though not statistically significant). 40 between the implementer and the CSO. High-level summary statistics for this question can be found in Table 4.3.1. Table 4.3.1. Share of CSOs Reporting Implementer Decided on Programmatic Activities Implementer decided which activities CSO would undertake (a) Share implementer Yes No decided (b) Total Sample 22 138 0.14** Africa 9 45 0.17** Asia 9 33 0.21** Eurasia 2 35 0.05 LAC 1 11 0.08 MENA 1 14 0.07 (a) Reported as number of CSOs per region. (b) Asterisks indicate that the share is statistically significant for all three hypothesis tests at the 5% (**) level. It appears that overall, CSOs play an active role in programming: across the entire sample, only 14% of respondent CSOs reported that the implementer made the decisions for the CSOs on programmatic activities. 13 Looking at the regional breakdown, the only breakouts that are statistically significant are for Africa and Asia, which have higher shares – 17% and 21% respectively – of implementers making the decisions. We conducted additional analyses that further break down these numbers by specific activities – namely design, awards management, M&E processes, and reporting. In this case, the numbers report the share of CSOs responding that IRI, NDI, and/or IFES involved them “to a great extent” in the implementation of the various program activities. Table 4.3.3 provides the results for the full sample; detailed breakouts by region can be found in Annex IX (in Table 4.3.3a). Table 4.3.2. CSO Involvement in Award Design and Management Implementer involved CSO in program activities to a high extent Share where CSO is involved to a high extent Response Option Yes No Program Design 66 93 0.42** Awards Management 52 105 0.33** M&E Processes 51 102 0.33** Reporting 68 88 0.44 Note: Asterisks indicate that the share is statistically significant for all three hypothesis tests at the 5% (**) level. 13 The hypothesis tests indicate that the share is statistically significant at the 5% level, for all hypotheses that the share does not equal zero, 0.5, or one. 41 Across the entire sample, 42% of CSOs responded to being involved to a great extent with the design of programmatic activities, while a third were involved with awards management and M&E processes. Involvement in reporting was higher, but not statistically significant Asia is the only region whose shares are significantly different from the global figure across all the categories, and in every case the level of involvement is strikingly lower. For CSO involvement in the reporting requirements of programs, 44% of the entire sample reporting being involved to a great extent, but this failed the hypothesis test that the share was not equal to 0.5, though it passed tests that it was not equal to zero or one. We conducted a similar analysis by implementer. NDI had the highest share of CSOs reporting collaboration on all four administrative tasks. IFES had the second-highest shares for all tasks. However, the F-tests to test if the shares are equal across all implementers show that these differences are not statistically significant. We cannot conclude with certainty that the differences in shares shown above were not due to chance. The corresponding table for these results can be found in Annex IX under Table 4.3.4. Involvement of CSOs in Design and Management Processes Respondents from IFES headquarters mostly agreed that CSOs are often very involved in management. One caveat is that the opportunity to involve local CSOs in design is constrained by the fact that the time allowed to respond to Program Descriptions ifs very short. One respondents said “We never have much time to respond to the PD. This keeps local partners out of the design process, unless USAID has done that already.” Nevertheless another staff member said they are extensively involved. “We engaged in direct consultation and discussions. We looked at documents drafted by local organizations.” An IRI respondent described the design process as hampering input. “USAID comes out with a PD that we have not previously viewed or commented upon. Perhaps have had general discussions, briefings, but not actually seen a draft. This is the most common, but not for lack of CEPPS trying to have more input. In these cases we do not have a lot of say in the PD, and hence our partners also do not. “ If there was an opportunity to provide insight or advice to USAID, then it would reflect conversations with our local partners ‒ particularly where we have a field office. This would include civil society as well as other stakeholders ‒ political parties, media, candidates, officials, etc. We are constantly having discussions ‒ looking for lessons learned, what worked, what are the needs ‒ it is part of our constant assess/reassess. The situation changes so much; very little is stagnant in the nature of politics and the area of our work.” After the PD is issued however, “We do engage the CSOs in developing project activities, and in thinking about who we are going to work with. Not all CEPPS projects work with CSOs. But we do involve them in defining the activities whether or not input was solicited in the PD itself.” USAID Guatemala’s Program Descriptions for CEPPS were informed by two studies that incorporated feedback from civil society. The first was an internal evaluation of a previous CEPPS project, and the second was an external study on political violence. CEPPS partners also incorporated feedback from civil society in its project design, to varying degrees. Similarly, at 42 the end of its first project under CEPPS III, one CEPPS implementer organized a series of discussions with partner groups on lessons learned/best practices in supporting domestic election observation projects; presumably the insights from these discussions fed into their project proposal in response to the subsequent program description. Many CSOs described it as a “consultative process”, or “continual coaching” whereby their respective sub-grants were accompanied by on-going coaching or mentoring by the CEPPS implementers. Even when the project idea was not theirs to begin with, and came from the CEPPS implementers, when it came to design and implementation, by and large, the CEPPS implementers provided the CSOs with room to initiate, create, and run the project with minimal oversight. Case study evidence confirms that the CEPPS local partner CSOs are often deeply involved in the design, awards management, M&E processes, and reporting of program with which they are affiliated. In Tunisia for example, according to the CEPPS implementers, local partner civil￾society organizations were substantially involved in every step of the program. According to CEPPS implementers there, they were able to coach CSOs in every aspect of organizational development, as well as program implementation and technical skills-building. Program activities included: voter education; training-of-trainers (to monitor polling stations); monitoring of polling stations; monitoring of campaign finance; and conducting a PVT. CSOs seem consistently involved in the monitoring and evaluation (M&E) practices, with 33% “involved to a high extent.” According to several IFES respondents: “We insist partners have an M&E plan. This requires both quantitative and qualitative data collection.” “We help them come up with advocacy program indicators.… We are doing this higher level indicator collection in Bangladesh where we want to know what the impacts are at the community level….We also work with the academic community on M&E. There have been some growing pains regarding M&E. We are trying to acclimate CSOs to need to do it. We let partners know what information to collect and when.” But another noted that “we involve partners more in data collection, not the analytical side.” EQ3.2: Did CEPPS incorporate CSO organizational capacity building as an integral part of collaboration or undertake it as a separate task? Principal Findings  Implementers were more likely than AORs to report that the purpose of grants was capacity-building.  According to implementers a substantial part of program activities involved capacity-building; and it was mostly integrated into their programs and not on a parallel track.  51% of CSOs found capacity building received to be useful, and 78% said that some or most of practices learned have been adopted and used.  Though we observe downward trends, there was no statistically significant trend – either globally, by implementer, or by region – in CSOs receiving support through joint work between CEPPS II and CEPPS III. 43 Form of Capacity Building Efforts Tables 4.3.3 through 4.3.8 present high-level summary statistics from the Implementer and AOR surveys. Table 4.3.3. Purpose of Grants Response Implementer AOR N Percent N Percent To support their provision of desired activities 45 66.2% 16 72.7% To build CSO organizational capacity 57 83.8% 12 54.5% To build CSO technical capacity 59 86.8% 16 72.7% Other, specify 12 17.6% 4 18.2% Do not know 0 0.0% 0 0.0% Total 68 100% 22 100% Responses to “What was the purpose of giving grants to local CSOs during CEPPS III? (Please select all that apply.)” Table 4.3.3 presents side-by-side comparisons of implementers’ and AORs’ opinions on the purpose of giving grants to local CSOs during CEPPS III. Implementers were more likely than AORs to respond that the grants were intended to build CSO organizational or technical capacity. According to most implementers (83.8%), grants were given to build CSO organizational capacity, while only 54.5% of AORs agreed. For technical capacity, the difference persists but is smaller – 83.8% versus 72.7%. Table 4.3.4 and 4.3.5 show the extent to which implementers believed the programs involved building capacity of local CSOs. Most implementers (78.2%) responded that programs had capacity-building as an explicit part of the program, while 66.4% believed the programs involved a substantial- or large-portion of capacity building. Only 27.7% of implementers’ programs were rated as having a small part capacity-building, and 2.5% of the programs included zero capacity building, according to implementers. Topics and Effectiveness of Capacity Building With respect to the type of capacity building needed, we asked the CSOs a series of questions about the types of assistance needed and received during the CEPPS II and CEPPS III periods to explore whether the training provided met the needs and whether there were some shifts over time. Fundraising is consistently identified as the topic most needed, and seems to have been offered less frequently under CEPPS III. Also interesting is that perception of CSOs about their need for support in financial management and organizational development is substantially lower than the implicit views of implementers. 44 Table 4.3.4. Type of Organizational Capacity Building Most Needed and Received under CEPPS II and CEPPS III, according to CSOs Topics in Organizational Capacity Building 2005-2008 2009-Present Needed Received Needed Received N Mean N14 Mean N Mean N Mean Financial management 125 52% 24 80% 124 40% 41 71% Organizational management 130 54% 23 77% 111 36% 48 83% Fundraising 192 80% 23 77% 245 79% 34 59% Database management 137 57% 21 70% 165 53% 37 64% Internal communication 76 32% 20 67% 71 23% 37 64% Responses to “What was the purpose of giving grants to local CSOs during CEPPS III? (Please select all that apply.)” CSOs who had received support both in the CEPPS II period and the CEPPS III period were asked directly in which period they believed capacity building needs were better addressed, and the CEPPS III period was the clear winner (Table 4.3.5). When asked about the effectiveness of the capacity building, the responses were quite positive. More than half of those responding rated the capacity building as very useful, with another 26% rating it as “somewhat useful. Only 3% thought it was of little or no use. Further, almost 80% of respondents reported having adopted and used some or most of the new practices introduced by NDI, IRI, or IFES. Approach to Capacity Building In countries such as Zambia and Nigeria, trainings and workshops have been the means of building CSO capacity. In Cambodia, responding to a recommendation for an evaluation, in￾country CEPPS implementers developed and organized trainings and workshops in financial management, program monitoring and evaluation; fundraising, proposal writing and proposal development. In Guatemala, skills training on electoral observation and vote verification methodology, as well as the workshops on strategic planning, project management, communication, outreach, and monitoring and evaluation, all help prepare local organizations to apply for and implement larger and more complicated awards. In Tunisia, fundraising, proposal development, strategic planning and advocacy and lobbying workshops were conducted in an effort to help organizations improve their capacities. M&E workshops were also conducted to ensure organizations were applying proper reporting requirements. 14 The number of respondents to the questions on receiving training is necessarily smaller, since only those who received capacity building responded to the question. 45 Table 4.3.5. CSO Opinions on Effectiveness of Capacity Building N Percent Period in which capacity building needs were better addressed Better addressed in the period 2005-2008 13 19% Addressed about the same in both periods 11 16% Better addressed during the period 2009-Present 24 34% Do not know 5 7% No response 16 23% Extent to which capacity-building was useful Not at all useful 3 2% Not very useful 1 1% Somewhat useful 49 26% Very useful 96 51% Do not know 9 5% No response 27 14% Adoption rates of IRI, NDI, and/or IFES practices No practice has been adopted and used 16 9% Very few practices have been adopted and used 6 3% Some practices have been adopted and used 58 31% Most practices introduced have been adopted and used 88 47% Do not know 17 9% Most implementers (73.9%) reported integrating organizational capacity-building activities into their collaborations on elections and political processes, while only 15.1% of implementers reported that the activities occurred on a separate parallel track. Table 4.3.6. Building Capacity as an Explicit Part of the Program, Implementer Survey Response N Percent Yes 93 78.2% No 25 21.0% Skipped 1 0.8% Responses to “Was building capacity of local CSOs an explicit part of this program?” [N=119; Implementer survey] 46 Table 4.3.7. Degree that Program Involved Building CSO Capacity, Implementer Survey Response N Percent A large part of program activities 25 21.0% A substantial part of program activities 54 45.4% A small part of program activities 33 27.7% No capacity building of CSOs 3 2.5% Do not know 4 3.4% Responses to “How much of this program involved building capacity of local CSOs?” [N=119; Implementer survey] Table 4.3.8. Organizational Capacity-building Integrated vs. Parallel Track, Implementer Survey Response N Percent Integrated capacity building 88 73.9% Parallel track 18 15.1% Do not know 13 10.9% Responses to “Did [implementer] integrate organizational capacity-building activities into your collaborations on elections and political processes, or did the activities occur on a separate parallel track?” [N=119; Implementer survey] To see whether there was a fundamental shift in the mode for delivery of capacity building efforts between CEPPS II and CEPPS III, we looked at the share of CSOs reporting that they received capacity building primarily through joint work with the implementer – which in general seems to be the most common form of capacity building We present summary statistics of the entire sample first for organizational capacity building, followed by the figures disaggregated by region. Table 4.3.9 presents the share of CSOs receiving capacity building through joint work with the implementer between CEPPS II and CEPPS III. We see downward trends for the total (from 70% to 57%) and for most regions, but these trends are not statistically significant except for MENA and Asia; but given the very small sample sizes, these results can only be taken as indicative. 47 Table 4.3.9. Share of CSOs receiving Organizational Capacity Building through Joint Work, Total and by Region Region Share of CSOs Receiving Joint Work - Organizational Change Pr > F (a) CEPPS II CEPPS III N Mean N Mean Total 20 0.7 47 0.57 -0.13 0.330 Africa 9 0.67 18 0.61 -0.06 0.788 Asia 1 1.00 7 0.29 -0.71** 0.006 Eurasia 7 0.57 14 0.79 0.22 0.366 LAC 0 0.00 1 0.00 0.00 NA MENA 3 1.00 7 0.43 -0.57** 0.020 Note: (a) F-test to test if the proportions of CSOs are equal between CEPPS II and CEPPS III With respect to technical capacity-building, the use of joint work stayed relatively level between CEPPS II and CEPPS III. For the total sample, the share of CSOs increased by only 2 percentage points between the two periods. The implementer-specific countries, the trends are similar. For the total sample and for each of the implementer-specific countries, the F-tests indicated that the differences in proportions of CSOs between CEPPS II and CEPPS III were not statistically significant. Table 4.3.10. Share of CSOs receiving Technical Capacity Building through Joint Work, Total and by Region Region Share of CSOs Receiving Joint Work - Technical CEPPS II CEPPS III Change Pr > F (a) N Mean N Mean Total 29 0.52 74 0.54 0.02 0.83 Africa 11 0.64 30 0.57 -0.07 0.697 Asia 4 0.5 19 0.58 0.08 0.802 Eurasia 9 0.56 13 0.62 0.06 0.793 LAC 4 0.25 4 0.75 0.5 NA MENA 1 0 8 0.13 0.13 NA Note: (a) F-test to test if the proportions of CSOs are equal between CEPPS II and CEPPS III Asia and Eurasia had slight upwards trends, while Africa had a slight downward trend, dropping seven percentage-points. The LAC region had the largest increase, with shares of CSOs increasing by 50 percentage points between CEPPS II and CEPPS III. MENA also had an increase of 13 percentage points. However, the results of the F-tests for equivalence of 48 proportions between CEPPS II and CEPPS III indicate that none of the changes by region are statistically significant. While there were no statistically significant changes across the programs, it is interesting to observe the diversity by region in the current CEPPS III program, both in supporting technical and especially in organizational capacity building We conducted similar tests for the different implementer-country groups, but in those cases not only were there no statistically significant change, but there was also extremely little variation across the three implementers. Q3.3: To what extent were best practices adopted by local CSOs? To help assess the effectiveness of capacity building, the evaluation looked at adoption of a few specific “best practices” by partner CSOs. We identified two specific practices, one technical (gender analysis) and one organizational (keeping timesheets). Across the entire sample, 41% of CSOs responded that they incorporated gender analysis into programming “to a great extent.” This share was found to be statistically significant at the 5% level, passing all three hypothesis tests that the share of CSOs responding “to a great extent” was not equal to zero, 0.5, or one. Africa had the highest level of adoption, with 50% of CSOs responding that they incorporated gender analysis “to a great extent.” The region with the lowest share was Asia. The only region to pass all three hypothesis tests for gender analysis was Asia, with 26% of CSOs responding that they incorporated gender analysis into programming “to a great extent.” Africa, Eurasia, Latin America/Caribbean, and MENA all failed the hypothesis test that the share was not equal to 0.5, but passed the tests that the share was not equal to zero or one. However, the survey also asked whether CSOs normally incorporated gender analysis into their programming, prior to CEPPS collaboration, and 63% of the sample responded in the affirmative. This share was significant at the 5% level and passed all three hypothesis tests that share was not equal to zero, 0.5, or one. Of all regions, Africa had the highest percentage of CSOs reporting that they normally incorporated gender analysis, with 69%, and this also passed all three hypothesis tests. Asia, Eurasia, Latin America/Caribbean, and MENA passed the hypothesis tests that the shares were not equal to zero or one, but failed the test that the Quantitative Findings  A high percentage of CSOs report that they have adopted best practices.  Of the three best-practices presented, timesheet systems were used by the highest share of CSOs (72%).  Africa consistently led the other regions in adoption of the three best-practices. 49 share was no equal to 0.5. Of these, Latin America/Caribbean had the lowest percentage of CSOs reporting normal incorporation of gender analysis, with 55%. Finally, we conducted additional regression analysis to determine if CEPPS training on gender analysis had had some effect, and no statistically significant relationship was found (although the relationship was positive). See Annex IX for additional analysis and full results in Table 4.3.10. Adopting Timesheets Almost three-fourths of the CSOs sampled (72%) reported using a timesheet system, and this was statistically significant at the 5% level, passing all three hypothesis tests that the share did not equal zero, 0.5, or one. All regions except for Latin America/Caribbean also passed all three hypothesis tests at the 5% level, though Latin America passed the tests testing that the share was equal to zero or one. Asia had the highest percentage of CSOs using a timesheet system, with 78%, followed by Africa with 75%. Latin America had by far the lowest percentage, with only 55% of CSOs reporting using a timesheet system. 4.4 EVALUATION QUESTION 4: FACTORS LEADING TO CEPPS III SUCCESS EQ4.1: Which factors contribute to the successful elevation of local organizations into sustainable providers of TA? When asked about the factors that increase the sustainability of local CSOs, CEPPS interviewees listed some of the factors as “A partner should have a clear mission statement and a vision of the future” and “Financial accountability first and foremost; technical skills are very much secondary.” Another important factor mentioned was “a consistent, predictable management structure” – as opposed to CSOs built around one individual, even when very accountable, and transparent, because if that person leaves there is uncertainty. A CSO can have a strong leader and a CSO built around them, but there should be processes so that if that person leaves there is no uncertainty about how to replace them. The case studies highlighted several other important factors: Principal Findings  In interviews CEPPS implementers cited several factors as being important for CSO sustainability, including having a mission statement and being “committed” to the mission; financial accountability; a consistent management structure; and specialization.  Positive relationship between sustainability and several CSO characteristics: (1) uses consistent donor-approved overhead rate; (2) more local awards than regional average; (3) more international awards than regional average; (4) large number of organizational good practices. 50 Specialization. In most of the case studies, having a specific area of acknowledged expertise was cited as an important attribute of sustainable CSOs. As one IFES representative in Nigeria put it: “Too many CSOs are jacks of all trades and masters of none.” Specialization was also cited by the Zambian National Women’s Lobby, which has existed since 1991 as an advocate for women in Zambia, as important to establishing a relationship with donors. Development of specific expertise was also mentioned in Tunisia, where specialization was associated with professionalism. In Georgia, both GYLA and TI were cited as organizations that developed unique skill sets, increasing their marketability to donors. Although in Cambodia, some stakeholders praised partners that “go beyond their comfortable base of knowledge and experience,” most key informants in the six case study countries stressed the importance of specialization. Dedication to the Mission. Some of the case studies, such as Tunisia, highlighted the importance of ensuring that CSO staff is committed to the mission. This is related to comments in Cambodia that cited staff willing to work as volunteers as the sign of a committed organization. In Zambia, church members of the CCMG coalition were found to have a greater sense of accountability and dedication than in a similar, but more diverse coalition. NDI has emphasized the increasing problem of civil society becoming “monetized”, meaning that some groups and their employees see their C-SO primarily as source of income, rather than a deeply￾held cause. Strong Governance. In Zambia, NDI criticized what they referred to as “briefcase organizations,” groups with minimal administrative capability. The importance of groups having demonstrated financial and management abilities was also emphasized in Nigeria and Cambodia. Organizations that have strong governance, are also considered more useful members of representative coalitions. Avoidance of “Founder’s Syndrome.” Several CSOs in Zambia referred to management problems caused by over-reliance on one person – a phenomena known as “founder’s syndrome.” If anything happened to that person, or if a donor felt funds were being mismanaged, donors would withdraw funding. In Nigeria, most CSOs are driven by one or two key figures, who become the driving factor as to whether the CSO has the leadership, innovation, and integrity necessary for the CSO to be sustainable. Positive Enabling Environment. The relatively free and open societies in Zambia, Nigeria and Tunisia were cited in interviews as important to the sustainability of CSO. Quantitative Analysis of Sustainability Quantitatively measuring sustainability is challenging. In our questionnaire, we do not ask direct questions about sustainability, nor is it directly observable. Moreover, in selecting variables to measure sustainability, one must be careful not to confound inputs to sustainability with sustainability itself. This may be hard or even counterproductive, as it is difficult to discern the difference between an ex ante sign of sustainability and an input or requirement for sustainability. To address the above problem, we constructed two separate sustainability indexes using standardized versions of the following four variables from the data: (i) change in the number of 51 full-time staff from CEPPS II to CEPPS III; (ii) change in the number of grants/contracts from local clients increased from CEPPS II to CEPPS III; (iii) change in the number of grants/contracts from international clients increased from CEPPS II to CEPPS III; and (iv) age of CSO in 2014. The first index is calculated using an equi-weighted average of the four variables, while the second index uses the first principal component of the four variables.15 Second, we identified a set of potential “inputs” to sustainability such as significant experience (number of local and international awards) and good organizational practices (such as established overhead rates and timekeeping systems). To measure the impacts of those inputs on the sustainability index, we ran a series of regressions, with the outcome variables being one of the two sustainability indexes. The explanatory variable of interest for each regression was one of a number of inputs to sustainability that we measured in our survey.16 We also included a vector of control variables, to control for characteristics that could affect the sustainability of a CSO.17 The full regression tables can be found in Annex VIII under Tables 4.4.1a and 4.4.1b. As the tables show, the four sustainability inputs that show a statistically significant relationship to our sustainability indices are: (1) whether or not the CSO uses consistent donor-approved overhead rates; (2) the dummy for whether the number of local awards for the CSO is above the regional average; (3) whether the number of international awards is above the regional average for the CSO; and (4) whether the CSO has a greater number of organizational items than the regional average.18 For all four inputs, there is a positive relationship between the input and the sustainability index. The strongest statistical evidence is for the “number of international awards” indicator, which is highly significant at the 1% level for both the weighted￾average and the principal component indices. The donor-approved overhead rate dummy variable is significant at the 5% level only for the principal component indicator, and is not significant for the weighted-average index. The indicator on the number of local awards is only significant for the weighted-average indicator but not for the principal components. Finally, the number of organizational items indicator has the weakest evidence for sustainability impact, being significant at the 10% level only for the weighted-average indicator. Running the 15 The two indices have a correlation coefficient of 0.8574, with a p value of 0.0000. The “first principal component” is a factor derived (using principal component analysis) to parsimoniously encapsulate the largest possible range of variation encompassed in all the variables. 16 The inputs to sustainability found in our survey are standardized versions of the following variables: uses consistent donor-approved overhead rate; uses a timesheet system; received non-CEPPS international funding for latest election; has adopted CEPPS's capacity-building approaches; needs help with financial management; needs help with fundraising; number of local awards above regional average; number of international awards above regional average; engaged in more activities than regional average; number of organizational items above regional average; CSO has experience abroad. 17 The covariates are the following: number of CSOs (per country); number of NGOs (per country); GDP per￾capita; population (2009); IRI-supported country dummy; NDI-supported country dummy; IFES-supported country dummy; dummy variables for each region in the data. 18 See note (b) of Table 4.4.1 for the list of organizational items.19 Looking at the complete regression results in Annex VIII, we find that none of the sustainability inputs, when interacted with a CEPPS3 grantee dummy, are statistically significant. 52 regressions with and without the implementer-specific dummy variables had negligible impact on the results. EQ4.2: What are the characteristics or prerequisites of the local and regional organizations receiving CEPPS III support which most align with potential for success? We asked both CSOs and implementers to select up to three critical factors for a CSO’s success, as presented in Table 4.4.1. Table 4.4.1. Implementer and CSO Opinions on Factors to CSO Success Response Implementer CSO N Percent N Percent Collaborations with other CSOs 53 44.9% 187 66.5% Absence of government intimidation 63 53.4% 68 24.2% Strong admin. Systems 51 43.2% 81 28.8% Expertise of senior staff 32 27.1% 87 31.0% Adequate funding 45 38.1% 172 61.2% Capacity building from int’l orgs 69 58.5% 115 40.9% Access to skilled staff 13 11.0% 70 24.9% Other(1) 9 7.6% 15 5.3% Other(2) 1 0.8% 1 0.4% Total 118 100% 281 100% Responses to “Please select up to 3 critical factors to CSO success in [country].” Both implementers and CSOs included inter-CSO collaboration and capacity building assistance from international donors in the top three factors, but the difference in views is perhaps the most striking result. A review of the top four factors selected by each of the groups (see shaded boxes in the table) reveals some interesting differences. While capacity building is Quantitative Findings  Main factors for success according to implementer and CSO opinions were funding, CSO collaboration, and capacity-building.  Indicators on strengths of administrative systems and workforce talent were all 30% or less according to implementer and CSO opinions.  Though there are consistent statistically significant relationships between sus￾tainability inputs and our sustainability indices, being a CEPPS III CSO did not affect the strength of the relationship. 53 mentioned by both groups (albeit top ranked for implementers and in third place for CSOs), only 28.8% of CSOs think strong administrative systems are important, compared to 43.2% of implementers. Absence of government intimidation was mentioned considerably more frequently by implementers than CSOs. Perhaps most surprisingly, adequate funding was mentioned by more than 60% of CSOs but by less than 40% of implementers. Coalitions and Networks. Interviews confirmed survey data from implementers and CSOs on the important role of CSO collaboration. Establishment of CSO coalitions or networks was a prominent feature in several countries, including Cambodia, where it was noted that coalitions can allow CSOs to reach more areas of the country. In Guatemala the use of networks – the linking together of organizations with similar missions or those engaged in similar activities – was used effectively. In that country, strong CSO networks have been developed for electoral monitoring, people with disabilities, and women’s rights, all of which have been nurtured through CEPPS III programming. It remains to be seen, however, whether these Guatemalan networks ultimately yield more organizations with capacity to manage and implement donor funds, or merely strengthen the lead organization. The experience in Zambia points out that PVT coalitions can vary widely in the level of cohesion and effectiveness, depending on the structure of the alliance. Nigeria provided another example of the problems created by weak coalitions, where the Joint National Association of Persons with Disabilities (JONAPWD) coalition fell apart because of squabbles among its member organizations. For a more statistically rigorous analysis on the drivers of CSO sustainability for CEPPS III grantees, we ran modified versions of the above regressions from question 4.1. In the new regressions, for each sustainability input we also added an interaction term, where we interacted the sustainability input with a dummy variable which equaled one if the CSO had received a CEPPS III grant. Results indicate the significance of the sustainability inputs goes away when interacted with the CEPPS III grantee flag. This indicates that the sustainability inputs, though statistically significant throughout the entire sample, were not necessarily significant for only the CEPPS III grantees.19 For many of the variables we see the coefficient turn negative and insignificant, as compared to the standalone sustainability input. The regression table can be found in the Appendix under Table 4.4.3. 19 Looking at the complete regression results in Annex VIII, we find that none of the sustainability inputs, when interacted with a CEPPS3 grantee dummy, are statistically significant. 54 4.5 EVALUATION QUESTION 5: ASSESSING CSO SUSTAINABILITY AND FINANCIAL MANAGEMENT DUE TO CEPPS III ASSISTANCE EQ5. Assessing CSO Sustainability and Financial Management due to CEPPS III Assistance EQ5.1: What CEPPS efforts have supported the opportunities of local organizations for becoming direct recipients of local funding? Level of CEPPS III Support to CSOs to Received Direct Funding According to many case study interviews, helping local CSOs become direct recipients of donor funding has not been a stated goal for most Missions. When asked directly about direct funding for local CSOs, some Missions expressed serious reservations. USAID Zambia sees the potential for direct funding as a bigger management burden than would be ideal for a small Mission. We evaluated, per region and across the entire sample, the level of effort that CEPPS III implementers made to support CSOs in becoming direct recipients of donor funding. The first columns in Table 4.5.1 summarize the percent of CSOs where CEPPS III assistance led to obtaining new funds. Across the entire sample, 31% of CSOs claimed that CEPPS III assistance led to obtaining new funds from other donors. Regionally, Eurasia had the highest share of Principal Findings  Almost a third of implementers indicated that helping CSOs become recipients of direct funding from other donors was a program objective.  The level of effort implementers expended in helping CSOs obtain funds had a statistically significant relationship to whether or not those CSOs obtained funds. An implementer making efforts “to a great extent” was associated with a 66 percentage-point increase in the likelihood of a CSO obtaining funding from outside sources.  Whether implementers involved the CSO in the project design, awards management, or M&E processes of their projects together “to a great extent,” all had positive and statistically significant relationships to whether or not that CSO was able to obtain funding from another donor.  A CEPPS III implementer introducing a CSO to a funder, helping the CSO prepare materials for gaining funds, and identifying activities for the CSO to find new funding were all positively associated with an increase in the likelihood of a CSO obtaining funds from another donor – and with very strong statistical significance. 55 CSOs receiving new funds, with 48% of responding CSOs receiving new funds. MENA had the second-highest percentage, with 42%.20 Table 4.5.1. CSO Experience with Sustainability and Financial Management Region Share of CSOs where: CEPPS III assistance led to obtaining new funds from other donors CEPPS III implementer made efforts "to a great extent" to help CSO receive funds CEPPS III implementer made efforts "to some extent" to help CSO receive funds N Percent N Percent N Percent Total 130 31% 134 19% 134 28% Africa 40 28% 42 12% 42 33% Asia 36 19% 37 22% 37 19% Eurasia 33 48% 34 21% 34 32% LAC 9 11% 10 1% 10 10% MENA 12 42% 11 36% 11 36% Prob > F (a) 0.043 0.432 0.224 Note: (a) F test to check if the proportions, per column, are equal across all regions. In the final four columns, Table 4.5.1 also summarizes the extent to which CEPPS III implementers made efforts to help CSOs receive funds from another donor. Only 19% of CSO respondents indicated that CEPPS III implementers made efforts “to a great extent” to help them obtain funding from other donors, while 28% indicated that implementers helped “to some extent.” There is variation between regions for both of these indicators, however, both F-tests to test equivalence of shares across regions indicate that the differences are not statistically significant. The web survey asked implementers directly whether or not helping local CSOs to become direct recipients of donor funding was an objective of their program. Almost 37% respondents indicated that helping CSOs become direct funding recipients was a project objective, although only a third those said it was an explicit objective (see Table 4.5.2). 20 We ran an F-test to test if the differences in shares were equal across all regions. For the share of CSOs reporting obtaining new funds from other donors, we were able to reject the hypothesis that the shares were equal across all regions. 56 Table 4.5.2. Helping CSOs to Obtain Donor Funding as a CEPPS Objective Response N Percent Yes, it was an explicit objective 14 12.3% It was not explicit, but it was one of the project's objectives 28 24.6% It was not a major objective, but it was viewed as a positive parallel benefit 42 36.8% No, it was not an objective at all 29 25.4% Skipped 1 0.9% Responses to “Was helping local CSOs in [country] to become direct recipients of donor funding an objective for your program?” [N=114; Implementer survey] Whether or not the CEPPS PD was explicit in requiring such assistance to CSOs, KIIs with headquarters CEPPS staff yielded some examples of explicit support to CSOs to become direct recipients. “In Burundi, we are working with a faith-based CSO (CJP) that could become a direct recipient. This group has staff in each of 17 provinces… But even in this case preparation for direct funding is not one of the objectives.” “We did do some institutional capacity building regarding financial management in Iraq – workshops on grants focused on compliance. We created a manual. Most field offices do one-one-one mentoring. It is not a larger goal. There is starting to be more of a push from USAID.” Whether intentional or not, capacity building results have indirectly moved organizations toward this goal. In Georgia, most of the local organizations interviewed credited each of the CEPPS partners with helping them prepare for and find opportunities to become direct recipients of donor funding. In particular, NDI groomed ISFED in Georgia over the course of many years, on both skills capacity and organizational development. In Guatemala, although this was never an explicit part of CEPPS programming, CEPPS partners have helped prepare key local partners to become direct recipients of donor funding. CEPPS has helped local organizations learn to network as well as to engage Guatemalan governmental officials. It seems that established CSOs have much better chances of succeeding. Many of the Cambodian CSOs involved with CEPPS III are older, experienced, organizations with an established track record of receiving and managing funding from multiple sources, such as Australian Aid and the Australian Embassy, DFID, the NED, OXFAM, the UNDP. In Zambia, CEPPS has favored an experienced CSO, Caritas, to implement its most recent PVT. However, developing only a few strong organizations may be preventing the development of others. Balancing the desire to develop the skills and organizational capacity of CSO partners against the benefits of a diverse civil society was a challenge in most of the countries examined. CSOs, and some implementers, believe it is important to support more CSO overhead costs through sub-grants. Sub-grants, usually cover program costs, but very little in the way of overhead or administrative costs. CSOs in Zambia, and, especially in Nigeria, advocated for 57 allowing sub-grants to pay more of these costs, given the importance of administrative staff and rent to long-term sustainability. NDI Zambia was sympathetic to this view as well. Some implementers identified ways in which work with CEPPS is good preparation for sustainable service providers with direct funding from donors. One example was that some of the CSOs learn skills that can be useful outside the borders of their home country. For example, for the select organizations charged with conducting PVTs, performing parallel vote tabulations is a skill that can be transferred from country to country. In Tunisia the PVT conducted by Mourakiboun was the first of its kind in the MENA region. Mourakiboun is taking steps to share their skills and experiences outside of Tunisia, traveling to Cote d’Ivoire and Mauritania to help train counterparts. In Zambia, NDI brought in representatives from other African countries to help train Caritas, the CSO conducting a PVT there. The sub-grant process has also in many cases enabled CSOs to learn first-hand the kind of budgeting and reporting that would be required for an organization receiving direct funding from USAID. In Zambia, NDI pointed out that the financial reporting requirements through sub-grants are as strict as what USAID requires. In Cambodia, CEPPS has provided coaching in the form of reviewing concept papers and proposals, as well as giving advice on how to approach donors. We also asked about implementers’ direct efforts to help CSOs become funding recipients, such as efforts to introduce a CSO to a funder, helping a CSO prepare customized materials, working with a CSO to identify new activities of interest to other donors, and making efforts “to a great extent” to help a CSO apply for new funds. Across the entire sample, 39% of CSOs reported being introduced to another funder from a CEPPS III implementer. Only 20% of CSOs reported that a CEPPS III implementer helped them to prepare customized materials, 35% reported that they were helped to identify new activities to undertake to attract new donors, and only 15% reported that a CEPPS III implementer made efforts “to a great extent” to help them apply for new funds. The differences between regions were not statistically significant for the customized materials and identifying new activities. However, the differences in shares between regions of CSOs reporting being helped “to a great extent” to apply for new funds were statistically significant at the 10% level. Below, we explore the relationship between CEPPS III implementers’ efforts to help CSOs obtain new funds from other donors, and whether or not the CSO reported obtaining new funds with CEPPS III assistance. We tested the relationship between the extent to which the CEPPS III helped the CSOs receive funding, and the rates at which CSOs received funding from other donors by conducting a series of t tests. We see that for all three levels of effort, there is a positive difference in shares of CSOs receiving funding. Indeed, those CSOs who reported receiving a great extent of implementer help (22 CSOs) had a difference of 66 percentage points in rates of obtaining funding from other sources – specifically, 86% of them received funding compared to 21% of those who chose another response. (This difference was highly significant at the 5% p-level.) Those who received help only to some extent saw a 20 percentage point difference in obtaining funding from those who did not. For the CSOs who reported for either a great extent or to 58 some extent, there was a 55 percentage point difference. For all three categories of effort, the differences were statistically significant. Table 4.5.3. Rate of Obtaining Donor Funding, and Level of Assistance Received Response categories for “CEPPS III made efforts to help CSO obtain funds…” Of CSOs who received different levels of help, which percent obtained funding? Difference Yes, chose this response No, did not choose this response N % got funding N % got funding To a great extent 22 0.86 102 0.21 0.66** To some extent 34 0.47 90 0.27 0.20** Either “To some” or “to a great extent” 56 0.63 68 0.07 0.55** Note: Total N = CSOs that reported obtaining funding from a different donor. ** Significant at 5% In addition to the t-tests, we ran a series of regressions to explore the relationship between inputs to receiving funding and whether or not a CSO actually obtained funds by running a series of regressions. The outcome variable of interest is a dummy variable for whether they received funding from another donor, and the explanatory variables of interest are various efforts implementers could take to increase the likelihood that a CSO might receive funding. Regression results showed, for example, that CSO involvement “to a great extent” in project design, awards management, or M&E has a positive and statistically significant relationship to obtaining funding. Though CSO involvement in reporting processes did have a positive relationship, it was not statistically significant. The full regression results can be found in Annex VIII. Table 4.5.4. Relationship between CEPPS III Assistance and CSO Obtaining Funds Variables (a) Obtained Funds with CEPPS III Assistance CEPPS III introduced CSO to a funder 0.254*** (0.0817) CEPPS III helped CSO prepare materials for gaining funds 0.270*** (0.0976) CEPPS III identified activities for CSO to find new funding 0.392*** (0.0807) Constant 0.0591 (0.0590) Observations 109 R-squared 0.607 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: GDP per-capita; region dummies. 59 CSOs were also asked about specific assistance provided to them, including introducing the CSO to a funder, helping CSOs prepare materials for gaining funds, and identifying activities for CSOs to find new funding. Regression analysis showed that all three have a positive and statistically significant relationship to a CSO obtaining funds from another donor. EQ5.2: Have efforts been made to help local organizations’ planning in order to be prepared to apply for and receive larger and more complicated awards? To investigate the efforts that have been made to help local organizations’ planning to be prepared to apply for new awards, we asked both the implementers and the CSOs to what extent they believed that implementers helped local CSOs to become capable of applying for and receiving larger and more complicated awards. Implementers appear to have a more positive opinion than CSOs of the extent to which ithey built CSO capacity to apply for larger awards. Indeed, though only 4.4% of implementers responded “not at all,” over 30% of respondent CSOs responded that implementers made no attempt. Conversely, though over 30% of implementers believed they performed “to a great extent,” only about 15% of CSOs had the same opinion. A table summarizing responses can be found in Table 4.5.5 below. Principal Findings  Implementers have a more positive opinion than CSOs on the extent to which they built CSO capacity to apply for larger awards. o Only 4.4% of implementers responded that they helped CSOs “not at all,” while over 30% of CSOs responded that implementers made no attempt. o Conversely, though 30% of implementers believed they made efforts to build capacity “to a great extent,” only 15% of CSOs held that opinion.  A CEPPS III CSO’s belief that the implementer made efforts “to a great extent” to build CSO capacity to apply for larger awards did not depend on which implementer gave the support.  The level of effort an implementer expended in helping a CSO obtain funds from other donors, as reported by the CSO, was positively correlated to the CSO actually obtaining those funds. 60 Table 4.5.5. Implementer and CSO Opinions on the Extent to Which Implementers Built CSO Capacity to Apply for Larger Awards Response Implementer CSO N Percent N Percent Not at all 5 4.4% 53 31.2% To a small extent 18 15.8% 36 21.2% To some extent 53 46.5% 40 23.5% To a great extent 35 30.7% 26 15.3% Skipped 3 2.6% 13 7.6% Do not know N/A N/A 2 1.2% Total 114 100.0% 170 100.0% Responses to “To what extent would you say [implementers] helped local CSOs in [country] to become capable of applying for and receiving larger and more complicated awards?” To estimate the relationships between implementer efforts and whether or not a CSO actually obtained funds from another donor, we ran a series of probit models with an outcome variable which equaled one if the CSO obtained funds from another donor with CEPPS III assistance. The explanatory variables of interest were dummy variables corresponding to varying levels of effort from implementers to help CSOs apply for larger and more complicated awards – “to a great extent,” “to some extent,” or “to some or a great extent.” Results show that the level of effort implementers expended to help CSOs apply for larger and more complicated awards had statistically significant relationships to whether or not that CSO obtained funds. A CSO responding that they were helped “to a great extent” had the most highly positive and statistically significant relationship to a CSO obtaining funds from other donors. A regression table for this analysis can be found in Annex VIII under Table 4.5.11. EQ5.3: What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative, and project management practices)? Principal Findings  The most common activity to support sustainability, as reported by implementers, working with the CSO to identify activities they could provide that would be of interest to donors. The second most common was introducing the CSOs to potential funders.  Quantitative analysis found that involving CSOs in the reporting, design, awards management, and M&E processes of CEPPS III activities had no effect on the CSO’s sustainability. 61 “NDI prepares groups for direct funding in different ways,” according to a key informant from NDI headquarters. “Even when giving sub-grants there was a multistage process to prepare CSOs for a time when they do their own program and financial reporting, in preparation for getting direct funding from USAID. This is incorporated into skills training, and the administration and documentation elements of the regular activities NDI is funding.” Table 4.5.6. Implementer Efforts to Increase CSO Sustainability Response N Percent Introduce local CSOs to potential funders (via email, telephone, or in person) 67 60.4% Help CSOs prepare customized materials to give to potential funders 53 47.7% Work with the CSO to identify activities they could provide that would be of interest to donors 83 74.8% None of the above 11 9.9% Responses to “Which of the following actions did your program take? Please select all that apply.” [N=111; Implementer survey] We asked implementers to self-report actions they took to help CSOs obtain additional funding from outside resources. The most common activity as reported by implementers was that they worked with the CSO to identify activities they could provide that would be of interest to donors. Second to that was introducing local CSOs to potential funders. A summary of responses can be found in Annex IX under Table 4.5.12. 62 5.0 OVERALL RECOMMENDATIONS  Consider context and objectives in seeking to promote quantity, quality, and/or consolidation of CSOs. This includes the balance of challenging assistance towards fewer experienced CSOs vs broader competition.  Promote an explicit emphasis on the organizational aspects of capacity-building, which appears to be neglected. Both the lack of emphasis in the PDs reviewed, and the fact that there was no statistically significant increase in the amount of capacity building directed at organizational skills (unlike technical skills) suggest that this is not sufficiently emphasized.  Continue to encourage implementers explicitly to engage CSOs in the design, awards management, M&E processes, and reporting to support sustainability and future outside funding for CSO recipients.  Encourage implementers to help CSOs find outside funding sources as part of programming. This is a relatively easy step and has been shown to be highly effective.  Consider sharing CSO survey results with implementers – there are often some sharply differing perceptions that may be useful in working with local CSO partners.  Encourage better record-keeping in the field, especially with respect to local CSOs, at a minimum including contact information for all organizations engaged in the program.  Consider pursuing follow-up research, for example on the determinants of sustainability, where a number of different factors were raised that could not all be tested. (Because the limited period of performance for the evaluation, the survey and case studies operated in parallel, which meant that it was not possible to test in the survey some of the interesting issues that came up in the case studies, but it would be useful to test these in future analytic work.) DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY PERFORMANCE EVALUATION OF THE CONSORTIUM FOR ELECTIONS AND POLITICAL PROCESSES STRENGTHENING III LEADER AWARD APPENDIX: SIX COUNTRY CASE STUDIES Field visits: Cambodia Georgia Guatemala Nigeria Tunisia Zambia __ DRG-LER TABLE OF CONTENTS CAMBODIA CASE STUDY Introduction........................................................................................................................ 1 Cambodia Country Context............................................................................................. 1 The CEPPS III Program in Cambodia 2010 – 2014 ....................................................... 2 Case Study Methodology .................................................................................................. 4 Answers to Evaluation Questions .................................................................................... 5 Conclusion/Synthesis of Findings ...................................................................................11 Appendix I – People Interviewed...................................................................................12 GEORGIA CASE STUDY Introduction......................................................................................................................14 Georgia Country Context...............................................................................................14 Case Study Methodology ................................................................................................16 Answers to Evaluation Questions ..................................................................................16 Conclusion/Synthesis of Findings ...................................................................................24 Appendix I – People Interviewed...................................................................................25 GUATEMALA CASE STUDY Introduction......................................................................................................................28 Guatemala Country Context .........................................................................................28 Case Study Methodology ................................................................................................30 Answers to Evaluation Questions ..................................................................................31 Conclusion/Synthesis of Findings ...................................................................................37 Appendix I – People Interviewed...................................................................................38 NIGERIA CASE STUDY Introduction......................................................................................................................41 Case Study Methodology ................................................................................................41 Nigeria Country Context................................................................................................42 The CEPPS III Program in Nigeria 2010 - 2014 ...........................................................43 Answers to Evaluation Questions ..................................................................................51 FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | ii Appendix I – People Interviewed...................................................................................60 TUNISIA CASE STUDY Introduction......................................................................................................................63 Tunisia Country Context ................................................................................................63 The CEPPS III Program In Tunisia 2010 – 2014...........................................................65 Case Study Methodology ................................................................................................66 Answers To Evaluation Questions.................................................................................67 Conclusion/Synthesis Of Findings ..................................................................................72 Appendix I – People Interviewed...................................................................................73 ZAMBIA CASE STUDY Introduction......................................................................................................................75 Background on Zambia...................................................................................................77 The CEPPS III Program in Zambia ...............................................................................78 Answers To Evaluation Questions.................................................................................82 Appendix I – People Interviewed...................................................................................90 DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY CAMBODIA CASE STUDY TASKING N019 PERFORMANCE EVALUATION OF THE CONSORTIUM FOR ELECTIONS AND POLITICAL PROCESSES STRENGTHENING III LEADER AWARD Contract No. GS-10F-0033M/AID-OAA-M-13-00013 September 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or Government. __ DRG-LER FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 1 INTRODUCTION Contracted by the United States Agency for International Development (USAID), NORC at the University of Chicago is conducting an evaluation of the Consortium for Elections and Political Processes Strengthening (CEPPS) III Leader with Associate (LWA) instrument pursuant to the USAID DRG Learning, Evaluation, and Research activity. The purpose of the overall study is to gain a better understanding of how local CSOs are impacted organizationally by their association with CEPPS partners and the extent to which the inclusion of Objective 10, relating to building CSO capacity, impacted program design under CEPPS III. Spanning 92 awards in 52 countries, the evaluation is informed by a document review, electronic surveys of USAID representatives, CEPPS partner field officers, and local NGOs working in the electoral processes field, as well as key informant interviews with home office personnel of the three CEPPS partners. In addition, case studies in six countries with programming under CEPPS III were selected. The case studies were selected based upon the amount of programming under CEPPS III in a particular country and the degree to which programming involved civil society. Controls to ensure geographic diversity and programming from each of the CEPPS partners were also applied. Cambodia was selected from the Asia region based upon the fact that it was considered to be a large program involving all three CEPPS implementers; program activities included electoral and political processes strengthening during a critical time, and that it was due to complete September 30, 2015. Although all three IFES, IRI, and NDI enjoy long histories of working to promote democratic development in Cambodia, the CEPPS mechanism was never used in-country before CEPPS III in 2009. Several CSOs and CSO networks were recipients of sub-grants through CEPPS programming. Special emphasis was given to women’s participation in politics during the run-up to the elections. The dates of performance reviewed included the period from October 1, 2009 – September 30, 2014. Because CEPPS III was still continuing at the time of this case study, and was scheduled to complete September 30, 2015, some activities during this more recent and current time period were also reviewed and included in this case study. The field research in Cambodia was conducted September 4 - 13, 2015. Prior to deployment, Washington, DC-based representatives of the CEPPS implementing partners were interviewed. While in-country, Resident Country Directors and/or key staff leaders and programming staff from two of the three CEPPS implementing partners were interviewed, as well as representatives of eight (8) CSOs. Additionally, two USAID CEPPS AORs (past and present), and the M&E Specialist, gave interviews. CAMBODIA COUNTRY CONTEXT Cambodia is a relatively stable country due to the strict rule of the Cambodian People’s Party (CPP), led by Prime Minister Hun Sen. In national elections on 28 July 2013 the CPP maintained control of the parliament, but lost significant ground to the Cambodia National Rescue Party (CNRP), which is the country’s main opposition party. Furthermore, the CNRP disputed the results of the election and claimed that its members actually won a slight majority in the parliament. The dispute of the election lasted for a year and involved several large scale demonstrations. Finally in July 2014, the CPP and CNRP settled their grievances and agreed to form a government.1 Over the last decade Hun Sen has made efforts to move the country toward a more single-party, authoritarian system with increased restriction on the freedom of expression.2 Recently, an opposition member of the Senate was arrested for allegedly committing treason, and a new law that will create stringent reporting requirements for non-governmental 1 World Watch Report on Cambodia, August 18, 2015, p. 1. 2 World Watch Report on Cambodia, August 18, 2015, p. 2. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 2 organizations has been passed and enacted by the government.3 Critics within Cambodia’s civil society, including several CEPPS CSO partners, have voiced their concerns over this law for several months, and continue voicing their concerns, saying that it will restrict and limit the impact of non-governmental organizations. Poor socioeconomic conditions and crime, including kidnapping, vigilante justice and revenge attacks, are the main issues confronting the Cambodian government. The USAID program in Cambodia traces its roots to humanitarian assistance and road and bridge building activities in the 1960s. U.S. assistance to Cambodia accelerated sharply after the signing of the Paris Peace Accords in 1991, which, in turn, led to the re-opening of the USAID mission in 1992. With the establishment of a full-scale mission, USAID shifted to a more strategic, long-term approach to reflect the development needs of Cambodia. While the initial focus was on meeting basic human needs, USAID also developed programs to support the UN-sponsored move to establish a freely elected government. Improving health and education services also emerged as key concerns. From the outset, USAID funding was delivered primarily through contracts and grants to private voluntary organizations or international organizations, a pattern that continues to this day. In order to ensure sustainability, USAID increasingly partnered with Cambodian-run organizations, many of which are leading local NGOs in the country today. Cambodia’s transition to democracy started approximately 20 years ago with the signing of the Paris Peace Accords. In the post-conflict context, the Royal Government of Cambodia emphasized stability while taking key steps toward democratization. Cambodia’s parliamentary elections in 2008 were freer than any previously held in the country, and the 2012 local elections were considered free. Nevertheless, political power remains imbalanced, and opposition parties struggle for sufficient political space to reach out to the electorate. USAID works closely with civil society organizations to promote human rights, civil and political liberties, and strengthen the role of women and youth in the political process. By informing and fueling public debate about transparency and accountability, USAID lays the groundwork for building the political will for change. USAID programs encourage youth and women to get involved in the democratic process. THE CEPPS III PROGRAM IN CAMBODIA 2010 – 2014 USAID/Cambodia pursues the long-term goal to eventually improve political competitiveness in Cambodia. Building on its long-standing assistance, USAID/Cambodia entered into an award agreement in 2009 with the Consortium for Elections and Political Processes Strengthening (CEPPS) to implement a new Accountability in Governance and Politics Program (AGAP).4 The five-year objective of AGAP is to promote increased accountability and transparency in policy-making and implementation. The five-year cooperative agreement of USD $20,710,000 (AID-442-A-09-00001), which ran from October 1, 2009 to September 30, 2014, includes program objectives applicable to both the pre- and post-election phases; however, the approved program awarded in 2009 reflects a distinct post-election strategy, as it focuses on accountability of officials elected in the 2008 elections. Pre-election activities 3 World Watch Report on Cambodia, August 18, 2015, p. 6. 4 USAID/Cambodia’s Accountability in Governance and Politics Program (AGAP), CEPPS Associate Award Cooperative Agreement AID-442-A-09-00001, was awarded in October 2009 to reflect an emphasis on accountability of elected officials in the post￾election environment. As Cambodia prepared for the 2012 Commune Council Elections and 2013 National Elections, USAID/Cambodia modified the existing CEPPS Associate Award to include critical pre-election programming that was not represented in the program. USAID/Cambodia combined a $1,000,000 award from DCHA/DG’s Elections and Political Processes (EPP) Fund with anticipated resources from its FY11 and FY12 Governing Justly and Democratically (GJD) budgets to support the new activities. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 3 were not approved at the time that the cooperative agreement was signed. As Cambodia prepared for the 2012 Commune Council Elections and the 2013 National Assembly Elections, USAID/Cambodia modified the existing CEPPS Associate Award to include critical pre-election activities not represented in the original program. These pre-election activities include: voter education on voter registration and political party platforms; training for Ministry of Interior (MOI) and National Election Committee (NEC) members on developing and maintaining accurate voter registries; training for political parties on debates and constituency outreach; training for domestic election monitoring activities; and support for youth and women’s political participation. The main objectives of the pre-elections AGAP Program amendment were to increase the number of registered voters, promote integrity of the voter lists and foster a more competitive and inclusive political environment. Special emphasis was given to women’s participation in politics throughout this pre-elections program.5 The three main sub-objectives of AGAP are: Strengthen Accountability and Expand Transparency and Access to Information – By enhancing the oversight role of elected leaders at all levels, creating constructive policy focused interactions between citizens and elected/appointed leaders, and improving the quantity, quality and dissemination of policy research and analyses; Generate Progressive Leadership – Cultivate leaders who advocate for government accountability and data driven policy debate on a wider range of policy issues. The program will create a cadre of approximately 500 to 700 young people (18-35 years of age) who will become future leaders; and Support Election Systems – Provide targeted assistance in lead up to the 2012 local government and 2013 national elections to ensure a more level playing field. It is noted that building capacity of local organizations and conducting capacity-building activities, vis-à-vis Objective 10 in the CEPPS III LWA, were not included as objectives in the AGAP Program. IFES implemented, through USAID’s CEPPS III program, activities designed to strengthen the advocacy capacity of women’s civil society organizations to leverage the national electoral reform discussion and build support for progressive measures to increase women’s political empowerment; and provide technical assistance to the reform of laws and regulations governing electoral and political processes to provide for a more effective transparent, competitive, and accountable electoral system. IRI implemented, through USAID’s CEPPS III, a 5-year program to: increase Cambodian citizens access to information about political developments in their country; identify, with Cambodian citizens and international stakeholders, priorities for reform to the National Assembly; generate and develop a cadre of future leaders seeking to enhance accountable and transparent governance; and engage women and youth learning to participate in a foster a more competitive and inclusive political environment. NDI implemented, through USAID’s CEPPS III, a five-year program to encourage public discourse and policy exchanges by bringing together elected representatives and their constituents in public dialogues on key issues. NDI’s objectives under the CEPPS III program were to: increase the responsiveness of members of the National Assembly and commune councilors to constituent concerns; promote the 5 USAID/Cambodia Program Description Amendment to the current CEPPS award Accountability in Governance and Politics Program (AGAP), June, 2011. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 4 political and civil rights of Cambodians and support an informed and active citizenry; and to support the election system by advancing a set of electoral reform proposals with civil society and popular support. CEPPS Follow-On in Cambodia: At the time of the interviews, USAID/Cambodia had made the determination to not use the CEPPS mechanism going forward. Instead, the follow on award to CEPPS III in Cambodia was through an open, competitive bid. It is known the both IRI and NDI competed for this award, as well as other organizations, including those with democracy and governance expertise. CASE STUDY METHODOLOGY The case study of the CEPPS III mechanism support in Cambodia relies, in part, on qualitative date collection and analysis, which addresses aspects of research questions that are not well suited to quantitative analysis. This case study was informed by a review of CEPPS III programming documents, key informant interviews with USAID personnel as well as in-country staff of 2 CEPPS partners, and interviews with Washington, DC based staff of 1 CEPPS partner. Further, significant input from CSOs affiliated with different aspects of CEPPS III programming was solicited through individual interviews. Finally, some independent research on documents and events referenced in the course of the interviews was conducted. Document Review: Prior to commencing the field-based interviews, the following program documents were reviewed:  USAID Program Description Amendment Statement, June 2011;  USAID Evaluation: Accountability in Governance and Politics Program Cambodia (AGAP) 2009 – 2013, August 2013 (Updated March 2014)(This is the CEPPS III Program in Cambodia);  CEPPS Cambodia FY 2015 Qtr 2 Quarterly Report Jan-Mar 2015;  Freedom House Report, Jan-Feb 2014;  Freedom House Report, Mar-Apr 2014;  Freedom House Survey;  Freedom House Survey Questionnaire;  IFES Voter Registration Assessment;  IFES Comment on Election Reform Process;  CDPO Joint Statement on Election Law;  CDPO Recommendations on Election Law;  Election Reform Alliance (ERA) Statement on Election Law;  Women and Men’s Networks Joint Statement on Election Reform;  World Watch Report on Cambodia, August 18, 2015 Key Informant Interviews: Interviews were held with the Leader of USAID/Cambodia’s Elections and Political Process Team in Phnom Penh, USAID/Cambodia’s Democracy and Governance Specialist, who are past and current Agreements Officer’s Representative (AOR) respectively, responsible for managing the CEPPS III award which is called the Accountability in Governance and Politics Program (AGAP), as well as the USAID/Cambodia Monitoring and Evaluation Specialist. IRI’s Acting Resident Country Director was interviewed, as well as IRI’s Vice-President for Programs. Focus Groups: NDI’s Resident Country Director and six senior NDI staff were interviewed in a focus group, also in-country. Partner Interviews: Interviews were conducted with eight representatives from eight CSOs collaborating in different ways with CEPPS partners. Seven received sub-grant awards to conduct election-related work FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 5 before, during, and after the 2012 and 2013 elections. One received a sub-grant to conduct a survey related to online freedom of expression in anticipation of discussions on a draft cybercrime law. All eight of the organizations were linked by participating in the Election Reform Network (ERN), a coalition of 20+ CSOs working for election reforms. ANSWERS TO EVALUATION QUESTIONS QUESTION 1: ARE THERE NOW MORE LOCAL CIVIL-SOCIETY ORGANIZATIONS ACTIVE IN: A) ELECTIONS ASSISTANCE; B) POLITICAL PROCESSES; AND C) PROVIDING TECHNICAL ASSISTANCE TO POLITICAL PARTIES THAN THERE WERE SIX YEARS AGO? Are there now more local civil-society organizations active in: a) elections assistance; b) political processes; and c) providing technical assistance to political parties than there were six years ago? According to stakeholders, there are over 4,000 registered CSOs in Cambodia, but estimates indicate that less than one-third of those are operational and functioning at any level.6 This is a nation-wide number that includes democracy, governance, and human rights CSOs as well as CSOs operating in other sectors. There is consensus overall that there are now more local civil-society organizations active in the three areas of CEPPS work: a) elections assistance; b) political processes; and c) providing technical assistance to political parties in Cambodia than there were six years ago, and that the most recent elections-related activities provided an opportunity and political space for new organizations to become involved through networks and coalition efforts. If there are more CSOs today than six years ago how many of the newer organizations worked with CEPPS institutions or their local partners? Exact figures could not be ascertained, but organizations participating in the most recent round of Commune elections (2012) and National Assembly elections (2013) include: those relatively new to elections assistance work, who joined in emerging networks or coalition efforts based on self-interest; as well as those organizations considered to be “traditional” CEPPS partner organizations, such as the Youth Council of Cambodia (YCC)7 and the Committee on Free and Fair Elections (COMFREL).8 Specifically discussed by stakeholders were youth-based organizations, noting that more youth overall are now participating in the elections, which is seen by the interviewees as a good thing, even though it may not have been a program priority in the initial CEPPS III redesign for Cambodia. In addition, in the past two to three years, many of these new and old organizations joined efforts to form new networks and coalitions. Two examples of this are the Electoral Reform Alliance (ERA)9, a coalition of twenty-plus organizations headed by a secretariat led by COMFREL, which is a long-time CEPPS partner of the National Democratic Institute (NDI); and the Youth Coalition for Unity and Development 6 Estimates from interviewees. 7 YCC is the Youth Council of Cambodia, an organization established in December 2001 to advocate for peaceful democratic reform and promote voter and civic education among Cambodian youth and students. 8 COMFREL is the Committee for Free and Fair Elections, one of Cambodia’s leading organizations established to conduct election-related activities. 9 ERA is a coalition of 20+ Cambodian organizations working together to conduct election-related activities for the 2012/2013 election cycle. The secretariat of ERA is led by the Committee for Free and Fair Elections (COMFREL). FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 6 (YCUD)10, a coalition of seven youth-based organizations including the Youth Council of Cambodia (YCC), which is a long-time CEPPS partner of the International Republican Institute (IRI). Both ERA and the YCUD are relatively new, having formed only in the last five to six years at most. Another noted evolution is that in past years for elections, organizations that were involved mainly focused on voter-registration and registration audits, Get-Out-The-Vote (GOTV) efforts, and election-day monitoring of polling stations. Now, it appears that more organizations are involved in electoral and political reform through policy advocacy and legislation efforts.11 Again, this is being viewed as a positive step forward by civil society activists, including those interviewed, at a time in Cambodia’s political history when there may be room, albeit limited, for such advocacy efforts to take place and to take hold. QUESTION 2: HOW WERE PROGRAMMATIC STRATEGIES DEVELOPED UNDER CEPPS II ALTERED OR EXPANDED (E.G., DIFFERENT APPROACHES TO ACTIVITIES, THE USE OF SUB-GRANTS) UNDER CEPPS III TO ADDRESS LOCAL CAPACITY BUILDING GOALS UNDER OBJECTIVE 10? According to the USAID representatives in Phnom Penh, CEPPS II was not utilized in Cambodia, and the CEPPS instrument was first used in country in 2009, under CEPPS III. Therefore, comparisons between CEPPS II and CEPPS III could not be made. Although long-time observers, managers, and implementers of CEPPS note that there was a deliberate strategy in CEPPS III, which was new and different from CEPPS II, to build the capacity of local partner organizations,12 building capacity of local organizations and conducting capacity-building activities, vis-à-vis Objective 10 in the CEPPS III LWA, were not included as objectives in the USAID/Cambodia CEPPS III Accountability in Governance and Politics (AGAP) Program. Those interviewed could draw some comparisons between activities during the early implementation of CEPPS III in Cambodia and those conducted just several months to one year before the close-out of the award in September 2015, likely due to recommendations made in the USAID Evaluation of the AGAP Program.13 Was the CEPPS process for local partner selection optimal in the sense that competitive selection processes were balanced against the CEPPS need for establishing ongoing relationships? Competitive selection processes were not used for the CEPPS III program in Cambodia. Instead, there was a greater emphasis on promoting long-term, on-going relationships with organizations. In responding to this question, several examples of CEPPS III partner sub-grant awards were discussed for illustrative purposes. Sub-grants that involved larger amounts of money, or which were longer-term in nature, such 10 YCUD is the Youth Coalition for Unity and Development, a coalition registered with the Ministry of Interior in 2014, of youth-focused CSOs who actively engage youth, government, and elected officials to advocate for the adoption and advancement of youth-centered public policy. For example: the freedom of youth networks to monitor elections across Cambodia. 11 This is not to say that the traditional election-related activities of voter registration, GOTV and election-day monitoring were abandoned, but were instead supplemented by newer more current policy reform efforts, which efforts were highlighted under CEPPS III in Cambodia in the past year. 12 The USAID Request for Applications (RFA) for CEPPS III included specific information about capacity-building. Capacity￾building included assisting local organizations to diversify their funding sources; manage multiple streams of funding; improve accounting and financial management practices; expand their memberships, and; better articulate their missions, and that the USAID Request for Applications (RFA) for CEPPS III. 13 See Recommendations: USAID Evaluation Accountability in Governance and Politics Program, which was a mid-term performance evaluation of the program. Once the evaluation’s findings were released, which included a recommendation for capacity￾building, the CEPPS implementers adjusted program activities, or emphasized them, as appropriate. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 7 as the IRI sub-grant to long-time partner YCC and the IRI sub-grant to Freedom House and the Cambodian Center for Independent Media (CCIM)14, as well as the NDI sub-grants to Transparence International Cambodia and COMFREL, are all examples of sole-source awards. These awards were typically managed and administered from IRI’s and NDI’s Washington, DC – based offices and involved conducting and passing the Non-U.S. Organization Pre-Award (NUPA) Surveys and audits in order to ensure the recipient organization’s readiness to manage the funds. Smaller sub-grant awards were made to CSOs direct from IRI’s and NDI’s Phnom Penh offices. To what degree did CEPPS involve local partners in the design, awards management, M&E processes, and reporting of programs with which they were affiliated? A USAID commissioned mid-term evaluation of Cambodia’s AGAP Program recommended to “place stronger emphasis on Cambodian ownership and sustainability, and ensure greater participation of all sub-grantees in activity design and achieving results that fulfill program objectives (AGAP and future project).15 According to stakeholders, it was clear that the CEPPS III implementing partners were making remarked progress in ensuring maximum and appropriate participation of their sub-grantees in program design, implementation, and reporting. Many, including ADHOC16 and CDPO17, described it as a “consultative process”, whereby their respective sub-grants were accompanied by sustained, on-going coaching or mentoring by the CEPPS III implementers. Even when the project idea was not theirs to begin with, and came from the CEPPS III implementers, when it came to design and implementation, by and large, the CEPPS III implementers provided the CSOs with room to initiate, create, and run the project with appropriate oversight. The technical assistance that was provided by the CEPPS implementers was described as accessible, appropriate, consultative, helpful, necessary, and on-going. QUESTION 3: TO WHAT DEGREE DID CEPPS INVOLVE LOCAL PARTNERS IN THE DESIGN, AWARDS MANAGEMENT, M&E PROCESSES, AND REPORTING OF PROGRAMS WITH WHICH THEY WERE AFFILIATED? Did CEPPS integrate organizational capacity-building activities into their collabora￾tions on elections and political processes, or did the activities occur on more of a parallel track? Stakeholders interviewed, as well as reports, described both integrated as well as parallel track capacity-building efforts. IFES organized capacity-building training with their partner CSOs, the Cambodian Disabled People’s Organization (CDPO) and the Committee to Promote Women in 14 CCIM is the Cambodian Center for Independent Media, an organization established in 2007 whose mission aims at promoting democratic governance and human rights enforcement in Cambodia through the development of a pluralistic and independent media environment. 15 USAID Evaluation - Accountability in Governance and Politics Program, Cambodia 2009-2013, August 2013 (Updated March 2014), p. 35. 16 ADHOC is the Cambodian Human Rights and Development Association, an organization founded by a group of political prisoners in 1991, whose mission is to address the absence of basic rights, freedoms and liberties in Cambodia by providing people with knowledge and understanding of human rights, law and democracy and of how to defend these rights and freedoms themselves. 17 CDPO is the Cambodian Disabled People’s Organization, an organization of persons with disabilities established in 1994 whose mission to: represent the voice of persons with disabilities; develop networks that work towards promoting and protecting the rights of persons with disabilities; and monitor and encourage the government and stakeholders to implement and improve legal frameworks advocating for persons with disabilities. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 8 Politics (CPWP). 18 These trainings appeared to take place on a parallel track with that of their program work.19 IRI discussed their capacity-building activities as mostly taking place on a parallel track with that of their program-related activities. For example, IRI developed specific trainings and workshops on proposal writing, financial management, and monitoring and evaluation, and provided these topical trainings to all of their partners, whether these partners were receiving sub-grant awards or not. In addition, the IRI has been working closely with the YCC since 2002, providing sub￾grant awards to them along with board development training. However, IRI also provided continual, sustained, coaching and mentoring. One CSO partner summed up their experience working with the IRI in this way: “Our capacity to conduct the type of research necessary and to develop a survey instrument and survey methodology was gradually built through the process of working together. We may not necessarily be able to do this again totally on our own, but we are one step closer to being able to do so. If we had one more experience of a similar nature we would likely be able to conduct this work without assistance.” NDI conducted an assessment of several of the provincial parliamentary outreach offices in an effort to gauge their current capacities. Based on the assessment, NDI’s plan was to draw on its institutional resources to provide the provincial outreach offices with tools to better process and respond to constituent concerns.20 To what extent have CEPPS local and regional partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? While the collecting and adopting of best practices did not appear to be a regular practice among the CSOs interviewed, there were clear and specific examples that were repeatedly and enthusiastically discussed among stakeholders as being representative of best practices, and those that would be attempted to be replicated in the future. Examples include: Electoral Reform Network (ERA) The Electoral Reform Network, with the more experienced COMFREL heading up the efforts of mostly less experienced organizations that made up the ERA, serves as an example of a best practice, according to several stakeholders. In addition, the fact that the ERA was made up of member organizations who brought a diverse set of skills and experiences to the table serves as yet another example of a best practice. The true value of the ERA, as one CSO put it, “was that it brought all of us to the table in a way that has never happened before.” Prior to the 2012/2013 elections, many organizations provided some type of monitoring of the polling day process. But 18 CPWP is the Committee to Promote Women in Politics, who since 2005, has been working to promote and advocate for equal participation and representation of women and men in effective decision-making in public office; To strengthen women’s awareness of, voices, capacity and collective action in political decision making. 19 USAID CEPPS Quarterly Report 20 USAID CEPPS Quarterly Report, p. 17. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 9 these efforts were disparate, and had disparate impact as a whole. “ERA brought us all to the table together, which made our voice, and the effort, stronger.”21 The Situation Room Shortly before the 2013 National Assembly Elections, ERA (and COMFREL) led an activity called “The Situation Room.” The Situation Room brought all of the ERA members together to report in real time, election-related activity at the polling station during election day, and while the polls were being counted after election day. Real time activities were reported, documented, reported to the public, and when necessary, reported to the NEC and the Ministry of Interior for action. The Situation Room activity is currently being seen as a best practice in programming, and will likely be replicated and expanded to the Provincial and District levels, should the opportunity present itself to do so. Monitoring and Evaluation and Reporting Monitoring and evaluation, and reporting, and training the CEPPS partner CSOs in incorporating M&E and reporting into their daily work is also seen as, if not a best practice, a critical one that should be considered and incorporated into all programming. Which factors contribute to the successful elevation of local organizations into sustainable providers of technical assistance related to elections and political processes? According to stakeholders, when partners go beyond their comfortable base of knowledge and experience and move into something more broad to begin to define themselves in different ways, provides a measure of growth and sustainability. Examples given were of the women’s organizations and minority-based organizations that developed a broader base of support by moving beyond their traditional boundaries of membership and defining themselves more broadly. When partners move from being urban based to being part of the rural communities and begin defining themselves as something broader, and begin serving a broader base membership. Stakeholders cited other factors that need to be present for an organization’s success. These included: When a committee, coalition, or network is formed to move our work forward together; When multiple experiences and sets of expertise (human rights, legal, advocacy) are involved; When the organization can reach all areas of the country; When we have specific expertise in election reform; When staff is willing to work as volunteers, and pay their dues as volunteers; When we are provided with timely, sustained, and appropriate technical assistance by our donors or others; and 21 Comments by CEPPS III CSO partners involved in the 2012/2013 elections. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 10 When we have patient, flexible, understanding and accessible donors. QUESTION 4: WHICH FACTORS CONTRIBUTE TO THE SUCCESSFUL ELEVATION OF LOCAL ORGANIZATIONS INTO SUSTAINABLE PROVIDERS OF TECHNICAL ASSISTANCE RELATED TO ELECTIONS AND POLITICAL PROCESSES? Considering what is required for potential success, what are the characteristics of the local and regional organizations receiving CEPPS III support? Stakeholders listed characteristics that demonstrate the ability to be self-reliant and that encourage organizations to have a greater degree of independence and sustainability as: When partners obtain funding on their own; When finances are in order; When audit compliances are met. What efforts on behalf of CEPPS have been made to support local organization opportunities for becoming direct recipients of donor funding? Even though building capacity of local organizations and conducting capacity-building activities, vis-à-vis Objective 10 in the CEPPS III LWA, were not included as objectives in the AGAP Program, the USAID mid-term evaluation of the AGAP Program (CEPPS III) provided a recommendation to “Place stronger emphasis on Cambodian ownership and sustainability. Further interventions should place stronger emphasis on sustainability through lasting capacity-building of sub-grantees, local implementing partners and beneficiary institutions, and progressive transfer of skills to Cambodian-owned initiatives. Program design should prioritize activities that can be ultimately replicated by Cambodian partners, notably to allow increased frequency and larger reach. For the remainder of the AGAP program as for future interventions, implementers should also ensure greater participation of all sub-grantees in activity design and achieving results that fulfill program objectives.22 It was noted by long-term members of the CEPPS mechanism that, under CEPPS III, USAID now looks for a significant commitment, in terms of time, energy, and resources, from their local, host-country national partner organizations. In response to the USAID recommendation in the AGAP Evaluation, in-country CEPPS implementers developed and organized trainings and workshops in financial management, program monitoring and evaluation; fundraising, proposal writing and proposal development. While most partner CSOs need and expressed appreciation for these efforts at capacity-building, one believed, due to their advanced experience and sophistication, that the efforts did not go far enough, or worse, were not the best use of their time. 22 USAID Evaluation - Accountability in Governance and Politics Program, Cambodia 2009-2013, August 2013 (Updated March 2014), p. 35. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 11 QUESTION 5: WHAT EFFORTS ON BEHALF OF CEPPS HAVE BEEN MADE TO SUPPORT LOCAL ORGANIZATION OPPORTUNITIES FOR BECOMING DIRECT RECIPIENTS OF DONOR FUNDING? To what extent has CEPPS made efforts to assist local organizations become prepared to apply and receive larger and more complicated awards? What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative and project management practices)? In-country CEPPS implementers noted that in addition to providing specific workshops and trainings on fundraising and proposal development, they have helped coach and mentor their partner organizations in the processes in which they are engaged or engaging in, in order to seek funding from other sources. This coaching has come in the form of reviewing concept papers and proposals, and providing comment and feedback to providing advice on how to approach donors. It bears noting that many of the Cambodian CSOs involved with CEPPS III are older, experienced, organizations with an established track record of receiving and managing funding from multiple sources. Those sources include Australian Aid and the Australian Embassy, DFID, the NED, OXFAM, the UNDP. CONCLUSION/SYNTHESIS OF FINDINGS The period between 2009 and 2014 saw many positive changes in Cambodia’s political environment, especially immediately before, during, and after the 2012 Commune Council Elections and the 2013 National Assembly Elections. For the CEPPS implementers, it was a period marked by more “more.” More youth engaged in election-related activities; more CSO involvement, and perhaps involvement by those CSOs who normally do not engage in political processes; and more unity among the CSOs to work together for purposes of reform, particularly in relation to electoral reform. It was also a period of creativity – although elections and election-related activity have been supported by USAID for years, under CEPPS III, the Election Reform Network was created that served to knit previously disparate efforts together, creating unity of effort and voice. The Situation Room activities served to excite and stimulate partner organizations and their constituencies and served as a catalyst for several sets of electoral reform recommendations to be released – without compromising the individuality of the different networks, yet all under the broad umbrella of the ERA. Some of the recommendations, particularly in the formation of the NEC, have been accepted by the Government, which may provide an opening for more election￾related work and reform efforts in the upcoming round of elections. CSO participation in both the ERA and the Situation Room provided capacity-building opportunities by learning new program activities, and how activities are implemented and reported, promoting efforts publically through various organizational constituencies, and in building relationships with other organizations. But these capacity-building opportunities were not promoted by the addition of Objective 10 in the CEPPS III LWA in Cambodia. Capacity-building activities, although present before USAID recommendations, increased with specificity after USAID commissioned the AGAP evaluation. While the utility of these capacity-building recommendations can be argued by some, it is clear that for others they are necessary to lead to increased levels of independence, and should be included, and reported on, in future programs. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 12 APPENDIX I – PEOPLE INTERVIEWED USAID Mr. Jean-Marc Gorelick, Elections and Political Processes Team Lead, Office of DG, USAID/Cambodia Ms. Socheata Vong, Democracy and Governance Specialist, USAID/Cambodia Ms. Peolida Ros, Monitoring and Evaluation Specialist, USAID/Cambodia IFES Ms. Juliette Schmidt, Asia Deputy Regional Director, IFES (TBD) IRI Mr. Thomas Garrett, Vice President for Programs, IRI Mr. Derek Luyten, Regional Director, Asia, IRI (TBD) Mr. Bendarom, Senior Program Officer, IRI NDI Mr. John Cavanaugh, Cambodia Country Director, NDI Ms. Ros Sinak, Senior Program Officer, NDI Ms. Hang Sayaneth, Senior Program Officer, NDI Mr. Ne Jony, Senior Program Officer, NDI Ms. Chhiv Kimsrun, Senior Program Officer, NDI Mr. Lee Sothearayuth, Program Manager, NDI Mr. Ravy Uong, Resident Princeton in Asia Fellow, NDI Local Partners Mr. Kim Chhorn, Senior Program Coordinator, Cambodia (COMFREL) Mr. Ngin Saorath, Executive Director, Cambodia Disabled Peoples Organization (CDPO) Ms. Amanda King, Dir., Resource Mobilization &Comms, Cambodia Center for Independent Media (CCIM) Mr. Im Sothy, Executive Director, Youth Center for Cambodia (YCC) Mr. Kim Eng Yong, Founder and Member of Board, Youth Coalition for Unity and Development (YCUD) Mr. Ny Chakrya, Head of HR Section, Cambodian Human Rights Association (ADHOC) Mr. Pen Raksa, Program Mgr. Partnerships & Coalition Bldg., Transparency International Cambodia (TIC) Ms. Naly Pilorge, Director, Cambodian League for the Promotion and Defense of Human Rights (LICADHO) FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 13 DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY GEORGIA CASE STUDY TASKING N019 PERFORMANCE EVALUATION OF THE CONSORTIUM FOR ELECTIONS AND POLITICAL PROCESSES STRENGTHENING III LEADER AWARD Contract No. GS-10F-0033M/AID-OAA-M-13-00013 September 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or Government. __ DRG-LER FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 14 INTRODUCTION NORC at the University of Chicago is conducting an evaluation of the CEPPS III Leader With Associate (LWA) instrument pursuant to the DRG Learning, Evaluation, and Research activity. The purpose of the overall study is to gain a better understanding of how local CSOs are impacted organizationally by their association with CEPPS partners and the extent to which the inclusion of Objective 10, relating to building CSO capacity, impacted program design under CEPPS III. Spanning 92 awards in 52 countries, the evaluation is informed by a document review, electronic surveys of USAID representatives, CEPPS partner field officers, and local NGOs working in the electoral processes field, as well as key informant interviews with home office personnel of the three CEPPS partners. In addition, case studies in six countries with programming under CEPPS III were selected. The case studies were selected based upon the amount of programming under CEPPS III in a particular country and the degree to which programming involved civil society. Controls to ensure geographic diversity and programming from each of the CEPPS partners were also applied. Georgia was selected from the Europe and Eurasia region based upon the fact that there have been multiple projects, involving all three CEPPS partners under CEPPS II and III. Moreover, CEPPS partners in Georgia have worked with a wide variety of NGOs, including some who are currently implementing USAID programs with a direct grant. The field research in Georgia was conducted September 7-11, 2015. Resident Country Directors and programming staff from each of the CEPPS implementing partners were interviewed, as well as representatives of 12 NGOs. Finally, from USAID, the CEPPS AOR, the alternate AOR, and the DRG Office Director gave interviews. GEORGIA COUNTRY CONTEXT USAID began programming in Georgia in 1992, shortly after it gained independence from the Soviet Union. NDI, IFES, and IRI started working in Georgia soon thereafter, in 1994, 1995, and 1999 respectively, with funding from the Swedish International Development Agency (SIDA), the National Endowment for Democracy (NED), and USAID, through the various CEPPS mechanisms after 1995. Throughout this period Georgian politics has been highly polarized, with bitter exchanges accompanying each shift in power. President Shevardnadze’s regime was overthrown in what became known as the Rose Revolution, popular protests over widespread allegations of electoral fraud in the parliamentary elections of 2003. Mikheil Saakashvili was elected president in the ensuing snap elections, and his party, United National Movement (UNM), subsequently won a large majority of parliament in 2004. In turn, the popularity of President Saakashvili’s administration waned as concerns over corruption and persecution of opposition parties mounted, culminating in violent protests in November 2007 and again in the spring of 2009. Mr. Saakashvili’s coalition lost to the current administration, the Georgian Dream, in the parliamentary elections of 2012. With a persistent emphasis on electoral integrity, political party building, and inclusivity, USAID has used the CEPPS LWAs to address these issues in different ways in Georgia over the past 20 years. Under CEPPS III, USAID has issued two awards to each of the CEPPS partners. July 2010 – July 2014, the following programs were supported: “Parliamentary Strengthening Project” by NDI “Strengthening Political Competition in Georgia” by IRI; and “Increased Trust in the Electoral Process” by IFES. During that time, various modifications were made to address needs as they arose, including support for a parallel vote tabulation in 2012, voter education programming in 2013, and additional funding for the FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 15 presidential and parliamentary elections in 2014. Moreover, a separate award was made to IRI to fund an international observation mission for the parliamentary elections in 2012. The second series of awards under CEPPS III started in July 2014 and continues through January 2019. Those projects are: “Strengthening Political Processes in Georgia” by NDI; “Strengthening Political Parties in Georgia” by IRI; and “Strengthening Electoral Processes in Georgia” by IFES. Institutional Development: NDI has been called upon under CEPPS III to strengthen parliamentary and political processes. Although these mandates have required NDI to work intensively with parliament, and parliamentary factions as its institutional counterparts, NDI has also engaged civil society. In strengthening parliamentary processes, NDI supported the NGO Liaison Office, encouraging Members of Parliament to solicit input and feedback from civil society organizations on policy issues, as well as to promote transparency, accountability, and effectiveness. Moreover, CEPPS has supported NDI’s work on election observation and vote verification, which in turn has led to building the organizational capacity of the International Society for Fair Elections and Democracy (ISFED), and to a lesser degree of the Georgian Young Lawyers Association (GYLA) and the Public Movement “Multinational Georgia” (PMMG) under CEPPS III. In addition to its CEPPS-funded work, NDI has a dedicated civil society development component under its SIDA grant, which allows NDI to help these NGOs hone their advocacy skills to interact productively on policy issues. Political Party Development: IRI’s work in Georgia under CEPPS III has emphasized political party development. As political parties in Georgia tend to be highly centralized and personality-driven, IRI has worked to develop ideological party identities in the regions. Additionally, IRI has dedicated considerable efforts to issues of inclusion, helping parties reach out to broader constituencies as well as promoting diversity within party structures. To promote youth engagement, IRI conducted annual summer schools and debates to encourage greater youth engagement; initially these involved various CSOs representing youth throughout Georgia, although as political parties began developing youth wings, IRI’s shifted the emphasis to them as a more direct means of achieving its goal of encouraging greater inclusivity in political parties. Further, IRI was originally committed to improving the enabling environment by promoting dialogue between and among parties, government and civil society. However, with USAID’s concurrence this objective was dropped when the Government amended the law on political parties in 2011 to characterize consultations between CSOs and political parties as “in-kind contributions,” which had to be accounted for by the political party. Electoral Processes: IFES, as the CEPPS partner responsible for building public trust in the electoral system, has had the most CSO engagement under CEPPS III. Under Increased Trust in the Electoral Process (ITEP) IFES worked extensively with civil society, building its capacity to inform and motivate voters. IFES encouraged existing NGO networks like PMMG and the Coalition for Independent Living (CIL) to advocate for the electoral rights of their respective constituents, ethnic minorities and people with disabilities. IFES was also instrumental in the formation of two networks, the Gender Equity Network (GEN) and Journalists Network for Gender Equity (JNGE) advancing women’s engagement in political processes. Further, IFES helped promote vigorous monitoring of campaign finance, working with Transparency International (TI) to formulate a methodology to track campaign finance and use of administrative resources. Under its current program, Strengthening Electoral Processes, IFES is building upon its achievements to fortify civil society’s relationship with governmental institutions. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 16 CASE STUDY METHODOLOGY This case study is informed by a review of CEPPS III programming documents, key informant interviews with USAID personnel as well as in-country staff of the 3 CEPPS partners. Further, significant input from CSOs affiliated with different aspects of CEPPS III programming was solicited through individual interviews. Finally, some independent research on documents, organizations, and events referenced in the course of the interviews was conducted. Document Review: Prior to commencing the fieldwork, the following programming documents were reviewed: Program Descriptions: program description for each of the 3 current projects; Final Reports: final reports from 3 projects implemented under CEPPS III between 2010-2014; Quarterly Reports: 3 quarterly reports, on from each of the current projects, for the period January-March 2015; National Surveys: public perception surveys for 2011 and 2012; Assessment: Women’s Leadership as a Route to Greater Empowerment, a Georgia Case Study, 2014; Websites: organizational websites of CEPPS partners, USAID Georgia, and other donor organizations working in electoral and political processes. Key Informant Interviews: In depth interviews were held with the Director of USAID’s office of Democracy, Rights, and Governance, as well as with the Agreements Officer’s Representative (AOR) and alternate AOR responsible for managing the CEPPS project. Resident Country Directors and senior staff from each of the 3 CEPPS partners were also interviewed in depth at the beginning of the field research; a former Country Director who had managed NDI’s programming from 2009-2014 was interviewed prior to the field research. Partner Interviews: Interviews were conducted with 18 representatives from 13 CSOs collaborating in different ways with CEPPS partners. One CSO received support for its political party training series that had been initiated through the British Embassy; one CSO was subcontracted to run a summer program for youth interested in political processes; 5 organizations had conducted election observation and/or vote verification activities with support from the CEPPS partners; 5 organizations, representing women or minorities, received subgrant support as part of IFES’ political inclusion activities. ANSWERS TO EVALUATION QUESTIONS QUESTION 1: ARE THERE NOW MORE LOCAL CIVIL-SOCIETY ORGANIZATIONS ACTIVE IN: A) ELECTIONS ASSISTANCE; B) POLITICAL PROCESSES; AND C) PROVIDING TECHNICAL ASSISTANCE TO POLITICAL PARTIES THAN THERE WERE SIX YEARS AGO? Are there now more local civil-society organizations active in: a) elections assistance; b) political processes; and c) providing technical assistance to political parties than there were six years ago? Civil society engagement in electoral and political processes in Georgia has waxed and waned with changes in the political situation, as well as with availability of donor funding. Immediately following Georgia’s independence, the international community showed support for President Shevardnadze, hoping to ensure FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 17 a smooth transition to democracy. By the turn of the century, however, the international community had shifted towards supporting civil society’s call for more accountable governance. With increased availability of donor funds, plus a reasonably permissive enabling environment, and the impetus to fight corruption, crime, and economic hardship, civil society grew rapidly. Civil society organizations played an important role in the Rose Revolution, though soon after many civil society leaders took positions in the ensuing Saakashvili government. Donor funding also become more concentrated on good governance support, making survival more challenging for CSOs who remained. There was a re-emergence of NGOs as disgruntlement with Saakashvili’s government grew. There was also increased donor support to civil society, in part as a response to actions by the Saakashvili administration to consolidate power and squelch opposition. Finally, with the shift in power from Saakashvili’s UNM to Georgia Dream in 2012, many former government people have returned to civil society,23 while some former civil society leaders have taken governmental positions. Elections Assistance: The electoral assistance field is heavily dominated by 3 well-established NGOs: ISFED, GYLA, and TI. The undisputed leader in election observation is ISFED, which benefited from intensive and ongoing support from NDI since their inception in 1995. GYLA and TI have broader organizational mandates, but also participate in elections assistance, observing different aspects of the electoral process such as resolution of complaints and results reporting. As well-established, capable CSOs, all 3 are popular subgrantees for technical assistance providers, as well as for donors. However, while the existence of sustainable NGOs is certainly a positive outcome, it has had the tendency to squeeze out newer NGOs seeking to work in the electoral sphere. For example, ISFED, and NDI itself, have counseled donors against supporting anyone other than ISFED to conduct a parallel vote tabulation (PVT), actively trying to discredit other organizations seeking to do the same, or similar work. Whether this is a matter of best practice (i.e. multiple PVTs could undermine each other if the results are inconsistent, in reality or perception), a bid for self-preservation, or both, is subject to debate. While there are a few newer organizations doing observation, exit polls, and PVTs, they tend to work in niche areas—i.e. observing the minority vote. Accordingly while there may be more organizations trying to work in this sphere, noting an increase in absolute numbers could be misleading given the heavy predominance of ISFED, GYLA, and TI. Political Processes: Civil society has a long history of engagement in political processes dating back to the Rose Revolution in 2003. Large-scale protests against corruption and abuse of office also took place in 2007 and again in 2009. Public perception that elections represented the outcome of their collective vote was a mere 51% in 2011; even fewer thought that voting was an opportunity to influence decision-making (44%) or that they could they could influence decisions made by their government (29%).24 Under its Increased Trust in the Electoral Process project (ITEP), IFES engaged 32 CSOs with subgrants totaling $1.4 million to improve transparency of the electoral process and increase understanding about political issues. By the conclusion of this project in 2014, 49% of Georgians agreed that they could influence government decisions.25 Recognizing that these figures are substantially lower in minority communities,26 IFES has particularly targeted marginalized segments of the population, including youth, women, people with disabilities, as well as ethnic minorities, by working with existing CSOs representing these constituencies both in the ITEP project and continuing into its current Strengthening Electoral Processes project. 23 Several high ranking UNM officials fled Georgia, fearing political persecution after their defeat in the 2012 elections. Some have since returned and assumed opposition roles in civil society organizations. 24 IFES Georgia 2011 national survey report. 25 IFES Georgia 2014 national survey report. 26 IFES polling in 2014 indicates that far fewer ethnic Azeri, Armenian and Russian citizens of Georgia believe they can influence the governmental process, 35%, 27%, and 19% respectively. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 18 Accordingly, while these same CSOs existed in 2009, it can fairly be concluded that CEPPS programming has reinvigorated their involvement and contributed to empowerment in the political sphere. Political Parties: Through its CEPPS programming implemented by IRI, USAID has invested substantially in developing a multi-party, and policy-oriented political system in Georgia. Recognizing that the Tbilisi￾centric nature of Georgian politics leaves a dearth of political activity in the regions,27 IRI committed to promoting rural branches of political parties. Working with youth as a catalyst, IRI supported youth debates and workshops, involving youth branches of the political parties where they existed. Some NGOs representing youth also sought to participate. Initially, in order to encourage broader debate of the issues, IRI welcomed their participation. However, over time participation of NGOs began to dilute IRI’s main objective of strengthening the political parties. Additionally, the 2011 amendment to the Organic Law on Political Parties, broadening the definition of contributions to potentially include technical assistance from CSOs, had the desired chilling effect on collaboration between NGOs and political parties. Accordingly, IRI’s youth activities is now limited to the respective branches of political parties. If there are more CSOs in the three above evaluation-question areas today than six years ago how many of the newer organizations worked with CEPPS institutions or their local partners? CEPPS partners appear to have played an important role in encouraging existing NGOs to become more active in the electoral sphere, as described above. However, while CEPPS has contributed to increasing the number of CSOs active in electoral and political processes by reaching out to NGOs representing typically marginalized segments of the population, there were also two circumstances where CEPPS has purposely avoided working with new NGOs. First, the highly polarized nature of Georgian politics has resulted in politicization of certain aspects of the civil society sector. Following changes in administration, some outgoing government officials resorted to positions in civil society, either returning to CSOs with which they had a previous affiliation, or starting new organizations. In order to avoid the perception of partisanship on electoral assistance activities (i.e. observation, vote verification, polling), CEPPS partners tend to avoid working too closely with NGOs headed by or closely affiliated with political parties. A second phenomenon relates to the noticeable increase in NGO registrations dating back to 2012. Although there are more than 15,000 NGOs formally registered, USAID estimated that the number of active NGOs is in the hundreds. The influx of “mushroom” NGOs is generally associated with political efforts to discredit UNM and/or to dilute the impact of objective reporting on electoral processes by providing a parallel narrative. There appears also to be an effort to confuse the public by registering under names resembling those of established organizations. For example, one of the new organizations registered as the Young Barristers, which is easily confused with GYLA (the Young Lawyers) in Georgian. USAID and the CEPPS partners reported that many of the newly registered NGOs exist only during the election period and otherwise are wholly illusory. How were programmatic strategies developed under CEPPS II altered or expanded (e.g., different approaches to activities, the use of sub-grants) under CEPPS III in order to address local capacity building goals under Objective 10? Program descriptions under CEPPS III called for civil society engagement, without specifically referencing an intent to build their organizational capacity. The initial series of awards under CEPPS III (2010-2014) included the following objectives related to civil society: 27 For example, in the May 2010 local elections there were more than a dozen candidates in Tbilisi, whereas the regions typically had no more than 4 or 5 candidates. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 19 From Strengthening Political Competition, implemented by IRI: Objective 5: Address the overall enabling environment through dialogue among parties, government leaders and civil society stakeholders;28 From the Parliamentary Strengthening Project, implemented by NDI: Objective 2: improve policy collaboration with CSOs, parliament staff and MPs; Objective 3: institutionalize transparency; and From Increased Trust in the Electoral System, implemented by IFES: Objective 2: Improve transparency of electoral processes through improved media coverage and CSO involvement; The second series of CEPPS III awards, 2014-2019, also anticipated civil society engagement in the NDI and IFES programs: From Strengthening Political Processes implemented by NDI: Objective 2: Independent oversight of elections enhanced. From Strengthening Electoral Processes, being implemented by IFES: Objective 2: Enhanced Civic Engagement around Key Electoral and Political Process and Reform Although not specifically addressed in the program descriptions under CEPPS III, it is evident that USAID appreciated the extent to which CEPPS partners had built the capacity of its partner CSOs. In 2013, USAID began planning for local sourcing election observation and vote verification activities. Its Project Appraisal Document (PAD) for 2013-2014 included a budget line for an Annual Program Statement (APS) to fund local organizations to conduct short- and long-term electoral observation. At the time the field work for this case study was done, USAID anticipated making approximately 3 awards under the APS; USAID specifically encouraged organizations that have not previously received direct funding from USAID to apply. These local grants will be in addition to a 2014 award USAID made directly to ISFED for observation of the local elections. One CEPPS partner noted that there was a “greater level of stability in CEPPS III” which made their work with CSOs easier than under CEPPS II. They described a lot of program modifications and no cost extensions under CEPPS II that made it hard to plan. Uncertainty in the funding streams caused the CEPPS partners to help NGO partners fundraise for small grants from other donors. In contrast, under CEPPS III, they felt that donor coordination was better and that rather than scrambling to find funding for their local partners, they were struggling to effectively program new influxes of money allocated immediately prior to hotly contested elections. Ironically, the irregularity of funding under CEPPS II may have contributed more to CSO sustainability by forcing CEPPS partners to help them diversify their funding sources. 28 This objective was ultimately dropped when amendments to the Organic Law on Political Parties, referenced previously, made it difficult for political parties to engage with civil society without having to account for “in kind” contributions. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 20 QUESTION 2: HOW WERE PROGRAMMATIC STRATEGIES DEVELOPED UNDER CEPPS II ALTERED OR EXPANDED (E.G., DIFFERENT APPROACHES TO ACTIVITIES, THE USE OF SUB-GRANTS) UNDER CEPPS III TO ADDRESS LOCAL CAPACITY BUILDING GOALS UNDER OBJECTIVE 10? Was the CEPPS process for local partner selection optimal in the sense that competitive selection processes were balanced against the CEPPS need for establishing ongoing relationships? The process for local partner selection appeared well-tailored to meet the demands of the program descriptions as well as maximize efficiency. The different CEPPS partners utilized different methods of selecting subgrantees, reflecting the variations in their respective objectives. IRI’s program description, for example, emphasized the need to build the capacity of political parties, and civil society was only incidental to that goal. As such, IRI’s interaction was limited. In some instances IRI engaged CSOs for a discrete and specific purpose, such as organizing the political leadership schools it funded. In its youth programming, NGOs sometimes approached IRI seeking to participate along with the youth branches of the political parties; IRI indulged these requests, particularly in the beginning, in order to stimulate interest in politics among the youth. In one other instance, IRI picked up a program that had previously been supported by the British Embassy, seemingly with USAID’s encouragement. In none of these instances, however, was there an expectation of, or need for an ongoing relationship or capacity building. In contrast, NDI established long-standing relationships with the CSOs it worked with, building their skills and also their organizational capacity. NDI had worked with ISFED, for example, since its inception in 2005. NDI is also the primary partner in elections for GYLA, building their capacity to conduct election observation over several years. To a lesser degree NDI has provided grant support and technical assistance to TI and PMMG as well. NDI justified its approach of sole sourcing its grants on several grounds. First, it expressed a need for assurance that its grantees would be open to feedback and guidance so that subgrant activities met NDI’s minimum standards; second, building organizational capacity requires sustained interaction, as opposed to “a lot of one offs;” and finally, imparting technical skills, such as the intricacies of conducting a PVT, require extended time. IFES used both sole sourcing and competitive grant selection. In instances where they had invested in building an organization’s capacity in a specific skill area, they would secure that investment by sole sourcing another grant to the same organization. An example of this was IFES’ work on building TI’s approach and monitoring methodology. IFES’ subgrant to TI in 2010 helped develop a unique approach to monitoring the use of administrative resources during the local elections. IFES sole sourced an award to TI in 2011 to utilize that approach in the 2012 parliamentary elections. In 2013 IFES again sole sourced a grant to TI, building upon its administrative resource monitoring approach and piloting a methodology for parallel expense tracking in the presidential election. In other instances, however, where there was no tie to previous efforts to build particular skills, IFES selected grantees through a competitive process. To what degree did CEPPS involve local partners in the design, awards management, M&E processes, and reporting of programs with which they were affiliated? Local partners have influenced the design, awards management, M&E processes and reporting of programs in various ways. USAID conducted an assessment of CEPPS programming in 2012, which involved meetings with subgrantees, government representatives, and other stakeholders. Information from this FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 21 assessment informed the PAD for CEPPS activities. For their part, the CEPPS partners incorporate feedback from their respective partners in each of their quarterly and performance monitoring reports. QUESTION 3: TO WHAT DEGREE DID CEPPS INVOLVE LOCAL PARTNERS IN THE DESIGN, AWARDS MANAGEMENT, M&E PROCESSES, AND REPORTING OF PROGRAMS WITH WHICH THEY WERE AFFILIATED? Did CEPPS integrate organizational capacity-building activities into their collabora￾tions on elections and political processes, or did the activities occur on more of a parallel track? The skills needed to effectively manage subgrants—from accounting to narrative reporting to monitoring and evaluation--were very much integrated into their collaborations on elections and political processes. IFES conducted trainings for all of its grantees, covering topics such as financial administration and reporting requirements. TI credits IFES not only for building its technical skills, but also developing its administrative capacity to manage grants. PMMG expressed gratitude to NDI and IFES for helping them understand strategic planning, financial management, and monitoring and evaluation. Similarly, both ISFED and GYLA credit NDI with helping them develop their outreach and communication plans. CEPPS partners are also providing some assistance to their local partners parallel to the elections and political processes programming track. For example, IFES was instrumental in the emergence of two strong gender networks, GEN and JNGE. Stemming from a simple call for proposals in 2012 to address negative perceptions towards women participating in political life, IFES identified NGO leaders with strong networking skills. As like-minded groups were identified in the course of its regional roundtable discussions, IFES suggested the possibility of formalizing their relationship into a registered NGO. IFES provided training on proposal writing, communication, leadership, and financial and administrative management. With 19 NGO members throughout Georgia GEN has been tremendously effective in mobilizing women to participate in local politics. Similarly, IFES provided the impetus for the formation of JNGE—a coalition of more then 35 journalists throughout Georgia--in September 2013. The formation of these effective coalitions is an unexpected benefit to the CEPPS programming, attributable to IFES’ support and vision. QUESTION 4: WHICH FACTORS CONTRIBUTE TO THE SUCCESSFUL ELEVATION OF LOCAL ORGANIZATIONS INTO SUSTAINABLE PROVIDERS OF TECHNICAL ASSISTANCE RELATED TO ELECTIONS AND POLITICAL PROCESSES? To what extent have CEPPS local and regional partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? Civil society in the electoral sphere in Georgia is dominated by the large and highly capable NGOs ISFED, GYLA, and TI. Each of these NGOs has worked extensively with CEPPS partners, and readily credit them for helping develop their technical and organizational capacity. From diversifying their funding base to collaborating and coordinating with other stakeholders, these organizations exhibit a variety of best practices. In addition there is a wide variety of NGOs participating in the electoral and political processes sector, advancing the interests of women, people with disabilities, and ethnic FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 22 minorities. CEPPS partners have also worked to ensure that these organizations have the opportunity to develop, using best practices ranging from networking to strategic planning, organizational governance, and leadership. Which factors contribute to the successful elevation of local organizations into sustainable providers of technical assistance related to elections and political processes? Georgia provides several examples of local NGOs crossing over from subgrantees to sustainable providers of technical assistance in the elections and political processes sphere. Besides the standard minimum requirements of financial and administrative capacity, and demonstrated technical ability, the Georgian recipient organizations identified a couple of additional factors: Timing: ISFED described a “right time, right place” scenario in receiving its first direct subgrant. Initially international donors had not planned to observe these elections. However, as the political situation intensified, USAID sought to engage ISFED directly to organize an observation mission and conduct PVTs in critical locales.29 Although initially hesitant, in discussion with USAID and NDI, ISFED concluded that they were capable of implementing the grant. ISFED described having to strengthen their financing and internal policies, such as revising procurement and human resources manuals in order to meet USAID standards. To address the one area where ISFED initially felt insecure, conducting the PVTs, it gave a subgrant to NDI for expert assistance. Unique skills: Both GYLA, who received funding from USAID for the first time in 2005, and TI, which is applying for a grant under the current APS, mentioned having developed a unique skill set as key to their marketability to donors. GYLA was identified by several stakeholders as having carved a niche for itself in monitoring the efficacy of Georgia’s electoral complaint resolution mechanisms. A network of lawyers reaching most regions of Georgia, GYLA was described as having the capacity to “swoop in” quickly to regions experiencing high levels of complaints. Similarly TI developed a unique methodology to monitor use of administrative resources, with the assistance of IFES. This activity is consistent with the organization’s broader anticorruption mission, but expands its application to the electoral field. Unique constituency: Two other organizations that have received grants directly from USAID and other donors point to their ability to serve a specific constituency. PMMG advocates on behalf of 19 different ethnicities in Georgia. Similarly, CIL recognizes that its niche as an advocate for people with disabilities has been attractive to several donors. CIL has 26 NGO members representing the spectrum of disabilities. CIL has been enormously successful in attracting funding,30 but credits its affiliation with IFES to their expansion into the electoral spectrum as well as contributing to their organizational development. What efforts on behalf of CEPPS have been made to support local organization opportunities for becoming direct recipients of donor funding? Most of the local organizations interviewed credited each of the CEPPS partners with helping them prepare for and find opportunities to become direct recipients of donor funding. As described above, NDI groomed ISFED over the course of many years, on both skills capacity and organizational 29 Although the USAID representatives interviewed indicated that ISFED had approached USAID with an unsolicited project proposal, ISFED recalled it differently, specifically recalling 2 USAID officers who made the initial inquiry. However, as neither of these officers were still with USAID Georgia at the time of the field study, it was not possible to verify this, other than a categorical surmising that they would not have solicited a proposal. 30 CIL has received two $600,000 awards from USAID Washington and also gets funding from Save the Children. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 23 development. Even after ISFED became a direct recipient, NDI supported them with substance matter expertise.31 IFES has provided trainings on key organizational development issues, in addition to helping its grantees improve their technical skills. Although IRI has been less involved with civil society given that its scope of work relates almost exclusively to political party building, the NGOs that had received grants from IRI reported improving their technical skills as well. Irrespective of the degree of their organizational development, several of the Georgian NGOs identified an impediment to having local organizations implement programs that are politically sensitive. For example, although CIL has very capable advocates, they report that “without IFES our voice would not have been heard,” explaining that an international organization was still needed to get the attention of the Central Election Commission. Similarly the International Centre for Civic Culture (ICCC) said that working on reforms to the electoral code, “IFES was instrumental; by getting us in with the interfactional group and helping us to be heard by the parliament…when it is a political necessity we need some international support to be heard by the government.” Although not a question of organizational development, to the extent that USAID and CEPPS are able to encourage their institutional partners and stakeholders to engage with local CSOs substantively, the impact of the capacity development work will be multiplied. QUESTION 5: WHAT EFFORTS ON BEHALF OF CEPPS HAVE BEEN MADE TO SUPPORT LOCAL ORGANIZATION OPPORTUNITIES FOR BECOMING DIRECT RECIPIENTS OF DONOR FUNDING? To what extent has CEPPS made efforts to assist local organizations become prepared to apply and receive larger and more complicated awards? What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative and project management practices)? While this was never explicitly part of their respective program descriptions, the CEPPS partners have all been willing to assist their CSO partners in pursuit of grants in accordance with their organizational capacities. For example, the extensive support to ISFED, TI, and GYLA has been detailed throughout this report. The predominance of these organizations, however, has had a tendency to crowd out smaller and less developed organizations from the electoral sphere. Both USAID and CEPPS partners, particularly IFES, have sought ways to help these organizations get established as well. CEPPS partners have helped newer organizations, such as GEN, apply for and receive grants from the Central Election Commission (CEC) to conduct observation of a particular aspect of the elections (such as the minority vote) or to consult on draft reforms to the electoral code. As these grants tend to be under $25,000, they are of less interest to the larger organizations. Similarly, USAID specifically encouraged organizations with capacity to observe on behalf of a particular constituency (ethnic minorities, women, people with disabilities) to apply for funding under its APS. By helping partners find funding commensurate with the expertise and capacity, CEPPS and USAID are helping the civil society sector grow organically. ISFED presents a different type of problem with respect to sustainability. While the organization is well￾developed and highly capable, as a domestic election monitoring organization, its substantive niche is quite narrow. Although funding is abundant during election periods, to be truly sustainable ISFED must find relevant activities during years in which there are no elections. NDI has supported ISFED’s participation in regional election observation missions, as a member of ENEMO, as one means of boosting its activity 31 There appears to be some ambiguity as to whether NDI’s continued expertise on the conduct of a PVT is being utilized as a matter of reassurance or necessity. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 24 level on a sustained basis. ISFED is also starting to look at its potential role in other strategic areas such as monitoring hiring and recruitment practices in local governments and legal aid.32 CONCLUSION/SYNTHESIS OF FINDINGS Through distinctly different approaches each of the CEPPS partners has contributed to the development and growth of civil society in the electoral and political spheres. The emergence of an elite election monitoring organization, capable of organizing a national observation mission and conducting PVTs, is directly attributable to NDI. Capacity to observe other elements of elections, including minority vote, access and inclusion issues, use of administrative resources, and compliant resolution, has also been developed through the tutelage and support of the CEPPS partners. However, having developed a few strong organizations in some ways is preventing the development of others as the established NGOs are most attractive to donors. Balancing the desire to develop the skills and organizational capacity of CSO partners against the benefits of a diverse civil society is therefore a challenge in Georgia. A second obstacle to a fully vibrant and impactful civil society sector is the lack of respect from governmental entities. CEPPS partners are working with their institutional partners to forge a constructive relationship with local NGOs, even absent the participation of an international organization. Although the goal of building organizational capacity is not explicit in the PDs under CEPPS III, USAID’s support of this objective is evident in its commitment to direct funding of former CEPPS partners. 32 ISFED is providing legal assistance to victims of wrongful dismissal from civil service with support from the Dutch embassy. FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 25 APPENDIX I – PEOPLE INTERVIEWED USAID David Stonehil, Director of Office of Democracy, Rights, and Governance Ninka Bauchidze, AOR of CEPPS programming Kristin Shouba, alternate AOR of CEPPS programming IFES Nermin Nišić, Chief of Party IRI Andrea Keerbs, Resident Director Vladimer Razmadze, Senior Program Officer Nino Dolidze, Program Officer NDI: Luis Navarro, Resident Director (2009-2014) Laura Thornton, Resident Director Tamara Sartania, Parliamentary Program Manager Maka, Program Officer Local Partners Center for Development and Democracy: Ketevan Chachava, Chairperson Coalition for Independent Living (CIL): Gia Dzneladze Gender Equality Network (GEN): Maia Kuprava-Shavashidze, Coordinator Georgian Foundation for Strategic and International Studies (GFSIS) Ekaterin Metreveli, Acting Director Ketevan Emukhvari, Project Coordinator Georgian Institute of Politics: Kornely Kakachia, Director Georgian Young Lawyers Association (GYLA): Irma Pavliashvili, Election Projects Lawyer Human Rights Center (HRIDC): Aleksandre Tskitishvili, Executive Director Nino Tsagareishvili, Co-Director International Centre for Civic Culture (ICCC): Kote Kandelaki, Director International Society for Fair Elections and Democracy (ISFED): Nino Lomjaria, Executive Director; FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 26 Tamar Bartaia, Deputy Executive Director, Mikheil Benidze, Programs Manager; National Council of Youth Organizations of Georgia (NCYOG): Ekaterine Tsvariani, Chairperson New Generation-New Initiative (NGnI): Mikheil Devdariani, Executive Director, Ekaterine Tsvariani, Deputy Director Public Movement Multinational Georgia (PMMG): Arnold Stepanian, Chairman Transparency International Georgia (TI): Levan Natroshvili, Projects Manager FINAL REPORT: PERFORMANCE EVALUATION OF CEPPS III: APPENDIX | 27 DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY GUATEMALA CASE STUDY TASKING N019 PERFORMANCE EVALUATION OF THE CONSORTIUM FOR ELECTIONS AND POLITICAL PROCESSES STRENGTHENING III LEADER AWARD Contract No. GS-10F-0033M/AID-OAA-M-13-00013 September 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or Government __ DRG-LER 28 INTRODUCTION NORC at the University of Chicago is conducting an evaluation of the CEPPS III Leader With Associate (LWA) instrument pursuant to the DRG Learning, Evaluation, and Research activity. The purpose of the overall study is to gain a better understanding of how local CSOs are impacted organizationally by their association with CEPPS partners and the extent to which the inclusion of Objective 10, relating to building CSO capacity, impacted program design under CEPPS III. Spanning 92 awards in 52 countries, the evaluation is informed by a document review, electronic surveys of USAID representatives, CEPPS partner field officers, and local NGOs working in the electoral processes field, as well as key informant interviews with home office personnel of the three CEPPS partners. In addition, case studies in six countries with programming under CEPPS III were selected. The case studies were selected based upon the amount of programming under CEPPS III in a particular country and the degree to which programming involved civil society. Controls to ensure geographic diversity and programming from each of the CEPPS partners were also applied. Guatemala was selected from the LAC region based upon the fact that there have been 3 projects under CEPPS III, as well as prior programming under CEPPS II; several CSOs and CSO networks are recipients of subgrants through CEPPS programming; and 3 objectives relate specifically to strengthening CSOs participating in political and electoral processes. The field research in Guatemala was conducted August 15-22, 2015. Resident Country Directors and programming staff from each of the three CEPPS implementing partners were interviewed, as well as representatives of 15 NGOs, one media company, and one local government official. Finally, from USAID, the CEPPS AOR, the DRG Office Director, and 2 representatives from the Contracting Office gave interviews. GUATEMALA COUNTRY CONTEXT Emerging from an extended civil war, Guatemala has struggled with corruption, organized crime, and ongoing instability and violence. USAID started working in Guatemala in 1996, helping implement the peace accords; presently USAID’s portfolio continues to promote citizen security, but also includes programming in agriculture, health, and education. Elections in 2007 demonstrated the need for electoral and political reforms and USAID responded with a program in 2008 under its CEPPS II LWA, implemented by NDI. Electoral programming continued under CEPPS III in preparation for the 2011 elections, and has been ongoing in advance of the anticipated elections in September and October 2015. There have been three projects in Guatemala under CEPPS III: “Strengthening of the Guatemala Supreme Electoral Tribunal to Conduct the 2011 General Elections” September 29, 2010 – November 30, 2012, by IFES with a budget of $2.7m; “Building Confidence And Participation In The Guatemalan Electoral Process” May 18, 2011 – July 31, 2012, by NDI with a budget of $1,005,945; and Elections: more Inclusion, Less Violence”33 August 16, 2012 – January 31, 2017, by IFES, NDI, and IRI with a total budget of $7.9m. Election Administration: CEPPS, through IFES, has provided technical assistance to the Supreme Electoral Tribunal (TSE) since 2010. Support has included software for the voter registration process and for transmission of preliminary electoral results; communication support for voter education; technical assistance for campaign finance oversight, election violence prevention and response, and inclusion of 33 The ceiling was increased by $1,500,000 in early 2014, and the original project name “Support Electoral and Political Parties Reform and Strengthen the Guatemalan Congress” was changed by USAID on December 30, 2014. 29 people with disabilities. CSO engagement was primarily related to inclusion of people with disabilities. In this regard, IFES worked through CONADI, the national council for assistance to people with disabilities, which has 59 CSO members. With assistance from IFES, CONADI noted that ramps were installed in specific polling stations, together with special tables accommodating the height of wheel tables for the 2011 elections; braille ballots were also made available. Public information campaigns were held and via a memorandum of understanding with the TSE, training for polling station commissioners on their responsibility to accommodate disable voters, was held. IFES has not been as involved with CSOs in preparation for the 2015 election, though advocacy efforts on behalf of the disabled continue via CONADI and its member organizations. Electoral Monitoring 2011: Under its first award under CEPPS III, and previously under CEPPS II, NDI worked to strengthen the technical capacity and coordination of national observation groups to organize and conduct a range of election observation activities. NDI’s primary partner, dating back to 2008, is Acción Ciudadana (AC), which is the Guatemalan affiliate of Transparency International. In advance of the 2011 elections, NDI worked extensively with AC, supporting its conduct of a “quick count.”34 Additionally, NDI provided skills training on observation methodology, report writing and public information strategies to 6 CSOs35 participating in electoral observation. NDI partner CSOs received manuals detailing international best practices for election observation, as well as individual help honing their respective approaches to the anticipated challenges of the 2011 campaign. NDI also provided assistance on preparing post-electoral media statements and reports to the consortium of CSOs participating in the broader domestic observation network, Mirador Electoral. After the first round of elections in 2011, NDI conducted trainings on political financing and electoral systems as well as inclusion of women and indigenous people in the electoral and political process for 6 CSOs36 observing the second round elections. Electoral Reform: The need for substantial reform of the electoral laws has been a common refrain among international donors for the past 15 years. A significant reform of the Ley Electoral y de Partidos Políticos (LEPP) took place in 2004, though local and international observers urged further reforms to LEPP as well as to the law on political parties following both the 2007 and 2011 elections. Accordingly, under its CEPPS agreement, NDI helped civil society productively advocate for electoral reforms following the 2011 elections. To this end, NDI established a coalition of CSOs, Convergencia Nacional por la Reforma Politica (CONAREP)37 advocating for electoral reform, including a quota for women and indigenous people. Workshops on the proposed reforms were provided to 4 women’s organizations forming the Article 212 Group.38 NDI continued working with AC to conduct research informing the reform proposal by CONAREP. Finally, NDI facilitated a series of electoral forums for civil society organizations39 to present 34 AC was implementing grants from NDI, as well as one directly from USAID in 2011. This is discussed further in the findings section. 35The 6 CSOs, who received funding from UNDP and technical assistance from NDI, are: Asociación para el Desarrollo, la Organización, Servicios y Estudios Socioculturales, DOSES; Facultad Latinoamericana de Ciencias Sociales, FLACSO; Instituto Centroamericano de Estudios Políticos, INCEP; Red Nacional de Organizaciones de Jóvenes Mayas, RENOJ; Organismo Indígena para la Planificación del Desarrollo—Naleb; and Universidad Rafael Landívar, URL 36 Five of the six NGOs receiving technical assistance from NDI in the first round plus a women’s organization, MMMP. 37 CONAREP members included: AC; the Article 212 Group (consisting of Asociación Mujeres Vamos Adelante, AMVA, Convergencia Cívico Política de Mujeres, CCPM, MMMP, and MOLOJ); Instituto Centroamericano de Estudios para la Democracia Social, DEMOS; DOSES; FLACSO; Instituto Maya de Investigación y Desarrollo, IDEMAYA; INCEP; MMMP; Coordinación de ONG y Cooperativas, CONGCOOP; Consejo de Instituciones de Desarrollo, COINDE; Instituto de Enseñanza para el Desarrollo Sostenible, IEPADES; Incidencia Democrática, IDEM; and Movimiento Humanista Democrática, HMD. 38 Article 212 refers to the article of the LEPP, relating to quotas; its members are listed above. 39 CSOs making presentations included NDI’s observation partners, AC, FLASCO, URL, and Naleb, its partner women’s organizations, MMMP and the Article 212 Group, and DEMOS from CONAREP. Also involved were Movimiento Civico Nacional (MCN) GuateCívica, Instituto Internacional de Estudias de la Mujer (INTERDEM) Instituto Universitario de la Mujer, Universidad de 30 their perspectives to the Congressional Committee on Electoral Affairs, as well as helping CONAREP prepare a pamphlet regarding these reforms. Unfortunately, there was little consensus among civil society leaders regarding the nature the reforms should take, affording Congress a convenient excuse to delay the issue further. Mitigating Violence in 2015 Elections: IRI came to Guatemala with the third award under CEPPS III in August 2012. The original thrust of this project was to continue promoting electoral reforms through civil society and also working directly with Congress. However, after more than a year of gridlock in Congress, USAID/Guatemala and the CEPPS partners began to discuss amending the Program Description (PD) in March 2014.40 The amended PD shifted the focus away from electoral reform, emphasizing the need to mitigate electoral violence. The PD anticipates significant civil society and media involvement in monitoring campaign violence and illicit financing, and inclusion of women, specifically including transgender women, in electoral and political processes. This PD also continues IFES’ work with the TSE, though the work with CONADI regarding the rights of disabled persons was not included. IRI and NDI share the work with civil society, with NDI continuing to work on observation efforts with AC and the members of Mirador Electoral, and IRI building media capacity to cover electoral processes, and strengthening the capacity of CSOs representing marginalized constituents to engage in the electoral and political processes. Via a subcontract with Laboratorio de Medios IRI has developed a 10-module certificate course on ethical political and electoral journalism, which specifically includes a focus on LGBT rights and inclusion; 7 modules have been delivered to journalists in the regions targeted under CEPPS and to members of the LGBT community. In addition, CEPPS conducted an assessment of LGBT organizations gauging their interactions within civil society as well as with the government, in addition to skills and organizational needs. Following the assessment, CEPPS has supported public information campaigns on the rights of the individuals in the LGBT community, as well as the activity Café Diverso, which engages politicians in an informal discussion with CSOs about LGBT rights. CEPPS has helped its LGBT partners successfully advocate the TSE to include a section in its training manual on the rights of transgender people to vote. Further towards the prevention of electoral violence, IRI is working with its several grassroots organizations in the Quetzaltenango vicinity. Tapping into its Women’s Democracy Network, IRI promoted women’s rights and participation in the general elections through a series of trainings on leadership as well as political and electoral violence for members of the Guatemala chapter of this Network. Finally, IRI has started conducting capacity building trainings, including organizational monitoring and evaluation and project management. NDI continued working closely with AC, providing training on strategic planning and helping it design methodologies to organize a coalition of CSOs to monitor electoral violence and campaign spending. CASE STUDY METHODOLOGY This case study is informed by a review of CEPPS III programming documents, key informant interviews with USAID personnel as well as in-country staff of the 3 CEPPS partners, and observation of 1 training session. Further, significant input from CSOs affiliated with different aspects of CEPPS III programming San Carlos de Guatemala, (IUMUSAC), Mujer y Poder (MP), Asociación de Capacitación y Asistencia Técnica en Educación y Discapacidad (ASCATED), Asociación Nacional de Ciegos de Guatemala (ANCG), Centro para la Defensa de la Constitución de Guatemala (CEDECON), Consejo Nacional de Atención al Migrante de Guatemala (CONAMIGUA), Coalición Nacional de Migrantes Guatemaltecos (CONGUATE), Grupo Articulador Sobre Migraciones (GASM), Youth Committee on Political Parties, CACIF, Indigenous Observatory, and ASIES. 40 This PD underwent several modifications, most of which were a subject of considerable consternation between USAID and the CEPPS partners. The most recent modification was completed March 2015. 31 was solicited through focus groups and individual interviews. Finally, some independent research on documents and events referenced in the course of the interviews was conducted. Document Review: Prior to commencing the fieldwork, the following programming documents were reviewed: Program Descriptions: 4 program descriptions, encompassing the 3 programs; Final Reports: final reports from the first 2 projects under CEPPS III; and Quarterly Reports: the current project’s report covering January – March 2015; News reports: various sources regarding CISIG, the political crisis, and violence in Guatemala; Surveys and Assessments: Policy Briefing: Clean Polls, Dirty Politics, June 2011: http://www.crisisgroup.org/~/media/Files/latin￾merica/B24%20Guatemala%20%20Clean%20Polls%20Dirty%20Politics.pdf; Electoral Security Assessment, Guatemala, April 2012: http://41pylqn86jp37e3n04us8vqq.wpengine.netdna-cdn.com/wp-content/uploads/2003/01/Creative￾Guatemala-Electoral-Security-Assessment-Final.pdf Key Informant Interviews: In depth interviews were held with the Director of USAID’s office of Democracy, Rights, and Governance, as well as with the Agreements Officer’s Representative (AOR) responsible for managing the CEPPS project. Additionally, USAID Guatemala’s Contracting Officer and Contracting Specialist handling the CEPPS agreement were interviewed separately. Resident Country Directors and senior staff from each of the 3 CEPPS partners were also interviewed in depth at the beginning of the field research. Focus Group: A focus group was held with the 5 members of the election observation coalition, Mirador Electoral, jointly supported by SIDA, USAID, and the Norwegian Embassy. Partner Interviews: Interviews were conducted with 15 representatives from 12 CSOs collaborating in different ways with CEPPS partners. Two CSOs were subcontracted to develop and conduct training; 1 long-term partner was responsible for organizing long and short-term observation missions; 3 organizations participated in trainings supported by CEPPS partners; and 6 were subgrantees. Personal Observation: An introductory training on Monitoring and Evaluation for 7 CSOs in the Quetzaltenango region was observed on August 20, 2015. ANSWERS TO EVALUATION QUESTIONS QUESTION 1: ARE THERE NOW MORE LOCAL CIVIL-SOCIETY ORGANIZATIONS ACTIVE IN: A) ELECTIONS ASSISTANCE; B) POLITICAL PROCESSES; AND C) PROVIDING TECHNICAL ASSISTANCE TO POLITICAL PARTIES THAN THERE WERE SIX YEARS AGO? Civil society began to flourish after the peace accord in 1996, when international donor funds supporting peace and security began to flow. Presently, however, civil society seems to be constricting as several donors have cut back on their activities in Guatemala.41 Significantly, however, despite the overall 41 This, however, is likely to change soon as there is an anticipated influx of donor money, especially from USAID as Guatemala begins tackling its corruption problem. For example, USAID announced a new $40m program to reduce gang violence in Guatemala City, half of which will be CSO grants. In addition, the Obama administration has requested up to $1 billion to fund 32 reduction in CSOs generally, the unanimous conclusion among stakeholders was that there were more local CSOs active in both electoral assistance and political processes than there were six years ago. Prior to the 2008 elections, election observation was less organized, less capable, and involved fewer CSOs. Similarly, CSO activity from the youth, indigenous, women, and LGBT communities was characterized more by apathy than by action. As recently as the 2011 elections, Guatemalans demonstrated a disturbing lack of confidence in electoral processes: 42% of respondents in a public opinion poll reported having little or no confidence that the results of the first round represented the collective will of the people.42 The lack of confidence threatened participation in the electoral and political processes, particularly in rural indigenous communities. From that nadir, CSO have been provoked into action by persistent crime, corruption, and instability. Presently CSOs representing diverse people from rural indigenous youth and women to transgender people to people with disabilities are actively working to demand that their suffrage rights be respected, encouraging their constituents to vote, engaging in the political processes, and demanding reform of the political systems. The surge in civic activity in electoral processes started in April 2015, when the international prosecutors’ panel (CISIG) uncovered a complex scheme of money laundering and customs fraud (la Linea) involving more than 150 government officials, including the President and then-Vice President. Impeachment proceedings against the President and Vice President were started, leading to the resignation of the Vice President in May 2015; the call for impeachment of the President did not pass Congress in May, although a new hearing was commenced following the arrest of the Vice President on August 21, 2015. Details of this scandal galvanized civil society, who have organized ongoing protests on successive Saturdays in Guatemala City and select other cities. In addition to organizations dedicated to electoral activities and/or promoting transparency generally, CSOs representing marginalized segments of the population (including people with disabilities, indigenous peoples, LGBT, youth, and women) have been more active in advancing the electoral and political rights of their respective constituencies. Assistance to political parties, on the other hand, is on the decline. Political parties in Guatemala are very fluid, if not wholly illusory entities, with candidates frequently switching parties, sometimes repeatedly within a single term. No incumbent party has been able to win re-election and most parties exist for only a few years. Consequently, assistance going to political party development is predominantly (if not exclusively) from German entities such as the Konrad Adenauer Stiftung, which work with ideologically aligned parties in other countries. However, even this form of partisan assistance is declining as political parties are increasingly infiltrated by criminal elements.43 If there are more CSOs in the three above evaluation-question areas today than six years ago how many of the newer organizations worked with CEPPS institutions or their local partners? Due to the lack of data documenting the volume CSO engagement in electoral processes, it is problematic to determine with specificity how many of these organizations have worked with CEPPS institutions and/or their local partners. However, consistent with requirements in the PD amendments of 2014, CEPPS has engaged more than 10 new CSO partners. the Alliance for Prosperity in the Northern Triangle (Guatemala, Honduras and El Salvador); Vice President Biden made it clear, however, that this funding will only be available Guatemala keeps the UN Commission on Impunity (CISIG). 42 NDI public opinion survey, reported in CEPPS Final Report, Guatemala: Building Confidence and Participation in the Guatemalan Electoral Process, ending June 31, 2012. 43 http://www.crisisgroup.org/~/media/Files/latin-merica/B24%20Guatemala%20%20Clean%20Polls%20Dirty%20Politics.pdf 33 Believing civil society in Guatemala to be very stratified, and keenly aware that AC was already a sustainable entity, USAID sought to have the CEPPS partners engage with new CSOs to give other organizations a chance to benefit from the skills and organizational training. CEPPS strongly advocated for continuing with AC for the long-term observation in the municipalities because it had an established methodology, but agreed to work with AC to recruit new organizations into their network. A compromise was struck that AC would lead the observation, but that 6 of the 10 organizations engaged would be new to AC’s network. When, through a modification to the PD, the number of municipalities was doubled (from 10-20), they agreed to strive for 60% new CSO partners, understanding that some CSOs would cover more than one municipality. In particular, USAID encouraged CEPPS to engage traditionally underrepresented groups such as women and indigenous populations to the greatest extent possible: “[t]he Project should focus on enabling coordination among several CSOs, including grassroots CSOs, to create a comprehensive monitoring effort…”44 Ultimately 7 out of 13 organizations selected by AC are new to their network. IRI has also engaged several new CSOs as it began working on the inclusion issues with CSOs representing women, youth, indigenous people, and LGBT individuals. How were programmatic strategies developed under CEPPS II altered or expanded (e.g., different approaches to activities, the use of sub-grants) under CEPPS III in order to address local capacity building goals under Objective 10? Although some of the programming under CEPPS III included a greater emphasis on strengthening the capacity of CSOs to “prevent electoral violence”, to “promote and implement key [electoral] reforms”, and to “participate” in the electoral and political processes45, it was not prompted by the addition of Objective 10 in the CEPPS III LWA. Rather, a confluence of independent circumstances created the impetus to expand civil society engagement under CEPPS: the political situation in Guatemala; the unexpected discontinuation of a dedicated civil society program; and the availability of funds through Women’s Peace and Security program. As outlined above, programming under CEPPS III initially focused on building the capacity of the TSE to administer the elections. Civil society engagement was limited to the issue of incorporating appropriate accommodations to ensure that people with disabilities could, and would, exercise their right to vote. More emphasis was placed on civil society engagement with the second CEPPS III PD. NDI provided technical skills training and assistance to its primary CSO partner, Acción Ciudadana, helping it develop its “quick count” vote verification methodology. However, it should be noted that by that time AC had already passed the Non-U.S. Organization Pre-Award (NUPA) Survey and attendant audits and was a direct recipient of USAID funding, consistent with the USAID Forward goals.46 The third PD under CEPPS III, issued in 2012, sought to provide technical assistance to reform the LEPP and strengthen Congress’ oversight capacity. These envisioned activities, however, were premised upon the critical assumption that political will existed in Congress. Recognizing that those objectives were essentially unattainable “in part due to paralysis in the Congress” USAID amended the PD to adopt a “targeted approach focusing on mitigating political violence and illicit sources of campaign financing.”47 Moreover, around the same time USAID’s civil society capacity building project had terminated prematurely due to financial control issues with the U.S. implementer. Consequently, USAID opted to 44 Id. 45 Revised Program Description, 2014. 46 AC was implementing an anti-violence project, using the Ushahidi application to map incidents of violence throughout Guatemala. 47 USAID, amended PD, 2014. 34 use the CEPPS LWA as a way to salvage its relationship with civil society organizations affiliated with the curtailed project, as well as to continue its institutional work with the TSE. Finally, a fourth objective was added to the third CEPPS PD in 2014, to strengthen cisgender and transgender women’s political participation. The impetus for this objective came from the transgender community itself. Having previously benefited from assistance via USAID’s Health Office on HIV issues, the community raised its need for technical assistance in the democracy sphere as well. As USAID was in the midst of preparing an application for additional funds through the Women and Security fund to promote greater inclusion of women in the electoral processes, it specifically added transgender women as a sector whose civic rights are often impeded. Accordingly, with additional funding for Women and Security, CEPPS partners took on the challenge of promoting women’s participation, including transgender women, in the electoral process, and as candidates. Due to the relative scarcity of CSOs representing the transgender community, the PD was later expanded to also include organizations representing lesbian or lesbian, gay, and bisexual individuals.48 QUESTION 2: HOW WERE PROGRAMMATIC STRATEGIES DEVELOPED UNDER CEPPS II ALTERED OR EXPANDED (E.G., DIFFERENT APPROACHES TO ACTIVITIES, THE USE OF SUB-GRANTS) UNDER CEPPS III TO ADDRESS LOCAL CAPACITY BUILDING GOALS UNDER OBJECTIVE 10? Was the CEPPS process for local partner selection optimal in the sense that competitive selection processes were balanced against the CEPPS need for establishing ongoing relationships? Competitive selection process has not been used in any of the CEPPS programs in Guatemala. Instead, there has been greater emphasis on promoting networks of organizations—specifically those organized by AC, and also operating under the Mirador Electoral rubric. As discussed above, there is some difference of opinion within USAID about the appropriateness of continuing to engage AC as the lead partner. However, by requiring that new members be brought into the network, the anticipation is that AC will share its skills and capacity among a broader array of CSOs. With respect to CEPPS’ work with women and the LGBT community, IRI conducted an assessment addressing the capacities of LGBT organizations throughout the country. It became apparent, however, that the two leading transgender networks, Red Multicultural de Mujeres Trans (REDMUTRANS) and OTrans, are fiercely competitive with one another. Given that one of the CEPPS staff was affiliated with REDMUTRANS, the other transgender organization has not expressed interest in participating in the program. Accordingly, IRI has sought to engage other LGBT organizations based upon their interest level and the relevance of their activities. To what degree did CEPPS involve local partners in the design, awards management, M&E processes, and reporting of programs with which they were affiliated? USAID did not seek input directly from CSOs in the preparation of the PDs or amendments thereto, with the exception of the addition of work with the LGBT sector, which was demand-driven, as described 48 Given that this objective is funded through the Women and Security Fund, CEPPS works with LGBT organizations that involve transgender and/or lesbian women, and not with organizations that exclusively serve gay or transgender men. 35 above. However, the PDs were informed by 2 studies that incorporated feedback from civil society. The first was an internal evaluation of a previous CEPPS project, and the second was an external study on political violence.49 CEPPS partners also incorporated feedback from civil society in its project design, to varying degrees. For example, prior to the start of its first award under CEPPS III, NDI performed a needs assessment looking at what the individual CSOs working in electoral monitoring needed in terms of skills training. Similarly, at the end of its first project under CEPPS III, NDI organized a series of discussions with partner groups on lessons learned/best practices in supporting domestic election observation projects; presumably the insights from these discussions fed into their project proposal in response to the subsequent PD. Finally, with the introduction of the objective to promote participation of cisgender and transgender women in the electoral and political processes, IRI sought input from REDMUTRANS for its project design. QUESTION 3: TO WHAT DEGREE DID CEPPS INVOLVE LOCAL PARTNERS IN THE DESIGN, AWARDS MANAGEMENT, M&E PROCESSES, AND REPORTING OF PROGRAMS WITH WHICH THEY WERE AFFILIATED? Did CEPPS integrate organizational capacity-building activities into their collabora￾tions on elections and political processes, or did the activities occur on more of a parallel track? In general organizational capacity building activities tended to be incorporated into the technical assistance provided to partner organizations. For example, NDI worked intensively with AC on its communication and leadership skills in managing a network of CSOs. Similarly, all of the CEPPS partners worked with their respective partner CSOs to ensure that they could adequately account for grant money and prepare sufficient narrative reports. However, some capacity￾building activities took place independent of an existing subcontract or grant. For example, IRI conducted an introductory training on Monitoring and Evaluation for CSOs in the Quetzaltenango area. It should further be noted that much of the organizational capacity building conducted by IRI is not within the CEPPS programming, as it is supported by NED funds. Some of the same CSO partners participate in both the National Endowment for Democracy (NED) and CEPPS programming. QUESTION 5: WHAT EFFORTS ON BEHALF OF CEPPS HAVE BEEN MADE TO SUPPORT LOCAL ORGANIZATION OPPORTUNITIES FOR BECOMING DIRECT RECIPIENTS OF DONOR FUNDING? To what extent have CEPPS local and regional partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? Civil society in Guatemala varies from the most sophisticated to the most rudimentary organizations, both of which work with CEPPS partners. AC, for example, has adopted and incorporated a variety of best practices, with ongoing assistance from NDI over a period of 49 Electoral Security Assessment, Guatemala, 2012 http://creativevoces.com/wp-content/uploads/2012/10/Creative-Electoral￾Security-Assessment-GUATEMALA.pdf 36 several years, and has successfully marketed these skills to other donors including the European Union and NED. Many of the rural organizations, representing youth and indigenous people, however were eager for trainings on best practices of all sorts. They consistently reported utilizing the skills and practices learned in their activities. For example, Pop Noj, one of the rural youth organizations, reported replicating the training received from IRI for other organizations in the communities they serve. Which factors contribute to the successful elevation of local organizations into sustainable providers of technical assistance related to elections and political processes? Sustained technical assistance, including both skills training and organization capacity development, has elevated some Guatemalan organizations into sustainable providers of technical assistance. USAID Guatemala used to have direct grants to 4 organizations, one of which was CEPPS’ long-time partner, AC.50 Recognizing that these organizations still required considerable assistance in managing large donor projects, USAID has a pending award that was originally intended to work with USAID’s direct grantees to strengthen their organizational capacity. However, due to a drastically increased budget, the USAID Guatemala Mission is finding that it lacks the internal capacity to manage these direct grants to domestic organizations. Accordingly, as the existing direct grants end, USAID does not anticipate renewing them, yet the activities will be integrated into larger procurements competed to US organizations. This, combined with the early curtailment of USAID’s prior civil society program, which abruptly cut the funding to several organizations as the program’s subgrants were terminated along with the program, has caused some exasperation within the civil society sector. Additionally, several organizations reported frustration with the degree of reporting and accounting required for USAID grants, as compared to other donors. A distinct preference for funding through the European Union and NED, whose reporting requirements are more manageable, was expressed. Another approach towards strengthening civil society as a whole is the use of networks, linking organizations with similar missions, or engaged in similar activities, together. This approach has been used effectively in Guatemala, with the emergence of strong CSO networks for electoral monitoring, people with disabilities, and women’s rights, all of which have been nurtured through CEPPS III programming. CEPPS is also supporting the development of networks of youth, indigenous, and LGBT organizations. It remains to be seen, however, whether these networks ultimately yield more organizations with capacity to manage and implement donor funds, or merely strengthen the lead organization. In the case of AC’s network, for example, while it is evident that AC has developed strong organizational capacity, the extent to which this capacity building reaching the network members is debatable. While in other countries the network itself may become a registered entity, fortified by its member organizations, this has not happened in Guatemala. Mirador Electoral, for example, is not a registered entity, but an informal structure of organizations that have done domestic election observation in the past. What efforts on behalf of CEPPS have been made to support local organization opportunities for becoming direct recipients of donor funding? Although this was never an explicit part of CEPPS programming in Guatemala, it is evident that the CEPPS partners have helped prepare key local partners to become direct recipients of donor funding. AC is the most evident example, given its long relationship with NDI. Organizations 50 The other 3 grantees were working in counter trafficking in persons, anticorruption, and youth violence. 37 representing people with disability also speak very highly of the assistance received from IFES in the first program under CEPPS III. The fact that they continue to advocate effectively on behalf of their constituents, despite no longer being a CEPPS grantee, suggests that their organizational capacity has been reasonably well developed. However, with respect to getting direct funding from USAID, given the arduous audit process required and the Guatemala Mission’s recent history of pulling back on its commitment to USAID Forward goals, CSOs reported being more motivated to work with European funders, such as SIDA and the Norwegians, or with NED. To what extent has CEPPS made efforts to assist local organizations become prepared to apply and receive larger and more complicated awards? What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative and project management practices)? USAID/Guatemala readily acknowledges that this has never been part of its CEPPS programming, particularly as there were other programs working on building the capacity of civil society organizations. Nevertheless, CEPPS has helped local organizations learn to network as well as to engage governmental officials. It has facilitated opportunities for its partner CSOs make presentations to other donor organizations working in their field, including SIDA, the Norwegian Embassy, and the European Union. The skills training on electoral observation and vote verification methodology, as well as the workshops on strategic planning, project management, communication, outreach, and monitoring and evaluation, as summarized above, all help prepare local organizations to apply for and implement larger and more complicated awards. CONCLUSION/SYNTHESIS OF FINDINGS CEPPS programming in Guatemala has spanned a challenging period in the nation’s history, and has accordingly had to adapt repeatedly to changing priorities and political realities. It has nurtured the development of domestic electoral monitoring organizations as well as national networks linking organizations advocating for the rights of women, youth, indigenous peoples, and the LGBT community. It has also promoted the engagement of new organizations in the rights sphere, and provided those organizations will help skills training and organizational development assistance. However, these positive findings appear to be incidental to CEPPS’ work on inclusion and election monitoring, rather than prompted by USAID’s intention to emphasize CSO capacity development under CEPPS III. Moreover, inconsistent attitudes towards directly funding domestic organizations by USAID have discouraged CSOs from pursuing these opportunities. 38 APPENDIX I – PEOPLE INTERVIEWED USAID Kerry Monaghan-Hogler, Director of Office of Democracy, Rights, and Governance Shannon Schissler, AOR of CEPPS programming Dion Gilsan, Contracting Officer Indyra Gonzalez, Acquisition Specialist IFES Maximo Zaldivar Calderon, Resident Director Ana Cecilia Diaz Lira, Program Specialist IRI Antonio Garrastazu, Resident Director Diego Palma, Program Officer Jorge Ceballos, Regional Program Officer NDI Eduardo Nuñez, Resident Director Sara Barker, Regional Program Manager Justo Rolando Perez, Program Officer Julio Guillermo Donis, Program Officer Ricardo Marroquin, Program Officer Legia Gonzalez, LGBT Coordinator Local Partners Carlos Alberto Sarti Castaneda, PROPAZ, Mirador Electoral member Patricia Pinto, Director, CODEFEM, Mirador Electoral member Marisabel Grijalva, CODEFEM, Mirador Electoral member Ana Maria Pablo, Director, RENOJ Mirador Electoral member Rafael Poiteval, Director, GuateCivica, Mirador Electoral member Jacquiline Avila, Manager, GuateCivica, Mirador Electoral member Carlos Romero, Secretary, REDNADS (LGBT network) Patricia Vargas, Lawyer, REDNADS Ami Hernandez, REDNADS 39 Sebastian Toledo, Director General, CONADI (umbrella organization for Disable People’s Organization) Cristina Domingo, Coordinator, Pop Noj (Huehuetenango) Florinda Sales, Treasurer, COMUNAJIH Jaime de Leon, honorary president, Gente Joven Anna Chaclan, Monitoring, Gente Joven Jorge Lorenzo, Sociedad Civil de Malacatan Juanna Gomez, Program Manager, Mojo Maya Luis Mario Martinez Turcios, Chief Researcher, INTRAPAZ (research division of Universidad Rafael Landivar (URL)) Marvin Pol, Coordinator of Projects, Acción Ciudadana Oscar Vasquez, General Manager, Acción Ciudadana Ischelle Marsicovetere, Director of Projects, Red Ciudadana Ana Carolina Alpirez, Project Manager, Laboratorio de Medio (Ojo con mi Pisto project) Beatriz Colmenares, Director, Laboratorio de Medio (Ojo con mi Pisto project) Anna LNU, Secretary, COJUSAM 40 DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY NIGERIA CASE STUDY TASKING N019 PERFORMANCE EVALUATION OF THE CONSORTIUM FOR ELECTIONS AND POLITICAL PROCESSES STRENGTHENING III LEADER AWARD Contract No. GS-10F-0033M/AID-OAA-M-13-00013 September 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or Government\ __ DRG-LER 41 INTRODUCTION USAID’s CEPPS III Leader with Associates mechanism has contributed significantly to the capacity of local and regional organizations in Nigeria, although the transition has benefited from a positive enabling environment for Civil Society Organizations (CSOs). The democratic transition in Nigeria has been plagued with fits and starts since the reemergence of civilian government in 1998. The results of the 2011 and 2015 election cycles, however, have provided a source of optimism. Activity funded under CEPPS III, implemented by the National Democratic Institute (NDI), International Republican Institute (IRI), and International Foundation for Election Systems (IFES) from 2010 – 2014 has played an important role in this process. The focus of this case study is on the three implementers and a number of CSOs with which they partnered. They included: NDI – Transition Monitoring Group (election monitoring); NDI and IRI – Youngtars Foundation, Young Initiative for Advocacy Growth and Advancement (Voter Education); IFES - National Association for Peaceful Elections in Nigeria (election violence prevention); and Center for Social Justice (electoral judge training and election finance). These CSOs were among the most important examples of capacity building by the implementers. Relationships continue, although they are no longer funded by CEPPS. The first part of this case study examines the country context in Nigeria. Another section highlights the activities of the implementers and the above-mentioned partner CSOs. The final section explores answers to the evaluation questions in the evaluation concept note, illustrated with examples and quotes from interviews conducted in Nigeria. CASE STUDY METHODOLOGY The NORC Evaluation Team conducted interviews with CEPPS III implementing staff, CSO staff and USAID staff in Nigeria. The qualitative analysis is focused on concrete examples that illustrate in greater detail the process and results of the program in that country. Qualitative Evaluation Approach The case study of the CEPPS mechanism support in Nigeria relies, in part, on qualitative data collection and analysis. Qualitative approaches can address aspects of research questions that are not well suited to quantitative analysis. The consultant conducted key informant interviews (KIIs) with USAID, CEPPS implementing staff and consultants in Nigeria who work directly with local partner organizations, and representatives of local CSOs cooperating with the CEPPS program. He relied on semi-structured, open-ended interview guides for the KIIs. NORC consulted with USAID and CEPPS implementers to schedule interviews and respondents who participated. Reasons for Selection NORC selected Nigeria by looking at the program’s level of activity and results. Nigeria was chosen to reflect diversity of geography, issue-area, budget, and overall project scale among the six total case studies that make up this evaluation. Under the NORC categorization, the Nigeria Enhancing the Credibility of Elections program was listed as a “program with explicit CSO technical capacity building in service provision.” As the following matrix indicates, IRI, NDI and IFES all had CEPPS programs during the 2010 – 2014 period of performance. Currently, only IRI has a CEPPS program. 42 Methodology for Conducting KIIs KII guides for various classes of stakeholders were prepared in advance of fieldwork, outlining the key questions and probes to address during the interview. The interviewer took written notes during the interview, and transcribed them within a day of the interview. Respondents were offered anonymity and the option of speaking “off the record.” Each interview started with structured questions and pre-determined topics and ended with more open￾ended questions that sought to reveal unanticipated topics and themes. Analysis of the KIIs includes a summary of common themes that emerge in response to each topic – highlighting the range of responses and experiences. This was supported by quotes from the participants. While most quotes are cited to individuals, and no sources explicitly requested anonymity, there some instances in which the affiliation rather than the name of the source was cited due to the potential sensitivity of the subject. The site visit in Nigeria was conducted by Jeffrey Swedberg of IBTCI, who spent four workdays in Abuja and met with USAID; the implementers, NDI, IRI, and IFES; and local CSOs TMG, Youngstars Foundation, Youth Action Initiative Africa, National Association for Peaceful Elections in Nigeria, and Center for Social Justice. As part of this case study, the evaluation team reviewed program documents, including the final CEPPS report and other documents provided by USAID, the implementers and partners. NIGERIA COUNTRY CONTEXT Stakeholders involved in democracy promotion in Nigeria were in an upbeat mood in mid-2015. For the first time in Nigerian history, a ruling party willingly ceded power to an opposition party after fair and free elections. During the March 28, 2015 general elections, the People’s Democratic Party (PDP), which had dominated Nigerian politics for 16 years, lost both the presidency and its majority in the National Senate. The All Progressives Congress (APC), a party formed in 2013 largely from PDP defectors, won the presidency as well as 60 seats in the new Senate, leaving PDP with 49 seats. The new President, Mohamadu Buhari, was seen as reformer who gained a popular mandate based on an anti- corruption platform. He defeated Goodluck Jonathan, who conceded defeat and turned over power to another civilian president – a first for Nigeria. Importantly, the 2015 elections reflected a continuing trend. The 2015 election represented a rematch of 2011, in which Jonathan defeated Buhari. It was the 2011 elections that occurred during the CEPPS III program. International observers agreed that both the 2011 and 2015 elections reflected a greater public confidence in the leadership of election authorities, bolstered by an increasing level of engagement by civil society. Nigerian elections have make remarkable progress since 2007, which were marred by “opaque election administration, widespread irregularities, violence and misconduct. Irregularities in 2007 included polling stations that opened hours late, closed early or failed to open at all; lack of secrecy in voting; errors in the voter register; ballot papers that did not include all of the candidates; presidential ballot papers lacking serial numbers; and ballot box stuffing and snatching.”51 The progress made between 2007, 2011 and 2015, was remarkable, as Nigerian officials seemed to take seriously the recommendations of international observation reports. According to interviews at USAID, the current focus on CSOs came in 2010 after the death of President Yar’Adua. “We looked at what 51 NDI, Final Report on the 2011 Nigerian General Elections 43 we could do to change the democratic tide after the failed elections of 2007 and decided to pull CSOs into the process.” THE CEPPS III PROGRAM IN NIGERIA 2010 - 2014 The CEPPS III program in Nigeria focused on the 2011 general elections, as well as numerous local elections. Assistance also focused on a portion of the period leading up to the 2015 elections, but not the 2015 elections themselves. Under CEPPS III, USAID provided assistance to political parties, elections, and civil society. These priorities were equally weighted under CEPPS III, which expired in 2014. Currently, USAID is providing assistance only to political parties under CEPPS III, through IRI. The charts below, excerpted from USAID tracking tables, show the status of the 2010-2014 CEPPS III program highlighted for this case study, and the current follow-on program. Program Period of Performance Implementer IRI NDI IFES Nigeria “Enhancing the Credibility of the 2011 Elections.” IRI #8041 NDI #10526 IFES 014 3/1/10-5/31/14 X X X Nigeria. “Supporting Political Party Development in Nigeria.” IRI # 8047 6/1/14 – 5/31/17 X The CEPPS program “Enhancing the Credibility of the 2011 Elections” (2010-2014) had three sets of objectives as follows52: IFES Strengthen the capacity of the INEC to carry out its mandate; Improve the Electoral Dispute Resolution System; and Increase the capacity of civil society to analyze, monitor, and advocate against election violence. IRI Strengthen the capacity of political parties to more effectively and responsibly participate in the electoral process; Strengthen Party Agents’ capacity to monitor the 2011 elections; Enhance the democratic role of the media during the electoral process; Promote stakeholder dialogues on the electoral process; and Promote international support and interest for a transparent and democratic electoral process in Nigeria. 52 CEPPS Nigeria Final Report, 2010-2014. IFES, IRI, NDI. May 31, 2014. 44 NDI Enhance the capacity of partner civic organizations to more effectively educate voters about the electoral process; Strengthen the capacity of domestic civil society organizations to more effectively observe the electoral process; Strengthen the capacity of domestic civil society organizations to employ advanced statistics-based methodologies, such as parallel vote tabulation (PVT), to increase the credibility of upcoming elections; Promote the transparency and integrity of Nigeria’s electoral process through comprehensive international election observation; and Enhance the capacity of partner civic organizations to advocate more effectively for electoral reform. NDI has worked with civil society in some fashion in Nigeria since the 1999 transition from military to civilian rule. As of 2015, NDI was not implementing a CEPPS program, but it has continued its work with civil society under other funding mechanisms. NDI’s highlighted work in under the CEPPS III program comes under election monitoring, through the Transition Monitoring Group; and under voter education, with the Youngstars Foundation and Youth Action Initiative Africa.53 Transition Monitoring Group (TMG) Given the importance of election monitoring to NDI’s work during the period of CEPPS III, TMG has been NDI’s flagship CSO partnership. This partnership provides a number of important lessons -learned regarding capacity building. TMG has become a critical voice in verifying the credibility of Nigerian elections. At the same time, the group’s participation in the Project Swift Count observation coalition was clouded by accusations of financial mismanagement – a cautionary example of the risks of providing funds to CSOs. TMG was established in 1998 as a non-profit organization and now has over 400 member organizations.54 It has worked with multiple donors, including UNDP, EU and CIDA. The CEPPS partnership with TMG began in May 2010, when NDI facilitated a partnership among four groups interested in undertaking a sample-based observation effort, known as a parallel vote tabulation (PVT) for the 2011 elections. The NDI sponsored coalition was called Project Swift Count (PSC). In addition to TMG, the coalition included the Nigerian Bar Association (NBA), Federation of Muslim 53 NDI’s activities with the voter education CSOs will be described in the section on IRI. 54 Many of the 400 members of the TMG coalition are not election groups. Many are groups focused on public health or other sectors. Member organizations provide man power for observation. Some are small and locally focused, while some are big – like Media Right Agenda or CDD West Africa. Figure 1: Logo of Project Swift Count – NDI CSO coalition – 2010 - 2012 45 Women’s Associations in Nigeria (FOMWAN) and Justice, Development and Peace/Caritas (JDPC). To build the capacity of PSC, NDI led the coalition in exercises about PVT organizational and leadership structures, work plan development, liaising with civil society, and coalition formation. Partners agreed to divide the roles in the project: NBA as the Chair; JDPC as project manager; FOMWAN as home of the secretariat; and TMG as trustee of the fund. As part of its capacity building efforts, NDI shared methodologies from an NDI-sponsored Ghana PVT project, discussed organizational structure and management, strategic planning, and other technical details necessary for a PVT. Key to PSC’s initial success was the relationship with the Independent Nigerian Election Commission (INEC). In 2010, NDI facilitated a meeting with INEC Chairman Attahiru Jega in which the Institute helped to assuage concerns that the PSC would release election results prematurely. NDI explained that it would only release estimates based on its statistical sample, and that the findings would be released after INEC’s. Media and stakeholder outreach remained a major priority for NDI and PSC. The coalition launched websites and organized press conferences in an effort to educate the general Nigerian public about PVTs. Turnout at press conferences, however, fluctuated widely. A Financial Scandal for PSC After a successful 2011 election and generally good press, the PSC board was found to have committed financial fraud. The 2014 CEPPS Nigeria Final Report said: “In 2012, NDI discovered that PSC had mismanaged funding. Following an audit of the accounts and discussion with USAID, NDI terminated its relationship with PSC. While all partners had been implicated in the mismanagement, TMG renewed its leadership, distanced itself from the wrongdoing, and complied fully with NDI’s investigations. The Institute therefore agreed to continue working with TMG on a trial basis for the final gubernatorial election in Anambra state, after undertaking a comprehensive review of TMG’s organizational and financial capacity.55 55 CEPPS Final Report: March 1, 2010 – May 31, 2014 NIGERIA: ENHANCING THE CREDIBILITY OF THE 2011 ELECTIONS Figure 2: PVT graphic produced by Transition Monitoring Group – NDI partner 46 According to TMG staff interviewed, the problem stemmed from PSC board members becoming too involved with decisions affecting staff salaries. Board staff demanded that a portion of staff salaries be diverted to support project overhead. “The board hijacked the process,” the TMG staff said, diverting 20% from staff salaries into their own pockets. After PSC administrative staff refused to alter staff vouchers to hide the fraud, NDI discovered the malfeasance during an internal review of the PSC sub￾grant. PSC was disbanded and all eight PSC board members, including two TMG board members, were barred from further participation in NDI and USAID projects. TMG Regroups NDI renewed its partnership with TMG, this time without Project Swift Count coalition, providing technical and financial assistance to conduct a PVT of Anambra State’s 2013 gubernatorial election. Capacity building throughout this process involved refresher workshops. At an October 2013 refresher training session on PVT activities and sub-grant principles, for instance, topics covered included: PVT structure and requirements, project accounting principles, observer recruitment and the development and implementation of a communications strategy. The training session gave NDI an opportunity to share best practices for observer recruitment based on previous PVT undertakings in Nigeria. NDI also assisted TMG to develop its own observer recruitment criteria. In 2014, TMG, with technical assistance from NDI, analyzed and visualized the data generated from the 2011 PVT observation and posted it on its website (see figure 2). TMG sees this site as a source of baseline information for political parties, observer groups, election management bodies and other electoral stakeholders. The PVT convincingly verified results of the 2011 election, and served to establish INCEC as a trusted election arbiter. The visualization and posting of the data on the website in 2014 contributed further to this perception. The relationship between INEC, NDI and TMG was also strengthened when INEC members approached NDI informally to ask about accessing and using the PVT data. Since then, TMG has been presenting PVT data directly to the INEC Chairman. 2015 Activities Although not funded under CEPPS, under the new program in 2015, NDI provided technical assistance to TMG, allowing it to conduct a quick count during the 2015 elections as well as pre-election observation. As part of the PVT, TMG deployed over 4,000 observers to a representative random sample of polling units located in every state and every local government area (LGA) of Nigeria and conducted a quick count of the presidential election. The election process in 2015 was not without fault, and TMG exposed manipulation of the collation process in four states in the South-South geopolitical zone. However, as it had in 2011, the PVT verified INEC’s results in 2015. IRI (WITH NDI) IRI had relationships with around 20 CSOs under CEPPS III. Given the large size and scope of the Nigeria program, interviewing all CSOs was infeasible. Therefore, for the purpose of this case study, the focus is on voter education partnerships that also received funding from NDI so as to focus on the potential synergies of the relationships. Youngstars Foundation The Youngstars are a youth-focused CSO that has worked with both NDI and IRI – at times on the same project. Most of their work involved voter education activities focused on youth, emphasizing the importance of voting as a civic right and responsibility. 47 Music Festivals Among the highest profile events were the 2011, "U Can't Swag My Vote" music festivals, sponsored by IRI in Lagos (March 27), Benue (March 29) and Port Harcourt (March 31). Five Nigerian music stars were enlisted to participate and to carry a message to more than 7,000 youth on the importance of the 2011 elections. IRI collected contact information from more than 5,900 attendees who signed pledge cards. Youngstars sent out 19,185 text messages to remind youths of the pledge they made to vote and to encourage them to report their election experiences by text. They received around 1,700 text messages in return. Work Shops for Youth In September 2010, IRI supported Youngstars’ four-day youth democracy academy for young Nigerian political aspirants in Abuja. At the conclusion of the conference, the participants agreed to mobilize their peers to participate in the voter registration process and vote. They also agreed to make efforts to reduce youth involvement in electoral violence. IRI also worked with Youngstars to provide social media training for local government elections. In February 2014, IRI held a training workshop with Youngstars and the Youth Initiative for Advocacy, Growth and Advancement (YIAGA), for 38 observers of local government elections in Plateau State. Participants were trained on how to use social media tools for monitoring and reporting on the elections. The workshop focused on information sharing, distribution of election materials, election monitoring procedures, and how to handle election-related violence. After the election, IRI found that 73 percent of youth trained at this workshop reported their Election-Day experiences through the use of phone calls, SMS, Twitter, and Facebook. In November 2013, IRI supported the Youngstars Foundation in hosting the second semester of the Lagos Young Aspirants Leadership Fellowship Program in Lagos. Forty one participants attended the first semester of the program (not supported by IRI), held earlier that year in Abuja. While the first semester focused on equipping participants with necessary knowledge about youth engagement in politics, the second semester explored all aspects of political leadership in Nigeria and the importance of political parties in Nigeria’s political environment. Videos NDI and IRI partnered with Youngstars to produce two separate voter and civic education films, “My Vote Can Change Nigeria” and “I Voted, Now What?” both of which were broadcast on national televisions, with the potential to reach tens of millions of people in Nigeria. Initially, NDI and IRI partnered with the Youngstars Foundation to produce the short film “My Vote Fit Change Naija” explaining to Nigerians the value of voting on election day. The story depicts young people who learn that to have effective government service delivery and to improve their general quality of life, they must get involved in government by voting for good leaders. Youngstars Foundation disseminated the clip via social media and mass-produced the film. An estimated 40 million people viewed the clip. 48 Building on that experience, in 2012 NDI and IRI supported Youngstars to create the civic education film “I Vote, Now Wetin?” (“I Voted, Now What?” in Nigerian pidgin). The film encourages Nigerian youth to stay involved in the political process post-elections through the story of a young man who is motivated to action when his sister is injured in an accident while carrying water home across a dangerous intersection. With the help of a local civic group, he lobbies his state representative to address the problem of poor access to water in his community. An accompanying theme song was released to radio stations across Nigeria to promote the film and its themes. Youngstars also released a handbook for the film to help youth activists facilitate discussion in viewing parties across the country (see Figure 3). NDI assisted Youngstars to disseminate copies of the film and handbook to youth across the country, with an estimated 11 million people reached. The group also broadcast the film twice in December 2012, with an estimated reach of nearly eight million viewers Youth Initiative for Advocacy, Growth and Advancement (YIAGA) (formerly known as Youth Action Initiative Africa (YAIA)) Both IRI and NDI have worked with YIAGA – formerly known as YAIA. In September 2010, NDI began work with eight local CSOs to reach youth, women, rural voters and other traditionally marginalized segments of the electorate with voter education and get-out-the-vote messages. The organizations formed a network to coordinate their activities, which they called the Network of Civil Societies for Voter Education (NETCIV). One of those organizations was YAIA, which undertook the following activities: • Produced and aired a weekly radio program to motivate and educate youth about voting, featuring guest speakers and allowing listeners to call in, and with an estimated listenership of 3.9 million; • Produced and aired television shows discussing electoral and political process themes, reaching over 2.1 million viewers; Figure 3: Cover of CEPPS-funded handbook, produced by Youngstars Foundation 49 • Produced a song called “My Vote My Future” with two popular artists of Plateau state. The song was shared with 100,000 people as a ringtone and played on two popular radio stations at least twice daily in April; • Produced and aired GOTV radio jingles directed toward youth on two popular radio stations, reaching over two million listeners; and • Distributed voter education and promotional materials to over 12,000 youth. In a different activity, YAIA solicited NDI’s technical assistance to form a youth coalition and to develop an advocacy strategy to give priority to youth issues during the National Assembly’s constitutional reform period. With NDI’s help, YAIA founded the Youth Alliance on Constitution and Electoral Reform (YACOR). In October 2012, NDI continued its support to YACOR as it organized a “Mega Youth Conference on Constitutional Review and Electoral Reform” which hosted over 400 participants including youth, representatives of the National Assembly, INEC, media, political parties and other civil society groups. At the conclusion of the conference, the youth agreed to support YACOR’s memorandum of recommendations -- which NDI assisted in producing -- and committed to using it as an advocacy tool with their legislators. IFES IFES’s program under CEPPS III was the four-year “Enhancing the Credibility of Nigeria’s 2011 Elections,” which concluded in 2014. As of 2015, IFES does not receive CEPPS funding Working with civil society was a secondary focus for IFES Nigeria under CEPPS. IFES saw its primary task as to provide technical support in electoral administration to Nigeria’s electoral management bodies. Civil society activities for IFES were used to augment and complement this process. The three areas in which IFES works with CSOs are: electoral violence monitoring, where the implementer works with the National Association for Peaceful Elections in Nigeria (NAPEN); and tribunal monitoring and campaign finance monitoring, where it works the Centre for Social Justice (CSJ). NAPEN Although NAPEN has existed since 1998, their recent work has been as an IFES mentee organization. According to IFES, they have now received grants from INEC, DFID and NDI. NAPAN moved into its own office in 2014, leaving its space within the IFES office. IFES and NAPEN worked closely to monitor and report on electoral violence in the lead up to, during and after elections. During the CEPPS III period, IFES trained NAPEN monitors in the IFES-developed Electoral Violence Education and Resolution (EVER) methodology and supported their efforts to monitor violence in the 2011 general elections, and in eight subsequent gubernatorial elections.56 56 EVER helps identify, track, and report incidents of electoral violence such as inter and intra party clashes, deaths resulting from political confrontations and destruction of campaign materials. 50 IFES and NAPEN undertook an assessment in 2010 of electoral violence hotspots in all six of Nigeria’s geopolitical zones. Subsequently, NAPEN monitors tracked incidents of electoral during and after the 2011 national elections. The monitoring exercise was also extended to track incidents for eight gubernatorial elections. The information gathered by the NAPEN monitors was verified and uploaded onto the website, www.NEVR.org. As of May 2014, 551 incidents had been verified and reported. After the 2011 national elections, IFES arranged a meeting between NAPEN and Chairman Jega and other members of the INEC leadership, to present the Nigerian Election Violence Report. Subsequently, INEC began a partnership in which INEC agreed to add the NEVR website to its resources at the commission’s Media Center. Center for Social Justice (CSJ) IFES’s work with Center for Social Justice (CSJ) is in two areas – campaign finance and election judge capacity building. Campaign Finance Beginning in 2010, IFES worked with civil society to highlight the issue of money in politics and build the capacity of CSOs to monitor and report on party financing. IFES partnered with CSJ to conduct campaign finance monitoring for gubernatorial elections in Edo and Ondo in 2012 and Anambra in 2013. CSJ monitors were deployed to observe political party spending on campaign rallies, media expenses and the use of state and administrative resources by incumbent parties. CSJ was able to document that the major parties were spending in excess of the 200 million naira ceiling established by law. CSJ’s report noted that there is no mechanism to enforce the ban on the use of state administrative resources, which were openly used by the incumbent party to the advantage of its gubernatorial candidate. Capacity-building of Election Petition Tribunal Judges CSJ also worked with IFES to monitor Election Petition Tribunals. In 2011, CSJ dispatched 37 monitors across the country to observe and report on the proceedings at the tribunals over a period of 15 months. IFES believes the presence of CSJ monitors at the tribunals improved the transparency of the proceedings and enhanced the credibility of the petition process as a whole. The increased visibility and popularity of the election tribunals made them a more attractive option for resolving political disputes, bolstering public confidence in election dispute resolution system. Figure 4 – IFES with CSO partner – Center for Social Justice 51 ANSWERS TO EVALUATION QUESTIONS QUESTION 1: ARE THERE NOW MORE LOCAL CIVIL-SOCIETY ORGANIZATIONS ACTIVE IN: A) ELECTIONS ASSISTANCE; B) POLITICAL PROCESSES; AND C) PROVIDING TECHNICAL ASSISTANCE TO POLITICAL PARTIES THAN THERE WERE SIX YEARS AGO? According to Audra Lycos of USAID, there are over 65,000 registered NGOs in Nigeria. This is a nation-wide number that includes democracy and governance (DG) CSOs and those in other sectors. While the number most likely includes many CSOs with very limited levels of activity, this large tally indicates both the immense scale of Nigeria, and the prominent place CSOs play in the culture of the country. On the question of whether there are more DG CSOs in the country now than in the pre-CEPPS III period, interview responses provided little precision. Nigerian nationals, working for implementers or CSOs, usually responded that there were more. Expats working for implementers were less sure. Anecdotally, however, there was consensus that the long term nature of elections work in Nigeria, coupled with the successes since 2011, has been instrumental in encouraging more CSOs to become involved. According to one Nigerian working for a IRI, CEPPS activities from prior elections “opened up boxes of other CSOs” and coalitions. The perception of democratic success seems to be bringing in an expanding number of civil society actors eager to participate in democracy-building activities. According to Jasper Veen of NDI, there is an increasing focus on “quality over quantity” as CSO coalitions are formed and the number of groups have collapsed into larger alliances for combined action. For NDI, this is especially true for election observation. In Veen’s view, the parallel vote tabulation is the primary driver of more CSO involvement. Publicity around the PVT is creating media coverage, and providing greater opportunities for CSOs to be heard. If there are more CSOs in the three above evaluation-question areas today than six years ago, how many of the newer organizations worked with CEPPS institutions or their local partners? The immense scale of Nigeria, means that national level CSOs are inevitably coalitions of state and local level organizations. NDI’s partner TMG, for instance, now has over 400 members. However, Veen said, “CSOs follow the money.” Most CSOs involved in election observation are usually not DG focused and come together “during elections only.” QUESTION 2: HOW WERE PROGRAMMATIC STRATEGIES DEVELOPED UNDER CEPPS II ALTERED OR EXPANDED (E.G., DIFFERENT APPROACHES TO ACTIVITIES, THE USE OF SUB-GRANTS) UNDER CEPPS III TO ADDRESS LOCAL CAPACITY BUILDING GOALS UNDER OBJECTIVE 10? CEPPS III in Nigeria was designed in 2010 after the death of President Yar’Adua. Although Yar’Adua had been elected during the flawed 2007 elections, he had agreed to a number of reforms that led to a more independent INEC.57 USAID decided “to pull CSOs into the process” in a way that would change the democratic trajectory in Nigeria. Consequently, CEPPS III was more CSO focused than its predecessor 57 INEC was made independent. It was eventually ruled that INEC members could not be political party members. 52 mechanism. The CSO sector was considered too weak and disbursed to be much of a factor during CEPPS II. With the change in leadership following the death of Yar’Adua, USAID felt a window of opportunity for CSOs might open up, and CEPPS III was designed accordingly. Was the CEPPS process for local partner selection optimal in the sense that competitive selection processes were balanced against the CEPPS need for establishing ongoing relationships? There are differences in how implementers select CSOs with which to work. IRI works primarily with organizations that have proved their “credibility” such as Youngstars and YIAGA. Its approach for CSO selection is based on recommendations and word-of-mouth. Generally, IRI will not release invitations to submit an Expression of Interest in the way NDI, for example, usually does. According to IRI’s Sentell Barnes, “our working relationships are with established organizations.” These are the organizations that “have shown they have the ability to work on their own. “ For IRI, working with less capable CSOs can be frustrating. “There are groups of less quality. We are trying to work with them too, but they are very reliant on implementing partners and often don’t have the capacity to implement. It is like you are doing the work for them.” There is evidence that some CSOs are contracting out their application processes to consultants – who may be working for competing clients. Shalva Kipidze of IFES noted that “we have seen competition among CSOs. Sometimes we see the same narrative from different organizations, or budgets done the same way. This indicates that CSOs are hiring the same person to write their applications.” Importance of Coalitions CEPPS III implementers primarily worked with established and/or technically specialized CSOs. However, the scale of Nigeria requires work with national level coalitions that contain many small and less established organizations. It is difficult in a large country like Nigeria for a single CSO to cover the entire country - limiting the number of CSOs implementers can work with at the national level. A way around this problem is the establishment of coalitions. To what extent were the approaches that USAID requested/recommended under Objective 10 appropriate for building local capacity? While there were no complaints regarding USAID’s recommended approaches for capacity building, some of the implementers found USAID’s guidance lacking in desired specifics. The CEPPS process has three primary stages at which USAID can suggest approaches for capacity building and the implementer can respond: the solicitation, the work plan, and the performance management plan (PMP). As described by one implementer representative, “the process begins when we respond to an RFP or RFQ. Based on USAID requirements, strategies for CSO capacity building are laid out there. Once an award has been made, the next step is to design a work plan based on the cooperative agreement. After that, we design a PMP and work with MEMS (a monitoring and evaluation contractor) to finalize it.” Specifics for how to work with CSOs usually come at the work plan and PMP stages. IRI, in particular, noted that that getting the necessary inputs from the CSOs for the work plan and PMP “requires considerable effort.” Although some CSOs are able to identify indicators, the consensus among CEPPS implementers is that Nigerian CSOs are not ready to design PMPs on their own. As NDI’s Veen put it, the trend by USAID is to give more input to CSOs, while at the same time increasing requirements for 53 more detailed work plans, PMPs and other reporting. This increased administrative and reporting burden, Veen said, will never by addressed adequately by CSOs in Nigeria. Appropriate Level of USAID Guidance Some implementers noted a lack of clarity from USAID in what constituted local capacity building. Two out of the three implementers indicated that, while USAID has defined what it means by capacity building, it is not clear the extent to which the agency is referring to the system or organization. One implementer asked rhetorically on the question of capacity building: “capacity to do what?” Another said, “I have not seen any guidance yet [on capacity building] from USAID.” Adamu Igoche of USAID indicated he would like USAID to have more input into the capacity building process. Implementers, he said, are “putting USAID in the passenger seat,” providing USAID with little input into the process. Igoche said he was “in favor of teasing out a definition of capacity building. Now, it can be anything and everything.” Audra Lycos of USAID confirmed that USAID is limited in the guidance it can provide. Because funding for CEPPS partners is in the form of cooperative agreements and not contracts, “we can only say what results we want and leave the details to the implementer. USAID tries not to be too prescriptive. “We don’t tell them how” to build capacity, Lycos said. Instead USAID is looking for suggestions from implementers to better define capacity building. According to Lycos, “that’s the nature of the cooperative agreement.” However, if more guidance from USAID were to be provided, funding would have to be structured instead as a contract, making CEPPS partners unlikely to bid. What were the reasons for better/worse follow through? Did CEPPS develop and implement any capacity-building approaches beyond those requested / recommended by USAID? If so, what were these approaches and what was the rationale for their development and adoption? Given that USAID was generally nonprescriptive with regard to capacity building, the agency had few complaints about follow-through. However, USAID Nigeria indicated that their “expectations should be higher” for creative ideas on capacity building from the implementers. Currently, they are “not getting much creativity” in their bids. Since the capacity building guidance under cooperative agreements is often limited, the expectation was that IPs were to develop innovative approaches that fit the circumstances of their programs. Some implementers provide versions of the same few proposals for multiple scenarios. “A PVT is not the solution to every problem,” USAID’s Lycos said. An example of capacity building beyond that suggested by USAID can be seen in the case of IFES, which created the NAPEN organization and nurtured it to a greater degree of than is seen in other CSO relationships. IFES believes that its capacity building of NAPEN took place on a broader scale than what USAID required. According to IFES, violence prevention requires more capacity building than other sectors. Recommendations from USAID USAID’s Lycos noted one area where USAID is providing more oversight over implementers and CSOs. USAID is putting requirements in some of their solicitations for bench marks for the gradual financial transition of projects to local partners. “We ask IPs to specify what shifts in responsibility will be made on an annual basis.” Such a system of financial system bench marks would be a very helpful best practice that can help CSOs gain capacity to receive direct funds. 54 Financial accountability is “what is really needed,” Lycos said. “Technical capacity of Nigerian CSOs is not the problem.”58 Lycos did acknowledge that some recent advances in technology, such as using debit cards and bank transfers instead of cash for payments to vendors and election monitors can improve financial accountability. “It can help build some confidence,” she said, “but technological innovation is not a substitute for a cultural change away from corruption. Technology has to be matched with political will.” QUESTION 3: TO WHAT DEGREE DID CEPPS INVOLVE LOCAL PARTNERS IN THE DESIGN, AWARDS MANAGEMENT, M&E PROCESSES, AND REPORTING OF PROGRAMS WITH WHICH THEY WERE AFFILIATED? The involvement of local partners in programs depended on the size, history and experience of the CSO. In cases of partnerships with more established organizations, involvement usually took the form of workshops and trainings scheduled around electoral events, along with an iterative process of mentoring CSO personnel. Sub-granting process The primary vehicle for involving CSO partners is the sub-grant proposal, in which the local partner has the opportunity to lay out their approaches for program design, management, M&E and reporting. The format for these proposals generally is designed by the implementer. For instance, the standard template for program proposals for IRI-Nigeria comes from IRI Washington. However, CSO input also comes from regular consultations between the implementer and CSOs. M&E is one area in which CSO input is most limited. For example, while IRI-sponsored CSOs come up with their own indicators, IRI will input CSO results into the PMP. Financial Management For some organizations, such as TMG, implementers have worked to put in place financial management reforms that rely on technology to reduce the potential for fraud. TMG has moved from manual accounting now to accounting software packages. NDI holds regular capacity building workshops in which TMG staff is introduced to management best practices. This has resulted in new policies and a financial management handbook. TMG now operates through electronic transfer of funds to vender and volunteers. Withdrawal limits have been imposed on the organizational bank account. According to TMG staff, “we now only use hotels that have functional bank account to which we can wire funds.” All observers must have functional bank accounts. TMG now has an online payment platform and a finance manager approves all transactions. The finance manager confirms that all payments are consistent with what was approved. Payments are confirmed with text messages. TMG staff interviewed claim that the organization has “not yet seen any leaks to the system. It is now very difficult to siphon accounts.” Monitoring and Evaluation M&E is considered by most implementers to be an issue where CSOs are not strong. Kingsley Bangwell of Youngstars cited M&E as an “area where we could use more training. If you miss M&E”, he said, “the whole proposal process will be wrong. We need more focus on what will be measured and why.” Bangwell thinks M&E training should be augmented by implementer staff from Washington. When talking about M&E in workshops, he sometimes hears field￾58 TMG would agree that their technical capacity is much improved. As TMG staff said, “in 2011 – everything was handled by NDI. Now we can handle our own statistical sampling.” 55 based implementer staff say “DC will not accept this.” Youngstars would like to hear more about the home office view of M&E. Did CEPPS integrate organizational capacity-building activities into their collaborations on elections and political processes, or did the activities occur on more of a parallel track Under CEPPS, capacity building generally was integrated into programs. For instance, the Youngstars Foundation reported that their accounting staff would attend capacity workshops prior to and during the grant period. During this one or two day financial training, NDI, sometimes augmented with Washington-based staff, would provide guidance on how to report on budget lines or how to manage fluctuations in foreign exchange. Sometimes an NDI accounting staffer would come to the Youngstars office, which is located outside of Abuja, but the visits generally related to individual project deliverables. This was a pattern seen with other CSOs as well. NDI, however, has a new partnership with World Learning, in which capacity building could be seen as occurring on a parallel track. During the case study field visit, World Learning was working with TMG on a capacity assessment of its Board of Directors. 59 QUESTION 4: WHICH FACTORS CONTRIBUTE TO THE SUCCESSFUL ELEVATION OF LOCAL ORGANIZATIONS INTO SUSTAINABLE PROVIDERS OF TECHNICAL ASSISTANCE RELATED TO ELECTIONS AND POLITICAL PROCESSES? Several key informants cited Nigeria’s relatively open political environment as a key factor for CSOs. The fact that CSOs have been around for several years and election cycles, and have been getting funding from other organizations, was mentioned frequently. As Kikelomo Salawu of NAPEN stated, “our government is supportive of CSOs, and their issues are addressed most of the time.” Specialization appears to be another important factor. For instance, TMG’s experience with PVTs, or NAPEN’s understanding to the patterns of electoral violence, have made them integral partners with the INEC. 59 A sub-question from the NORC Concept note that could not be answered here was: To what extent have CEPPS local and regional partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? Figure 5: World Learning (capacity building sub-contractor to NDI in post CEPPS project) conducts organizational assessment of Transition Monitoring Group, Abuja, August 17, 2015 56 Considering what is required for potential success, what are the characteristics of the local and regional organizations receiving CEPPS III support? Some groups such as TMG will probably continue receiving support of some kind as long as there is interest in the international community in election monitoring. Other, such as NAPEN, despite having developed a close relationship with the INEC, remain very reliant on their parent organization (IFES). Credibility and Specialization Ukeh of IFES spoke about the intangible aspects that contribute to the perception of credibility of an organization. “You look at the number of people who have worked with [a particular CSO] and the work they have done in the past.” Most important, he said, is experience in a particular area of expertise. “Too many CSOs are jacks of all trades and masters of none.” NAPEN, for instance, with their narrow focus on election violence, is now implementing projects directly for INEC. Since there is no research component in IFES’s scope of work, the only way INEC could receive this assistance was directly through NAPEN. This IFES mentee organization has helped INEC track indicators on reduction of violence and has shown how to identify triggers. DFID has also received NAPEN-provided institutional analysis of violence hotspots. CSJ has also worked directly for INEC, reviewing INEC manuals on campaign finance. INEC admits a lack of ability to enforce campaign finance enforcement and CSJ has been able to help them monitor campaign finance spending. Sub-grant profile IRI uses a “sub-grant profile” to assess the potential viability of a sub-grantee. The profile is a statement made by the CSO on their funding proposal regarding the internal management of their organization. It asks about the structure, number of staff, hierarchy, ongoing projects, staff salary and other factors that can help the implementer assess viability. For IRI, a successful outcome for a CSO project is indicated as they evaluate the work of their partner. As one IRI staffer put it, “if the CSO puts out a good product we were happy.” Ultimately, success breeds success. As a CSO gains the visibility that comes through a receiving international donor support, the organization gains confidence needed to gain more awards. Weak coalitions are problematic CSO coalitions are generally stronger than individual organizations, and serve to create more capable, potentially sustainable partners for USAID and its implementers. However, there are exceptions. IRIs experience with the Joint National Association of Persons with Disabilities (JONAPWD) coalition was plagued by leadership problems. The JONAPWD coalition was made up of six clusters, or categories, of disability. Instead of feeling united, some disability clusters felt they were being neglected and sought to establish independent relationships with donors, sometimes playing donors off against each other. The larger organization felt its authority was being compromised. It became clear to IRI that JONAPWD suffered from a leadership capacity deficit and did not have the capacity to deal with big grants. CSO Management As is the case throughout much of the developing work, most CSOs in Nigeria are driven by one or two key figures. These individuals necessarily become the driving factor as to whether the CSO has the leadership, innovation, and integrity necessary to for the CSO to be sustainable. As IRI points out, “sometimes, an organization can’t function if a key person leaves.” 57 QUESTION 5: WHAT EFFORTS ON BEHALF OF CEPPS HAVE BEEN MADE TO SUPPORT LOCAL ORGANIZATION OPPORTUNITIES FOR BECOMING DIRECT RECIPIENTS OF DONOR FUNDING? This effort is only in the very early stages. Helping CSOs become direct funding recipients was generally not stated as a primary goal of work with CSOs. Moreover, most implementers felt CSOs were not ready to receive direct funding due to factors such as poor capacity for compliance with USAID reporting requirements and risks of financial mismanagement Unsurprisingly, CSOs would like to receive direct grants. Youngstars, in particular, made the point that their group is “ready to handle bigger grants.” To what extent has CEPPS made efforts to assist local organizations become prepared to apply and receive larger and more complicated awards? Under the sub-grant mechanism used to fund most CSOs under CEPPS, implementers review all CSO reports before they are submitted to USAID, and will often do much of the reporting themselves. Implementers need to spend time with CSOs setting up the necessary meetings for implementing the sub grant and reporting on the results. Depending on the project, this can be a time consuming process. As IRI’s Barnes points out, “grantees may not have the same objectives as the funder,” and it is necessary to ensure that the CSOs “hit the right indicators.” According to Barnes, it can also be a challenge to work with CSOs with multiple funders to ensure that sub-grantees stick to the objectives contained in their USAID sub-grant. “Their focus can wander” depending on what other funding it is getting. “We have to make sure they stay in their lane.” For example, one prominent CSO, which also receives funding from Open Society and DFID, has a “temptation to merge programs together. Working with them takes time and energy,” Barnes said. For some implementers, such as Obaje Ukeh of IFES, direct funding of CSOs would be “a good thing.” It would be in the spirit of USAID Forward, he said. Although he conceded that some organizations do not have the necessary infrastructure in place to accept direct funding. IDEF’s partner CSJ wants more capacity building on financial monitoring and management of donor funds. They would like to learn about acceptable standards for donor reports. CSJ is also interested in more training on technology and social media, as well as on basic office skills such as typing. From an M&E perspective, they are also interested in “how to track achievements.” What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative and project management practices)? Both implementers and CSOs reported that work on sustainability relies heavily on finance and accounting. “Financial processes are complicated” says Barnes with IRI. Accounting skills were the most frequently cited aspect of sustainability. Accounting and Financial Management Finance has been critical to all successful capacity building efforts in Nigeria. According to IFES, there has been consistent management training of NAPEN, much of it on finance and accounting procedures. “We hold new workshops every time there are new financial policies, or new elections for which to mobilize.” According to CSJ, the IFES partner involved in campaign finance and judicial training, accountancy was the subject of most training, with topics including 58 how to spot false documents. CSJ has also received M&E training. IFES workshops have taught them how to track and score indicators, as well as how to prepare reports. The financial misdeeds of the PSC coalition described earlier have been a cautionary example for the CEPPS implementers, especially NDI. IRI reports no problems with the sub-grant process. “We make payments to the venders themselves,” IRI states. The Direct Funding Approach As IRI’s Barnes puts it, “USAID assistance comes with a binder of restrictions.” This is not necessarily popular among the recipient CSOs, but no USAID implementer stated that Nigerian CSOs are ready yet for such an approach. CSOs prefer the frequently-cited approach used by the British Department for International Development (DFID), in which funds are given directly by the donor to the CSO.60 According to Samson Itodo of YIAGA, the DFID model does work well, if an organization “has capacity” as his does. The frequent reporting requirements of sub￾grants, he said, can have a negative effective on the relationship with the implementer. Another approach that came up frequently was that used by the National Endowment for Democracy (NED), a non-profit organization that receives regular US government funding. NED gives small direct grants that require less reporting than a CEPPS sub-grant. For groups that have received NED grants, such as Youngstars, this approach is preferred. Kingsley Bangwell of Youngstars said: “on any day, NED would be my choice” for a funder. The Youngstars leader spoke glowingly of how NED had a predictable time frame for submitting proposals and announcing the results. “This approach,” Bangwell said, “helps you narrow and channel your energy. It allows use of money for overhead and helps us maintain some staff” between projects. The Youngstars, which has been receiving an annual grant from NED for 11 years, say that NED field officers visit several times a year and offer regional capacity building workshops. The lack of funding for overhead was a theme also mentioned by YIAGA. “Sub￾grants are very tedious,” said YIAGA’s Samson Itodo. The lack of overhead funding means that “staff are being short changed.” YIAGA’s Itodo echoed these sentiments. While stopping short of criticizing NDI or IFES, Itodo said that having to send weekly reports can every week effects the working relationship. The risk of CSO malfeasance, however, casts a constant shadow on efforts to provide direct funding to CSOs. For NDI, this experience was with Project Swift Count (PSC), the election observation coalition. In 2012, an NDI audit found PSC was using falsifying insurance policies, air tickets, vehicle rentals, and expenses for voter education efforts. While the fraud was uncovered, thanks to whistle blowers within PSC, the experience has made NDI cautious about the process. NDI’s Veen acknowledged that the debate of direct funding is related to the USAID Forward initiative. However, If USAID wants to go this way, Veen said, “they need to be prepared to deal with more operation swift counts.” USAID’s Igoche said there needs to be an assessment of financial capabilities of CSOs. Currently, the Interfaith Media Center (IMC), a church based group, is the only Nigerian CSOs receiving direct funding from USAID. He did point out that DFID gives direct assistance to CSOs, but did not think this was an approach USAID was ready to take. 60 NDI and IFES receive DFID funding. 59 NDI’s Veen said that more emphasis needs to be paid to changing CSO culture at micro, mezzo and macro level. Accordingly, NDI is currently using a subcontractor – World Learning – as part of an effort to reshape its CSO capacity-building efforts. At this stage, World Learning is administering a capacity assessment to TMG. USAID’s Igoche said that TMG “is a test case for helping CSOs become more empowered.” Veen made a recommendation that USAID should identify a threshold for sub-granting. The agency could allow direct funding at a threshold below around $100,000. For grants over this threshold, where checks and balances are required, USAID would continue to use a sub-grant mechanism. Below the threshold, however, direct funding could be considered. This would allow USAID micro manage less and reduce overhead costs. 60 APPENDIX I – PEOPLE INTERVIEWED Center for Social Justice (CSJ) Victor Emejuiwe, Program Officer Kingsley Naijialea, Program Officer International Foundation for Election Systems (IFES) Eniola Cole, Program Coordinator Seray Jah, Senior Capacity Building Manager Shalva Kipshidze, Country Director Obaje Ukeh, Voter Education Manager International Republican Institute (IRI) Sentell Barnes, Resident Country Director Emeka Diru, Assistant Program Officer Ijeoma Pamela Ugwueze, Program Officer National Association for Peaceful Elections in Nigeria (NAPEN) Kikelomo Salawu – Project Coordinator National Democratic Institute (NDI) Raymond Esebagbon, Governance Director Francis Madugu, Project Manager, Elections Machill Manidu Maxwell, Senior Program Manager, Citizens Participation Jasper Veen, Resident Country Director Transition Monitoring Groups (registered as Center for Civic Education) Lazarus Apir, Program Manager Paul Eneojo James, Training and Development Manager USAID Adamu Igoche, Deputy Team Leader, Office of Peace & Democratic Governance Audra Degesys Lykos, Democracy and Governance Officer, Office of Peace & Democratic Governance World Learning (consulted, not formally interviewed) Kevin Carew, Senior Director, Strategic Program Management, Senior Capacity Building Advisor Ana Lisa Silva Le Pera, Program Officer, Civil Society and Governance 61 Young Initiative for Advocacy Growth and Advancement Samson Itodo – Head of Research and Advocacy Youngstars Foundation Abim Bola, Director of Programs Kingsley N.T. Bangwell, Youth Specialist 62 DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY TUNISIA CASE STUDY TASKING N019 PERFORMANCE EVALUATION OF THE CONSORTIUM FOR ELECTIONS AND POLITICAL PROCESSES STRENGTHENING III LEADER AWARD Contract No. GS-10F-0033M/AID-OAA-M-13-00013 September 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or Government. __ DRG-LER 63 INTRODUCTION Contracted by the United States Agency for International Development (USAID), NORC at the University of Chicago is conducting an evaluation of the CEPPS III Leader with Associate (LWA) instrument pursuant to the DRG Learning, Evaluation, and Research activity. The purpose of the overall study is to gain a better understanding of how local CSOs are impacted organizationally by their association with CEPPS partners and the extent to which the inclusion of Objective 10, relating to building CSO capacity, impacted program design under CEPPS III. Spanning 92 awards in 52 countries, the evaluation is informed by a document review, electronic surveys of USAID representatives, CEPPS partner field officers, and local NGOs working in the electoral processes field, as well as key informant interviews with home office personnel of the three CEPPS partners. In addition, case studies in six countries with programming under CEPPS III were selected. The case studies were selected based upon the amount of programming under CEPPS III in a particular country and the degree to which programming involved civil society. Controls to ensure geographic diversity and programming from each of the CEPPS partners were also applied. Tunisia was selected from the Middle East North Africa (MENA) Region based on the fact that Tunisia is a high-profile Arab Spring country, domestic and international election monitoring were conducted under CEPPS III, and that building the capacity of Tunisian CSOs in the election process, and in their internal organizational development, was a stated objective of USAID (emphasis added). Three historic national elections took place during this time period (October – December 2014) enabling Tunisians to vote in what where determined to be transparent and credible parliamentary and presidential elections. The dates of performance reviewed included the periods from Mid-2011 through – April 30, 2015. The field research in Tunisia was conducted September 14 – 20, 2015. Prior to deployment, USAID/Washington’s former Agreement Officer’s Representative (AOR) for the CEPPS award for Tunisia was interviewed; and representatives of the CEPPS implementing partners were interviewed. While in￾country, Resident Country Directors and/or key staff leaders and programming staff from all three CEPPS implementing partners were interviewed, as well as representatives of seven (7) CSOs. TUNISIA COUNTRY CONTEXT Political conditions have remained relatively calm in the country since the three historic elections - parliamentary and presidential elections (and subsequent runoff election) were held in October and November 2014, respectively. The Nidaa Tounes party won the largest number of seats in parliament ­­ overtaking the ruling Islamist Ennahda party ­­ and its presidential candidate, Beji Caid Essebsi, won the run­off election in December. However, underlying social and political tensions, particularly between Islamists and secularists, have the potential to spark protest actions and unrest in the capital and elsewhere in the country, as seen throughout 2013.61 Prior to the elections a caretaker government took over the government at the beginning of 2014 following episodes of unrest in 2013. The unrest was sparked by a number of political developments, beginning with the assassination of opposition party members in February and July. Large­scale protests against the ruling­Islamist Ennahda party and months of political deadlock followed, prompting the resignation of the Cabinet in January 2014 in order to hand over the government to a transitional administration. The interim government, led by Prime Minister Mehdi Jomaa oversaw the adoption of a 61 World Watch Report on Tunisia, August 18, 2015, p. 1. 64 new constitution. Despite the recent political turmoil, in general, conditions have remained relatively calm in the country following the overthrow of longtime President Zine al­Abidine Ben Ali in 2011 and Tunisia is widely viewed as a model for other Arab Spring countries to follow in overcoming political adversity.62 The political situation referenced above is currently the most pressing security concern in the country. Petty crime, anti­ Western sentiment, terrorism and kidnapping are also security concerns in Tunisia. Over the last several months, there have been increased indications that both the short- and long-term terrorism threat in Tunisia has increased significantly. Short-term terrorism concerns have been raised following the two terrorist attacks targeting tourist locations, the Tunisian and foreign governments’ response to these attacks, and terrorist communications indicating serious intent to stage further attacks.63 Longer-term terrorism concerns have been exacerbated by Tunisia’s position between two high￾threat countries and the rise in number of Tunisians traveling to Syria to fight with Islamic State (IS).64 Economic concerns have increased due to a significant loss in tourism and trade revenues as well as a general decline in economic development.65 Since January 2011, when protestors poured onto the capital streets of Tunis, USAID has been at the forefront in supporting peaceful transitions in the Middle East and responding to the legitimate aspirations of citizens across the Arab world.66 The Agency’s work is part of a comprehensive U.S. Government approach. USAID was one of the first donors on the ground in Tunisia, providing over $19 million in support of Tunisia’s transition toward democracy, in addition to $3 million for humanitarian assistance, as part of a broader U.S. aid effort that has totaled over $32 million to date. The Agency’s activities have focused on helping political parties develop, and engaging citizens in the political reform process. Primarily working with new and existing civil society organizations, USAID is supporting Tunisian-led efforts to ensure a successful transition. Following the revolution, USAID conducted a series of on-the-ground assessments as well as broad outreach to civil society, youth, and women to evaluate the priorities and needs expressed by Tunisians. These consultations played a critical role in developing timely and targeted programming in Tunisia post￾revolution. 62 World Watch Report on Tunisia, August 18, 2015, p. 1. 63 On June 26, 2015, gunmen opened fire on a beach near two hotels in Sousse, approximately 90 mi/145 km from Tunis, killing at least 23 people including 20 foreign tourists. The attack follows a March 18 incident in which assailants attacked the Bardo National Museum in Tunis, also killing several foreigners. Although the overall threat of terrorism is lower in Tunisia than it is in most other North African and Middle Eastern countries, the March 18 and June 26 attacks highlights the ever-present concern of significant attacks. 64 World Watch Report on Tunisia, August 18, 2015, p. 1. 65 World Watch Report on Tunisia, August 18, 2015, p. 1. 66 Mohammed Bouazizi, a young Tunisian fruit seller, set himself on fire on December 17, 2010, to protest the difficult economic conditions he faced and the humiliation he experienced at the hands of local police. Bouazizi’s protest was personal, but it resonated with millions of people across Tunisia and the Middle East who identified with his suffering and his defiance. Across the region, people took to the streets, calling for political and economic reform. They expressed frustration with high unemployment, deteriorating living conditions, and a lack of economic opportunity. They called for transparency and accountability from their governments and a greater say in the decisions affecting their lives. They stood up and demanded basic rights in a region long dominated by authoritarian governments. In Tunisia, the public's anger and violence intensified following Bouazizi's death, leading then-President Zine El Abidine Ben Ali to step down on 14 January 2011, after 23 years in power. 65 Through this assessment process, USAID listened to the needs expressed by people both in the capital, Tunis, and in the country’s interior and border areas. Because Tunisia’s revolution began in these interior areas, USAID has recognized that a fully successful transition requires an inclusive process that involves all Tunisians. Therefore, while much work has focused on the capital, USAID is also working in historically marginalized regions. THE CEPPS III PROGRAM IN TUNISIA 2010 – 2014 USAID was one of the first donors on the ground following the Tunisian revolution.67 Since then, USAID’s assistance in the areas of democracy, human rights, and governance has focused largely on supporting the October 2011 elections and an ongoing Office of Transition Initiatives (OTI) program that supports constructive citizen engagement in the transition and encouraging positive interaction between the government and citizens. USAID’s Center of Excellence on Democracy, Human Rights and Governance (DCHA/DRG) provided elections support through the Consortium for Elections and Political Processes (CEPPS) mechanism that included support to the Independent Elections Commission (ISIE), civil society organizations, and political parties. Working primarily with Tunisian partners, OTI conducted activities to extend the reach of critical civic and voter education activities in the run up to the 2011 elections.68 In the context of Tunisia’s 2014 legislative and presidential elections, the National Democratic Institute (CEPPS/NDI) is conducting a program to build the capacity of Tunisian civil society organizations (CSOs) to observe the process leading up to and surrounding the vote, including through statistically-based observation, and to offer findings and recommendations to strengthen the electoral process going forward. The International Republican Institute (CEPPS/IRI) fielded long and short-term international observation missions to contribute to a transparent election process and strengthen future electoral efforts.69 USAID & CEPPS III Objectives With the goal of promoting the transparency and integrity of the Tunisian electoral process, activities under this Tunisia program are contributing to the following objectives: USAID Objective: Tunisia continues to develop transparent and accountable political processes and democratic institutions that respect human rights. CEPPS Objective 1: Foster improved coordination among civil society groups and build their technical and organizational capacity to conduct effective and credible electoral oversight (emphasis added); CEPPS Objective 2: Develop the capacity of a select civil society network to apply sample-based observation and other innovative techniques in election monitoring (emphasis added); CEPPS Objective 3: Promote the transparency and integrity of Tunisia’s electoral process through international assessment of the pre-election, election day and post-election periods.70 67 Although Tunisia is still a non-USAID country, this will change shortly as USAID establishes a full Mission. 68 USAID Program Description for Associate Award under Leader/Associate Cooperative Agreement with the Consortium for Elections and Political Process Strengthening (CEPPS III) DFD-A-00-08-00350-00 Tunisia Domestic Election Monitoring (T￾DEM) Activity, Spring 2013? 69 CEPPS Quarterly Report, January 1, 2015 – March 31, 2015, p.1. 70 CEPPS Quarterly Report, January 1, 2015 – March 31, 2015, p.1. 66 IRI’s objective under the CEPPS Program was to promote the transparency and integrity of Tunisia’s electoral process through international assessment of the pre-election, election day and post-election periods. NDI’s objectives under the CEPPS Program were to foster improved coordination among civil society groups and build their technical and organizational capacity to conduct effective and credible electoral oversight; develop the capacity of a select civil society network to apply sample-based observation and other innovative techniques in election monitoring/ and to promote the transparency and integrity of Tunisia’s electoral process through international assessment of the pre-election, election day and post￾election periods. CASE STUDY METHODOLOGY The case study of the CEPPS III mechanism support in Tunisia relies, in part, on qualitative date collection and analysis, which addresses aspects of research questions that are not well suited to quantitative analysis. This case study was informed by a review of CEPPS III programming documents, key informant interviews with USAID personnel as well as in-country staff of 3 CEPPS partners, and interviews with Washington, DC based staff of 1 CEPPS partner. Further, significant input from CSOs affiliated with different aspects of CEPPS III programming was solicited through individual interviews. Finally, some independent research on documents and events referenced in the course of the interviews was conducted. Document Review: Prior to commencing the field-based interviews, the following program documents were reviewed: USAID Program Description for Associate Award under Leader/Associate Cooperative Agreement with the Consortium for Elections and Political Process Strengthening (CEPPS III) DFD-A-00-08-00350-00 Tunisia Domestic Election Monitoring (T-DEM) Activity, Spring 2013?; CEPPS Quarterly Report: January 1, 2015 – March 31, 2015; CEPPS/IRI Final Report: July 12, 2013 – January 31, 2015; Tunisia Parliamentary Election 2014 Report, IRI, October 26, 2014; World Watch Report on Tunisia, August 18, 2015 Key Informant Interviews: Interviews were held with USAID/Washington’s former Agreement Officer’s Representative (AOR) for the CEPPS III Program in Tunisia, which was called the Tunisia: Domestic and International Election Monitoring Program (T-DEM); IFES, IRI, and NDI’s current Country Directors and key staff members in Tunis, as well as IRI’s Vice-President for Programs. CSO Partner Interviews: Interviews were conducted with twelve representatives from seven CSOs collaborating in different ways with the CEPPS partners. Some received some form of technical assistance to conduct election-related work before, during, and after the 2011 elections, although most did not receive a sub-grant award to do so. Some received technical assistance, including a sub-grant award, between early 2013 to date in preparations for the 2013 elections (which were conducted in 2014). 67 ANSWERS TO EVALUATION QUESTIONS QUESTION 1: ARE THERE NOW MORE LOCAL CIVIL-SOCIETY ORGANIZATIONS ACTIVE IN: A) ELECTIONS ASSISTANCE; B) POLITICAL PROCESSES; AND C) PROVIDING TECHNICAL ASSISTANCE TO POLITICAL PARTIES THAN THERE WERE SIX YEARS AGO? According to stakeholders, today, there are more Tunisian civil-society organizations (CSOs) active in: a) elections assistance; b) political processes; and c) providing technical assistance to political parties than there were six years ago. Prior to the 2011 elections and the January 2011 Arab Spring Revolution, there was little if any involvement or attempts at engagement by or with CSOs. In 2011 CEPPS III early on began working in-country with several more experienced CSOs, as well as select emerging groups and CSOs, in part, to determine which might be interested in working towards the (then anticipated) 2012 elections. During the period of the evaluation, 2010 – 2014, there has been an increase in youth-led organizations and an increase in youth participation in political processes overall. Some of the most active organizations include: Mourakiboun, Jeunesse Sans Frontieres (JSF), Eye on Elections/Ofiya, I Watch Organization, and Chahad Observatory. 71 If there are more CSOs in the three above evaluation-question areas today than six years ago how many of the newer organizations worked with CEPPS institutions or their local partners? Although there is greater involvement of civil society overall, most of this involvement was stimulated by the January 2011 Arab Spring Revolution and subsequent elections of 2011 and 2013/14, in which many international organizations and donors participated. As a result of the “rush” in participation, stakeholders say that there may be upwards to 10,000 CSOs now registered in Tunisia. However, once the elections and election-related activity ended, and USAID’s Office of Transitions Initiatives (OTI) Program ended, most of these active CSOs have disappeared.72 Today, according to those interviewed, there may only be ten CSOs that remain active. Most CSOs cited the lack of funding opportunities to support their election￾related activities, even though several of them staged large, credible election-monitoring efforts using their own financial resources, and with the technical assistance of NDI. QUESTION 2: HOW WERE PROGRAMMATIC STRATEGIES DEVELOPED UNDER CEPPS II ALTERED OR EXPANDED (E.G., DIFFERENT APPROACHES TO ACTIVITIES, THE USE OF SUB-GRANTS) UNDER CEPPS III TO ADDRESS LOCAL CAPACITY BUILDING GOALS UNDER OBJECTIVE 10? CEPPS II was not utilized in Tunisia, and CEPPS was only introduced in-country in its most recent iteration. Therefore, those interviewed could not draw comparisons between the two CEPPS instruments. However, long-time observers, managers, and implementers of CEPPS note that there was a deliberate strategy in CEPPS III, which was new and different from CEPPS II, which was to build the capacity of local partner organizations (emphasis added).73 Capacity-building efforts included, but were not limited 71Mourakiboun, JSF, Eye on Elections/Ofiya, I Watch Organization, and Chahad Observatory are all registered Tunisian CSOs, all of whom are less than five years old. Some managed multiple streams of donor funding. 72 Many, if not most of the registered CSOs in Tunisia, may be CSOs in name only. 73 The USAID Request for Applications (RFA) for CEPPS III included specific information about capacity-building. Capacity￾building included assisting local organizations to diversify their funding sources; manage multiple streams of funding; improve 68 to, efforts to assist organizations diversify their funding; improve their abilities to manage financial awards; expand their memberships; better articulate their missions, and conduct strategic planning. Was the CEPPS process for local partner selection optimal in the sense that competitive selection processes were balanced against the CEPPS need for establishing ongoing relationships? A competitive bidding process was not used by CEPPS III during the evaluation period under review in Tunisia. Instead, the focus was placed on the election-related activities of a handful of select CSOs. All of the sub-awards made under CEPPS III in Tunisia were sole-source awards, due to the fact that civil society was thought to be nascent and less experienced in managing large financial awards. NDI believed, based on its experience, that there was one organization that demonstrated capacity and motivation to both manage the finances, and conduct a parallel-vote tabulation (PVT) for the 2013 elections, and because of this, part of the sub-award provided for the costs of conducting the PVT, and part of the sub-award supported the costs for the coalition to conduct an election monitoring effort on election day. There was some history of the organization being engaged in political processes, and on the strength of this history, the sub-award was made to Mourakiboun. It was learned from the CSOs that several of them, who did not receive sub-grant awards, organized and staged credible (and in some cases large-scale) election observation missions on polling day with the technical assistance of NDI. These CSOs considered their efforts as their own, separate from others. For example, Chahed Observatory reported that they had increasing numbers of polling station monitors through the three election cycles that included: 2,443 observers for the legislative elections; 3,028 observers for the first-round presidential election, and 3,304 observers in the presidential run-off.74 QUESTION 3: TO WHAT DEGREE DID CEPPS INVOLVE LOCAL PARTNERS IN THE DESIGN, AWARDS MANAGEMENT, M&E PROCESSES, AND REPORTING OF PROGRAMS WITH WHICH THEY WERE AFFILIATED? According to the CEPPS implementers, local partner civil-society organizations were substantially involved in every step of the program. This was confirmed by key local partner organizations, who indicated that, at times, at their insistence, the CEPPS implementers engage with them at their level in every aspect of the program. Generally, the way the sub-award process was conducted in Tunisia went as follows: The CEPPS implementers approached USAID with the initial concept note. The CEPPS implementers had significant, and many, consultations with Mourakiboun in order to draft their concept note prior to approaching USAID; USAID responded by putting out their Program Description which in large part, was based on the CEPPS implementers’ concept paper; Following USAID’s issuance of the Program Description, NDI, for example, developed a program along with Mourakiboun that included Mourakiboun’s input; accounting and financial management practices; expand their memberships, and; better articulate their missions, and that the USAID Request for Applications (RFA) for CEPPS III. 74 CEPPS Quarterly Report, January 1, 2015 – March 31, 2015. 69 Once the agreement was reached between USAID and NDI, and the sub-award was made to Mourakiboun, NDI and Mourakiboun continued to work together closely, with NDI providing appropriate space and distance to Mourakiboun to allow them the greatest degree of independence and self-direction. The approach of NDI towards working with Mourakiboun was described as “continual coaching.” According to NDI, from 2011 up to date, they were able to coach Mourakiboun in every aspect of organizational development75, as well as program implementation and technical skills-building.76 Program activities from 2011 through 2014 included: voter education; training-of-trainers (to monitor polling stations); monitoring of polling stations; monitoring of campaign finance; and conducting a parallel vote tabulation (PVT). Program innovations included the use of SMS service and text messaging; Google Docs; and the PVT simulation exercise. Even though other partner CSOs, such as Chahad Observatory, JSF, and Ofiya/Eye on Elections, did not receive direct financial support, they all received NDI’s technical assistance. All of them indicated that NDI worked with them appropriately, effectively, and respectfully, providing each organization with what they needed at the time. Did CEPPS integrate organizational capacity-building activities into their collabora￾tions on elections and political processes, or did the activities occur on more of a parallel track? The CEPPS implementers both integrated organizational capacity-building activities into their work with partners as well as conducted specific capacity-building activities on a parallel track. Because USAID specifically built and encouraged capacity-building activities into the CEPPS program in Tunisia from the very beginning, the CEPPS implementers were deliberate and intentional about their working approach with the sub-grantees. At least in part because capacity-building was a USAID requirement in Tunisia, the CEPPS implementers had to give capacity-building thought and emphasis, and had to implement in a way that lent itself to building the capacity of the organizations. It had to be reported on, and measured against indicators. An example of integrated capacity-building activities: Parallel-vote tabulation (PVT) simulation training: A large part of the CEPPS III financial/technical support went to assisting Mourakiboun conduct a PVT. Since they had never conducted one before, they began a simulation exercise several months before the elections where scheduled, in order to practice the process and learn from their experience. Much of NDI’s technical assistance was in the form of coaching and mentoring Mourakiboun through the several-months long simulation exercise. An example of parallel track capacity-building activities: Tailored workshops were designed and provided on topics such as fundraising; proposal development and drafting; committee development and organizational structure; financial management; monitoring and evaluation; reporting; communications and public relations work; and lobbying and advocacy. 75 Organizational development would include financial management of the sub-award, strategic planning, fundraising, proposal development, monitoring and evaluation systems, reporting, communications and PR, lobbying and advocacy. 76 Technical skills-building included that of conducting the PVT simulation exercise, conducting the PVT, conducting polling day monitoring, developing positions/statements and working with the media. 70 To what extent have CEPPS local and regional partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? USAID pushed for two issues to be included in the CEPPS III work in Tunisia: i) Gender balance was not given attention, and potential barriers to women participating in political processes were not being addressed; ii) There was a lack of post-election consultations built into the program and there needed to be some level of after-action processing to develop best practices and lessons learned to apply in the future. USAID also knew that in the post-election period, the participating organizations needed to be able to consider next steps and to consider in which types of activities they would engage between election cycles. Mourakiboun’s director traveled to Washington, DC and through the encouragement of CEPPS, made contacts with experienced election-related organizations in Albania and Georgia to gain experience from them in which to apply in the Tunisian context. Recently the CEPPS partner organizations, including Chahed Observatory, Mourakiboun, and Ofiya, conducted after-action debriefings, and planned and began to implement comprehensive, inclusive evaluations of their observation efforts, engaging all levels of observers throughout the country to share their experiences and provide recommendations.77 During these evaluation workshops, the partners analyzed their strengths, weaknesses, opportunities, and threats, and identified challenges and potential responses to those challenges in order to make adjustments and improvements going forward. QUESTION 4: WHICH FACTORS CONTRIBUTE TO THE SUCCESSFUL ELEVATION OF LOCAL ORGANIZATIONS INTO SUSTAINABLE PROVIDERS OF TECHNICAL ASSISTANCE RELATED TO ELECTIONS AND POLITICAL PROCESSES? In discussing this question, those interviewed engaged in lively discussion and had much to say. Factors that contributed to the successful elevation of their own organizations, and of Tunisian organizations overall, include:  Receiving timely financial assistance; flexibility of funding;  Receiving timely, appropriate, and sustained technical assistance; coaching, mentoring;  That there were delays in the 2012 election played to our advantage and provided more time than expected to learn, absorb, and adapt as needed based on the simulation exercise;  A thorough knowledge and understanding of the PVT before actually implementing it;  Repeated replication over three electoral processes (Legislative and two rounds of Presidential) in which to practice, implement, learn, reflect and improve;  Taking time to think through an activity before just jumping in to try to implement it;  Plan to do something challenging, but be realistic; 77 CEPPS Quarterly Report, January 1, 2015 – March 31, 2015. 71  Ensure staff is committed to the mission, and that there is clarity of goals and objectives;  Conduct honest self-evaluations and provide yourself constructive feedback;  When political parties appreciate, respect, and draw from the value civil society adds78 Considering what is required for potential success, what are the characteristics of the local and regional organizations receiving CEPPS III support? Characteristics discussed included:  Being brave, bold, and revolutionary;  Talking yourself into saying “I can do this”;  Branding yourself and developing expertise in areas of interest;  Listen well, and be determined;  Develop good monitoring and evaluation of your work to check and gauge progress;  Professionalization; development of specific expertise;  On the other hand, the willingness to learn new skills that could lead to new expertise;  A board of directors that provides good oversight; appropriate governance;  Being motivated and organized;  Being convinced of values of work – conviction of vision, mission;  Being equipped with the right technical tools. QUESTION 5: WHAT EFFORTS ON BEHALF OF CEPPS HAVE BEEN MADE TO SUPPORT LOCAL ORGANIZATION OPPORTUNITIES FOR BECOMING DIRECT RECIPIENTS OF DONOR FUNDING? The pre-award audit was conducted and headquarters based financial managers of the CEPPS implementers traveled to Tunisia to teach, mentor and coach to the point where Mourakiboun’s financial controls were 100% consistent with USAID requirements for managing the award. The PVT conducted by Mourakiboun was not only the first of its kind conducted in Tunisia, but the first in the MENA region. Mourakiboun is aware of and is taking steps to share their skills and experiences outside of Tunisia – Cote d’Ivoire and Mauritania being named as two countries where Mourkiboun staff have traveled to recently in order to share their experiences with counterparts. Burkina Faso, Ghana, and Slovakia were countries also mentioned as ones where Mourakiboun staff traveled in order to share experiences or from which to learn. 78 Political parties in Tunisia became interested in the work of civil society, particularly related to election monitoring and the PVT. Parties requested and received support from civil society to learn how to conduct their own election monitoring, and to coordinate their efforts with those of the neutral, impartial, nonpartisan observers. Parties also worked closely with civil society in an effort to expand their own reach into the country. There was also an appreciation by the parties that the role of civil society is to watchdog the parties, provide input on party platforms, and then hold the parties accountable to performing on their platforms. 72 Current efforts are underway to assist Mourakiboun to diversify their experiences, and to use their technical skills and experiences in areas other than election-related activities. For example, the skills and techniques learned for monitoring elections are being tested and applied to monitor health care reform, and possibly other reforms. To what extent has CEPPS made efforts to assist local organizations become prepared to apply and receive larger and more complicated awards? What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative and project management practices)? Fundraising, proposal development, strategic planning and advocacy and lobbying workshops were conducted in an effort to help organizations improve their capacities. Monitoring and evaluation workshops were also conducted to ensure organizations were applying proper reporting requirements. NDI also provides ongoing consultations and advice regarding proposals, proposal development and approaching donors. Models and examples of efforts taking place in other countries, whether in the MENA region or beyond, have been shared and processed with Mourakiboun in order to build upon their knowledge. Current efforts are also underway to assist Mourkiboun with international donor relations, such as study tours to Canada and the United States. CONCLUSION/SYNTHESIS OF FINDINGS The period between 2011 and 2014, and the Arab Spring, ushered in many positive changes in Tunisia’s political environment, immediately before, during, and after the 2011/12 Legislative Elections and the 2013/14 Legislative and Presidential Elections (and subsequent runoff). For the CEPPS implementers, it was a period of experimentation and creativity – although elections and election-related activity have been conducted by Tunisian organizations for years, now with USAID support, under CEPPS III, the PVT was the first of its kind to be implemented not only in Tunisia, but in the MENA region. Campaign finance monitoring activities were conducted, and election-day polling station monitoring took place with several Tunisian organizations, the most prominent being the Mourakiboun coalition of three organizations. These activities served to excite and stimulate partner organizations and their constituencies and served as a catalyst for post-election activities and activities which were carried forward.79 CSO participation provided capacity-building opportunities by learning new program activities, and how activities are implemented and reported, promoting efforts publically through various organizational constituencies, and in building relationships with other organizations. Capacity-building activities, encouraged and required by USAID from the very beginning, were necessary to lead to increased levels of knowledge and skills, all leading to increased levels of independence, and should be included, and reported on, in future programs. 79 Despite the optimism and enthusiasm demonstrated by the CSOs during this period of time, it should be noted that there is now growing impatience for government to implement reforms, and growing dissatisfaction that democracy, and the democratic election of 2014, have not delivered on their promise. One CEPPS implementer noted that these periods of highs and lows are those that CEPPS has observed and weathered in other countries undergoing reform. 73 APPENDIX I – PEOPLE INTERVIEWED USAID Ms. Julie Denham, Elections and Political Processes Team Lead (Acting), USAID/DCHA (fmr CEPPS AOR Tunisia) IFES Mr. Nikolas Kaczorowski, Tunisia Resident Country Director, IFES IRI Mr. Thomas Garrett, Vice President for Programs, IRI Mr. Mohieddine Abdellaoui, Tunisia Acting-Resident Country Director, IRI NDI Mr. Jeffrey England, MENA Region Deputy Director, NDI (TBD) Ms. Angela Short, MENA Senior Program Manager, NDI (TBD) Ms. Marie-Eve Bilodeau, Tunisia Resident Country Director, NDI Ms. Rim Ben Hassen, Program Officer, NDI Ms. Nadia Ben Salah, Program Manager, NDI Ms. Assil Kedissi, Interpreter/Program Manager, NDI Local Partners Mr. Aschref Aouadi, President – Co-Founder, I Watch Organization Mr. Mouheb Garoui, Executive Director – Co-Founder, I Watch Organization Mr. Rafik Halouani, Mourakiboun and Association Tunisienne pour l’Eveil Democratique (ATED) Ms. Lahrabi Manel, Finance Manager, Mourakiboun Mr. Moez Bouraoui, Association Tunisienne pour l’Eveil Democratique (ATED) Mr. Mr. Ms. Mr. 3+ Lassad Moussa, Attorney, Founder, Chahed Observatory Mr. Aladdin Dridi, Founder, Democratic Youth Union Dr. Kamel Gharbi, President, Eye on Elections/Ofiya Mr. Ahmed Allouch, President, Jeunesse Sans Frountieres (JSF) 74 DRG LEARNING, EVALUATION, AND RESEARCH (DRG-LER) ACTIVITY ZAMBIA CASE STUDY TASKING N019 PERFORMANCE EVALUATION OF THE CONSORTIUM FOR ELECTIONS AND POLITICAL PROCESSES STRENGTHENING III LEADER AWARD Contract No. GS-10F-0033M/AID-OAA-M-13-00013 October 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or Government. __ DRG-LER 75 INTRODUCTION USAID’s CEPPS III Leader with Associates mechanism has contributed to the capacity of local and regional organizations in Zambia in their ability to undertake a Parallel Vote Tabulation (PVT), thereby improving the transparency and perceived legitimacy of elections in that country. PVTs are complex undertakings that require a high degree of capacity from any Civil Society Organization (CSO) involved. This usually involves a coalition of CSOs that can cover all parts of the country. However, an unexpectedly tight electoral timetable, resulted in a coalition in 2015 focused more on successful implementation of a PVT than on CSO capacity building, highlighting the limits of a capacity building approach during complex undertakings. Zambia was the only case study in which programming relied exclusively on one CEPPS implementer (NDI). In turn, NDI relied primarily on one well-established CSO, Caritas￾Zambia. Given the almost single focus in Zambia during the CEPPS III period, this case study offered limited insights into some of the evaluation questions, but there were a few notable findings generated by key informant interviews. These focus primarily Question 4, on the characteristics of sustainable CSOs, and Question 5 on the factors contributing to the successful elevation of local organizations into sustainable providers of TA. Specifically, sub￾questions in the full report which Zambia case study informed included the following four. EQ4.1: Which factors contribute to the successful elevation of local organizations into sustainable providers of Technical Assistance? Specialization: Having a specific area of acknowledged expertise was cited by key informants as an important attribute of sustainable CSOs. For instance, specialization was cited by the Zambian National Women’s Lobby, which has existed since 1991 as an advocate for women in Zambia, as important to establishing a relationship with donors. Despite this, however, the Women’s Lobby played only a minimal role in the 2015 election PVT. Dedication to the Mission: Key informant interviews highlighted the importance of ensuring that CSO staff is committed to the mission. In Zambia, church members of the 2015 PVT coalition were found to have a greater sense of accountability and dedication than in a similar, but more diverse coalition than had operated in an election four years earlier. Strong Governance: In Zambia, NDI criticized what they referred to as “briefcase organizations,” groups with minimal administrative capability such as financial and management abilities. Organizations that have strong governance tended to be more useful members of representative coalitions. Avoidance of “Founder’s Syndrome:” Several CSOs in Zambia referred to management problems caused by over-reliance on one person – a phenomena known as “founder’s syndrome.” If anything happened to that person, or if a donor felt funds were being mismanaged, donors would withdraw funding. Positive Enabling Environment: The relatively free and open civil society in Zambia was cited in interviews as important to the sustainability of CSOs. 76 EQ4.2: What are the characteristics or prerequisites of the local and regional organizations receiving CEPPS III support which most align with potential for success? Coalitions and Networks: Establishment of CSO coalitions or networks were a prominent feature in Zambia, where it was noted that coalitions can allow CSOs to reach more areas of the country. The experience in Zambia points out that PVT coalitions can vary widely in the level of cohesion and effectiveness, depending on the structure of the alliance. EQ5.1: What CEPPS efforts have supported the opportunities of local organizations for becoming direct recipients of local funding? In Zambia, trainings and workshops have been the primary means of building CSO capacity, mostly for PVTs. Performing parallel vote tabulations is a skill that can be transferred from country to country. In Zambia, NDI brought in representatives from other African countries to help train Caritas, the CSO conducting a PVT there. Sub-granting Provides Valuable Exposure to CSOs in Reporting: The sub-grant process has enabled CSOs to learn first-hand the kind of budgeting and reporting that would be required for an organization receiving direct funding from USAID. In Zambia, NDI pointed out that the financial reporting requirements through sub-grants are as strict as what USAID requires. EQ5.3: What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative, and project management practices)? Established CSOs are Often Favored: In Zambia, CEPPS has favored an experienced CSO, Caritas, to implement its most recent PVT. However, developing only a few strong organizations may be preventing the development of others. Balancing the desire to develop the skills and organizational capacity of CSO partners against the benefits of a diverse civil society was a challenge. Missions are Skeptical of Direct Funding: When asked directly about direct funding for local CSOs, USAID Zambia sees the potential for direct funding as a bigger management burden than would be ideal for a small Mission. Overhead Issue: CSOs, and some implementers, are in favor of supporting more CSO overhead costs through sub-grants. Sub-grants usually cover program costs, but very little in the way of overhead or administrative costs. CSOs in Zambia advocated for allowing sub￾grants to pay more of these costs, given the importance of administrative staff and rent to long￾term sustainability. NDI Zambia was sympathetic to this view as well. CASE STUDY METHODOLOGY NORC conducted a careful process to select the six country programs to be the object of the case studies, as described in the main evaluation report. The objective was to select a varied set of programs that cover different regions, approaches, and subject matter in a variety of contexts. Zambia was an unusual case, with programming relying exclusively on one CSO and 77 prioritizing its development. It also offers an example of what happens when CEPPS must respond with accelerated election programming due to unanticipated circumstances. Finally, it offers an example of a country with a single CEPPS implementer, offering a degree of comparison with the other case studies of countries with multiple implementers. Qualitative Evaluation Approach NORC’s case study of the CEPPS mechanism support in Zambia relies on qualitative data collection and analysis. A subject matter expert on the Evaluation Team conducted key informant interviews (KIIs) in Zambia with USAID, CEPPS implementing staff in Zambia who work directly with local partner organizations, and representatives of local CSOs cooperating with the CEPPS program, using semi-structured, open-ended interview guides for the KIIs. KII guides for various classes of stakeholders were prepared in advance of fieldwork, outlining the key questions and probes to address during the interview. The interviewer took written notes during the interview, and transcribed them within a day of the interview. Each interview started with structured questions and pre-determined topics and ended with more open-ended questions that sought to reveal unanticipated topics and themes. Analysis of the KIIs includes a summary of common themes that emerge in response to each topic – highlighting the range of responses and experiences. This was supported by quotes from the participants. As part of this case study, the evaluator reviewed program documents, including CEPPS quarterly reports and other documents provided by USAID, the implementers and partners. BACKGROUND ON ZAMBIA Zambia has had more than two decades of successful multi-party democracy and is considered a bastion of peace and security in Africa. It has experienced 12 straight years of economic growth and its average per capita income of $1,760 makes it a lower-middle-income country. Despite this relative peace and prosperity, however, three out of four Zambians still live in extreme poverty and the country faces economic and health challenges. Since Zambia’s independence in 1964, the United States has provided close to $4 billion to support the country’s economic, political and social development and to respond to major crises, such as HIV/AIDS, malaria and periodic droughts. USAID democracy and governance assistance is one of six development objectives and focuses on advancing civil society, governance, and public service delivery. Zambia has been an emerging democracy since 1991. In 2011, Michael Sata of the Popular Front (PF) was elected President, marking a peaceful transfer of power from the long-dominant Movement for Multi-Party Democracy (MMD). On October 28, 2014 President Sata died of an undisclosed illness. In accordance with the constitution, the Electoral Commission of Zambia (ECZ), set a date for the presidential by-elections on January 20, 2015, less than 90 days later. Zambians went to the polls without any significant political disruptions. The ECZ declared Edgar Lungu, the candidate of Satas‘s PF party, the winner. It was a close election, with Lungu securing 48.3 percent of the vote, and Hakainde Hichilema of the opposition United Party for National Development (UPND) garnering 46.7 percent. With only 32.4 percent of registered voters casting a ballot, the 2015 election suffered a low voter turnout, as compared to the 54 percent of citizens who voted in the 2011 general elections. The rainy season and voter apathy likely contributed to the low turnout. 78 After initially protesting the election’s results, Hichilema directed his party supporters to start preparing for the next election, scheduled for August 11, 2016. However, while unsuccessful in taking control of the presidency, the UPND received 46.7 percent of vote, as compared to the 2011 presidential race when the party only received 18.2 percent, suggesting the party will be best positioned to challenge PF in 2016 THE CEPPS III PROGRAM IN ZAMBIA The National Democratic Institute (NDI) began working on elections in Zambia in 1991 and has been the primary CEPPS implementer in that country ever since. President Sata’s death in 2014, and the constitutional requirement for an election 90 days later, meant that USAID and NDI, as well as Zambian civil society, political parties and voters, had less than three calendar months to prepare for this major event. USAID and NDI were additionally concerned that the political environment in Zambia had deteriorated since the previous presidential elections 2011, as recent parliamentary by-elections had been marred by inter￾party conflict and violence. Those implementing the CEPPS assistance were concerned about the possibility of further violence, particularly given the high stakes involved. NDI is the only CEPPS III implementer operating in Zambia in a significant way for the purposes of this case study.80 The primary focus of this study is the PVT carried out under a CEPPS Associate Award, which originally ran from November 26, 2014 – July 31, 2015 in the amount 80 IFES will be providing some analysis of the election management body in 2016. Otherwise, NDI is the sole￾CEPPS implementer in Zambia. PVT Results: From the CCMG Facebook page. 79 of $1,200,000.81 Although activities are ongoing in early preparation for the 2016 election, this case study looks primarily at the PVT for 2015 by-election and contrasts it with the previous effort in 2011. In 2015, NDI, through the CEPPS III mechanism, provided assistance through a coalition of faith-based organizations, named the Christian Churches Monitoring Group (CCMG). The PVT provided assistance to Caritas Zambia, part of a worldwide confederation of 165 Catholic relief, development and social service organizations. Caritas Zambia was the lead organization in a civil society network composed of four Christian church groups. Caritas had also been the lead CSO of a more diverse coalition for the PVT in 2011 that included several NDI partners that had been receiving capacity building assistance. Given the unexpected turn of events in 2015, however, NDI and USAID felt that Caritas and its new CCMG coalition had the best combination of resources and experienced personnel to put together a 90 day PVT in a manner the 2011 CSO coalition could not. In its late-2014 project proposal, NDI pledged to assist CCMG to: strengthen its capacities in statistical sampling and observer deployment; develop observation forms and training programs; establish rapid communication and data collection systems; conduct volunteer recruitment and management; perform data analysis; and provide strategic communication on PVT findings. While the 2015 PVT coalition was, by most accounts, more successful than the 2011 experience, the comparison highlights the relatively lower priority given CSO capacity building when the needs for experienced CSOs are at their highest. CEPPS Objectives As laid out in the CEPPS agreement, the three objectives of the program are to: 1. Support local civil society to conduct a credible parallel vote tabulation of the presidential by-election. 2. Enhance the capabilities of local actors to conduct a credible qualitative observation of the presidential by-election. 3. Apply lessons learned from the presidential by-elections to strengthen coordinated civil society action in advance of the 2016 elections.82 The focus of this case study is on the first of these objectives. As USAID has acknowledged, the urgency of the 2015 by-elections left little focus on traditional CSO capacity building. In fact, much of both the implementation and capacity building for the by-elections was carried out by Caritas rather than NDI. A Choice between Two Coalitions This case study offers a contrast between two different models for CSO coalitions, with different emphases on capacity building. In 2011, NDI supported Caritas to lead an eight organization coalition – known as the Civil Society Election Coalition (CSEC). In addition to Caritas Zambia, CSEC included the Anti Voter-Apathy Project 81 The award has since been extended for an additional 24 months to cover the 2016 elections. The CCMG Program in 2016 has three components: Voter registration observation; long term observation focused on violence, candidate nominations, gender bias, governmental interference, use of public resources; another PVT, to be funded by USAID and DFID. In addition, USAID is making three separate sub-grants to the three organizations conducting voter education – ZNWL, FODEP, and OYV. CCMG was not invited to bid on voter education component. Finally, USAID, along with DFID, will beginning work with political parties. 82 CEPPS Zambia Quarterly Report – October – December 2014. 80 (AVAP); Foundation for Democratic Process (FODEP); Operation Young Vote (OYV); Southern African Centre for Constructive Resolution of Disputes (SACCORD); Transparency International Zambia (TIZ); Young Women in Action (YWA); and Zambia National Women’s Lobby (ZNWL). With a few exceptions, such as Caritas and TIZ, most were small local CSOs with capacity building needs. CSEC succeeded in fielding around 9,000 trained and accredited election monitors during the 2011 tripartite elections. It also conducted a PVT in 710 polling stations in all 150 Zambian constituencies. Despite the outward success of CSEC, the coalition was torn by internal dissent and resentment among some of its members. The relationship between Caritas, an established international Catholic organization, and the junior members of the coalition was rocky and created resentments on both sides. Caritas, which had little need for CEPPS capacity-building itself, was frustrated by the lower capacity and perceived lack of accountability of some the CSEC members. By contrast, some of the non-church CSOs in CSEC felt dominated and micro-managed by Caritas. Before the unexpected death of President Sata, the expectation was that the next national elections would be in 2016 and that the next PVT might operate without Caritas. Samuel Mulafulafu, Executive of Caritas maintains that the non-church CSOs “bunched together and excluded Caritas,” since they were “not happy with our stringent oversight.” He appeared to be referring to the Zambia Electoral Alliance (ZEA), which was announced in mid-2014, and included AVAP, FODEP, OYV, SACCORD, ZNWL – all former members of the 2011 CSEC coalition, plus the Media Institute of Southern Africa (MISA). When the 2015 by-elections were announced, however, CSCC was not reconstituted. Instead, Caritas formed the Christian Churches Monitoring Group (CCMG) out of a coalition of the groups representing the majority of the Zambian Christian Church community. They included: the Council of Churches in Zambia (CCZ), an organization of mainline Protestant churches, including Anglicans and Methodists; the Evangelical Fellowship of Zambia (EFZ), the mainline evangelical churches; the Jesuit Centre for Theological Reflection (JCTR), an independent Catholic group; and the Zambia Episcopal Conference (ZEC), the Catholic Bishops group of which Caritas Zambia is a part. Caritas organized the group and took the role of Secretariat. In December, NDI hosted a meeting between CCMG and ZEA, the two rival coalitions, to explore coordination of election assistance efforts. As USAID stated in their Quarterly Report, the two groups “acknowledged the need for coordination and better communication among civil society and was a first step toward repairing relationships that had been damaged as a result of work during the 2011 election.” In the end, however, USAID and NDI agreed to use most of the CEPPS funding to support CCMG. USAID and NDI simply had more faith that CCMG could do the job on the timetable required. The CCMG MOU with NDI After conducting a sub-grants assessment and establishing a plan for disbursing funds to observers, NDI and CCMG signed a memorandum of understanding (MOU) describing the roles and responsibilities of each organization. As laid out in the MOU, Caritas would act as the secretariat of CCMG, assuming a coordinating role for technical, organizational and financial components of the monitoring effort. A steering committee of leadership from CCMG member organizations would make strategic and political decisions on behalf of the consortium. Administrative and financial decisions would reside with the Executive Director of the Secretariat - Caritas. 81 Under the MOU, CCMG “at its discretion” could request technical assistance from NDI to enhance its capacity to monitor elections, specifically related to parallel vote PVT methodology. This included developing an overall observation strategy; designing and drawing a random, representative sample of polling stations; recruiting observation team staff, trainers, supervisors and observers; developing an observation reporting system using SMS; developing and maintaining an observer database; drafting and reviewing observation manuals and checklists; conducting trainings of trainers; managing a data center; analyzing observer data; and developing and issuing public reports on observer findings. NDI would provide financial assistance to CCMG through Caritas Zambia through a “combination of direct funding and sub-grants.” Due to the abbreviated timeline of the by￾election, NDI directly funded activities and procurement on behalf of CCMG. NDI disbursed a sub-grant to Caritas Zambia beginning in January 2015. As per the MOU, CCMG and Caritas Zambia would “abide by NDI’s sub-grant administration procedures, ensuring that all funds are used for their programmatic intent, fully account for all funds received from NDI as mandated by the financial agreement between the two organizations and return to NDI any unused funds.” CCMG and Caritas Zambia agreed to submit financial and narrative reports. Capacity Building of CCMG In a formal sense, NDI did provide capacity building for CCMG, especially in the form of PVT training. Capacity building was embedded into the relationship with NDI as outlined in the MOU, primarily in the form of technical assistance. It included refresher courses for CCMG provincial officers in PVT methodology, as well as training for 83 district supervisors. On the administrative side, training was provided to eight provincial finance officers. In a train-the-trainers approach, CCMG supervisors went on to train 703 PVT monitors, 22 general observation roving monitors, and 300 stationary general observation monitors. In order to be selected, monitors had to reside within the proximity of the polling station for which they were recruited. In a significant change from the 2011 coalition, monitors had to be a member of one of the four member organizations of CCMG. Summary: NDI training to CCMG on PVT operations and the subsequent train-the-trainers contributed to the success of the PVT and the relatively successful 2015 election. NDI’s DC￾based elections and information and communications technologies staff members facilitated the training sessions, along with two regional consultants from NDI’s partner election monitoring organizations from the Zimbabwe Election Support Network (ZESN), and the Elections Observation Group (ELOG) in Kenya. However, the training was aimed less at building long￾term capacity than at ensuring the PVT was successfully carried out. Organizations that had been part of the 2011 coalition, with which NDI had worked with for years, were not able to build on the experience they had gained four years earlier. Given the importance of a credible PVT that needed to be organized in a short period of time, this may have been the appropriate strategy. However, the experience did show that the interests of long-term capacity building of local CSOs are sometimes in conflict with the need to ensure a positive strategic outcome. 82 ANSWERS TO EVALUATION QUESTIONS QUESTION 1: ARE THERE NOW MORE LOCAL CIVIL-SOCIETY ORGANIZATIONS ACTIVE IN: A) ELECTIONS ASSISTANCE; B) POLITICAL PROCESSES; AND C) PROVIDING TECHNICAL ASSISTANCE TO POLITICAL PARTIES THAN THERE WERE SIX YEARS AGO? Given the lack of a comprehensive database, the short answer is ‘no one knows.’ As the case study shows, however, there is a greater importance given now to CSO coalitions. Sidney Watae, Governance Advisor for USAID, expressed his belief that there are more CSOs, but says “there is no database to confirm it.” Greg Saili, USAID Civil Society Specialist, estimates there are about 600 registered CSOs, although he does not think all of these are active. However, the number of CSOs seem less important in Zambia than the number and quality of CSO coalitions. This is a point covered in the answer to Question 4. QUESTION 2: HOW WERE PROGRAMMATIC STRATEGIES DEVELOPED UNDER CEPPS II ALTERED OR EXPANDED (E.G., DIFFERENT APPROACHES TO ACTIVITIES, THE USE OF SUB-GRANTS) UNDER CEPPS III TO ADDRESS LOCAL CAPACITY BUILDING GOALS UNDER OBJECTIVE 10? In Zambia, the CEPPS III mechanism was not substantially altered. It was appreciated by USAID, but more for its programming flexibility and nimbleness than for capacity building. According to Watae and Saili of USAID, the CEPPS mechanism provided the flexibility to assemble the Caritas-led CCMG coalition. The 90 day window between the former president’s death and the by-elections meant that USAID had to mobilize quickly. “The CEPPS mechanism lends itself to this,” Watea said. “Partners were already pre-competed. A mechanism requiring new procurement would not work with such a fast turn-around,” he added. “We didn’t have to look around for other partners.” USAID needed groups that were already organized and on the ground. Churches had been providing services and had networks in place and there was “no issue of capacity.” CEPPS III has been extended for 24 months (through July 2017) so as to focus on the upcoming elections, voter registration and political parties. Most of the work will be through NDI, although IFES will have a small role to work with the ECZ. Watea said that capacity building was inherently part of the by-election activity, “but it couldn’t be a goal.” Capacity building has to have at least six months, he said, and a 90 day window is not enough. There is a challenge to use short term funds for capacity building. The Elections and Political Processes Fund (EPP) is looking for short term rather than long-term capacity building results.83 As Traci Cook of NDI explained, “Capacity building doesn’t apply in this 83 The Elections and Political Processes (EPP) Fund provides assistance for critical, unanticipated electoral and political processes worldwide. Since it was established in 2006, the EPP Fund has provided more than $208 million to 80 countries or regions. All USAID Missions may apply for support from the EPP Fund. To be considered for funding, Missions’ applications must meet at least two of the three following criteria: addresses snap elections or 83 case. There was too short time frame required.” The skills required for a PVT were too specialized for NDI to bring in more novice partners. To the extent there was capacity building during the 2015 elections, Caritas claimed responsibility for most of it. According to Caritas’s Samuel Mulafulafu, “We put CCMG together, and NDI supported it.” This was echoed by Father Justin Matepa of ECZ, who said that “NDI participated in the training, but Caritas does the training at the national level, and training at the parish level.” Was the CEPPS process for local partner selection optimal in the sense that competitive selection processes were balanced against the CEPPS need for establishing ongoing relationships? USAID was happy with 2015 coalition, despite the fact that partner selection was largely outsourced to Caritas. Watae of USAID pointed out that churches had led the campaign against former President Frederick Chiluba’s efforts to extend his term. Churches in Zambia are very political and have great reach, Watae said, and there is “no issue of capacity.” The fact that the coalition was religious was not seen as an issue. As long as faith based organizations are using funding for a social and non-religious agenda, USAID can fund them, Watae said. For purposes of expediency, the relatively successful 2015 coalition is likely to stay in place for the 2016 election cycle, with plans to retain most of the current structure, including provincial officers and the high performing district supervisors. Part of the calculus is that record keeping from the program activities, including hotels, vendor information, and transportation costs, particularly for events taking place outside Lusaka, will serve as valuable information for programmatic coordination ahead of the 2016 elections. To what extent were the approaches that USAID requested/recommended under Objective 10 appropriate for building local capacity? In Zambia, the flexibility of CEPPS III were considered more important than long-term capacity building. Although NDI has worked in Zambia since 1990, the sometimes unpredictable nature of the election cycle means that capacity building may take a back seat to getting the work done. The design of the 2014-15 program was built upon lessons learned from the 2011 elections program, which had used a broader, and arguably more representative, CSO coalition. The 2011 approach of building the capacity of the PVT coalition was seen as a failure, even among CSOs not chosen in 2015. According to McDonald Chipenze of FODEP, a participant in the 2011 coalition, NDI brought in other players to work with Caritas to build a coalition, causing problems at the outset. “The 2011 election coalition was forced, not natural. Forced marriages do not last.” According to Chipenze, there were too many meetings over management, resulting in a lack of progress. Caritas handled the money – while other CSOs were “surrogates.” Organizations would detail a staff person to Caritas, but the organization sending the person “was left behind.” other unanticipated needs; exploits a specific and narrow window of opportunity; and proposes unique and innovative approaches. 84 What were the reasons for better/worse follow through? Did CEPPS develop and implement any capacity-building approaches beyond those requested / recommended by USAID? If so, what were these approaches and what was the rationale for their development and adoption? While most of the capacity building training in 2015 was provided by Caritas, NDI participated throughout and brought the benefit of their international experience. NDI’s broad international network made this possible, and the international trainers were appreciated For instance, NDI brought in trainers from Zimbabwe and Kenya to talk about the PVT experiences there. According to Rev. Pukuta Mwanza of the Evangelical Fellowship of Zambia, “this was very practical and helpful.” Rev. Mwanza also cited training on observations in hard-to-reach areas as particularly helpful. The administrative aspects of capacity building were less appreciated given the tight timeline being faced. Ifoma Hamabele of Caritas/CCMG questioned the timing of some of the training regarding financial management. “The NDI trainer knew what he was taking about,” she said, “but we were busy.” The training was scheduled after we had already designed the budget.” It would have been more effective, she said, if the financial training involved reviewing drafts before they were finalized. Grants agreements, she said, should have better reflected Caritas’ operational systems. The need for financial training that incorporated existing CSO systems was a point also raised by other organizations which NDI had worked with in 2011. According to McDonald Chipenze of FODEP, financial capacity building training was helpful especially on financial reporting and allocation of time to staff. However, Chipenze said, CSOs have their own methods of management. Donors providing training should allow organizations to focus on own systems. Sometimes trainers didn’t have enough knowledge of the organizations they are training. FODEP feel there should be less focus on new principles and more on improving existing functions. “Auditors will be looking at whether you follow your own procedures.” Juliet Chibuta of the Zambian National Women’s Lobby (ZNWL), spoke about the 2011 coalition. Chibuta said the quality of the training NDI provided on financial management, project systems, project management, was good, but that “some donors don’t start training until after deliverables are due.” QUESTION 3: TO WHAT DEGREE DID CEPPS INVOLVE LOCAL PARTNERS IN THE DESIGN, AWARDS MANAGEMENT, M&E PROCESSES, AND REPORTING OF PROGRAMS WITH WHICH THEY WERE AFFILIATED? In the case of the 2015 election, NDI’s local partner, Caritas-Zambia, was involved in design, awards management, M&E processes and reporting. The compressed timetable of the 2015 by-election, coupled with the scale and reach of Caritas and the CCMG coalition, meant that Caritas was intimately involved in most aspects of program design and management. However, this was more for reasons of expediency than for capacity building. 85 To what extent have CEPPS local and regional partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? Although no references to best practices or gender analysis were explicitly mentioned during interviews, gender sensitivity was built into the program. NDI worked with the CCMG secretariat to ensure that questions for the long-term observers were designed to gather information on the nature of participation of women and other marginalized groups. The long term observer forms included three questions concerning whether observers witnessed any challenges to women candidates and whether voter education activities targeted women and other marginalized groups. In order to effectively integrate a gender component into the staffing of election observers and ensure balanced representation, NDI worked with the CCMG secretariat to establish clear guidelines for the recruitment of observers of both genders. However, a gender disparity within the PVT monitors group was more noticeable with a composition of 64 percent male, and 36 percent female associated with PVT-related trainings. This imbalance was primarily due to the fact that PVT monitors needed to be recruited within small geographical areas as assigned by the PVT representative sample of polling stations in order to not jeopardize the integrity of the PVT. The challenge of attaining a balanced gender representation was also compounded by the short timeframe for recruitment and training leading up to election-day. QUESTION 4: WHICH FACTORS CONTRIBUTE TO THE SUCCESSFUL ELEVATION OF LOCAL ORGANIZATIONS INTO SUSTAINABLE PROVIDERS OF TECHNICAL ASSISTANCE RELATED TO ELECTIONS AND POLITICAL PROCESSES? Unlike in the case of some evaluation questions and sub-questions that preceded it, key informants in Zambia were ready with answers to this question, especially when the question was framed in the context of comparing the 2011 and 2015 elections. CSO Coalitions CSO coalitions have been the key to development of political and elections focused civil society in Zambia. While most see coalitions as a positive development, there is some expressed wariness, especially when recalling the experience of 2011. According to most key informants, there was a rapid proliferation in the number of CSOs after the 1991 reemergence of a multi-party system in Zambia. By the late 1990s, donors, including USAID, began to encourage coalitions, partly to ease the management burden of working with so many nascent organizations. Watae of USAID says that CSOs have a better chance of getting money if they are part of a coalition. It is better for USAID to work with coalitions, Watae said, since it reduces the “management burden.” CSO coalitions, however, could be problematic. Traci Cook, Southern Africa Regional Director for NDI, spoke about the difficulties of working through 2011 CSEC coalition. Part of the problem from 2011, she explained, had to do with money. Sub-grants require a single organization to be financially responsible. In both 2011 and 2015 the lead CSO was Caritas, which in turn apportioned some of the sub-grant funding to coalition members. This was an 86 uncomfortable arrangement for some of the CSOs from 2011, who said they wanted to “touch the money” but could do so only indirectly. Edgar Mainza, a Program Officer for SACCORD, was a member of CSEC, and participated in the 2011 coalition. Some things worked well, he said, including coordination. Each member organization was given a location to monitor and there was no replication of effort. However, problems did result when consortium members were given monitoring assignments that involved overseeing organizations with different structures. For instance, SACCORD was given the southern province to monitor, but had to oversee “people who we didn’t know.” At a certain level, Mainza said, we felt SACCORD had influence within the coalition, but ultimately we knew we “just had to get on board.” The attitude was “If you don’t agree with us you can leave the coalition.” Administrative Capacity: An important factor, especially in the eyes of NDI, is CSO administrative capacity, or internal governance. According to Keith Jennings, Southern and East Africa Director in NDI’s Washington headquarters, organizations have strong governance, are inclusive and are part of representative networks. In Zambia, he said, some organizations can be described as “briefcase organizations”, with weak administration. Many CSOs claim to have chapters throughout the country, but often do not in reality. However, Jennings was complementary of CCMG’s ability to aggregate information, independent of NDI and to report in a transparent way. The 2015 CCMG coalition, were able to manage resources and oversee of coalition of hundreds of observers, determining who was best suited to monitor across the country. Commitment to the Mission: While the church organizations that made up the 2015 CCMG coalition were less products of CEPPS capacity building than the groups from 2011, NDI agreed to fund them because of their unquestioned commitment to the mission of civil society and electoral integrity in Zambia. Churches have a “moral authority that general CSOs do not have”, Cook of NDI explained. NDI was also aware that Caritas trusted the church groups that made up CCMG more than the non-church CSOs they worked with previously.84 Church organizations also have the ability to attract funding between elections. “There is no magic bullet,” says NDI’s Cook, but the ability to attract a steady source of bridge funding between elections may be the most critical factor in CSO sustainability. One of the reasons church groups are successful is that they have an independent source of funding, from local parishioners and their international affiliates. Moreover, church-affiliated CSOs are perceived as less likely to be motivated primarily by finances. An increasing problem, Jennings says, speaking about Zambia and other countries in which NDI works, is that civil society is often “monetized”. For example, people want per diem (in same city where they live) or a sitting fee for attending a meeting. This expectation has been generated over the years, as some local CSOs are increasingly focused on the financial rewards of their work. This, Jennings say, “makes donors part of the corruption problem.” CSOs, he says, have mastered the art of saying what westerners want to hear in order to receive funding. 84 FODEP was the only other group that did anything other than CCMG in 2015, receiving a small amount of direct funding from USAID. 87 Church groups are perceived as more dedicated to their respective organizations. The majority of those interviewed in Zambia praised the relative efficiency of the church organizations in conducting PVTs. According to Rev. Pukuta N. Mwanza of the Evangelical Fellowship of Zambia (EFZ), the majority of volunteers recruited for his organization were from Evangelical churches and were supervised by Evangelical pastors. This relationship seemed to have created a cohesion not present in the 2011 coalition. As Rev. Mwanza put it the EFZ portion of the PVT was a “family effort” where “no one was a stranger.” According to Father Justin Matepa of the Episcopal Conference of Zambia (ECZ), who participated in both coalitions, the current coalition (CCMG) is more effective than its predecessor. “The background of the four groups is the same (faith based). The previous coalition had less goodwill and comfort with itself.” Enabling Environment: A positive “enabling environment,” is also important. Zambia has provided a relatively welcoming space for CSOs for 25 years. Although Zambia has some restrictive laws in place, such as the Zambian Public Order Act, which requires notification of police for even small gatherings, Zambia’s political environment did provide for freedom of movement and expression at a level beyond many African countries. Avoidance of Founders Syndrome: A problem also noted in Nigeria and in other countries, is “founders syndrome.” According to McDonald Chipenzi of FODEP, a member of the 20ll PVT coalition, many now-defunct CSOs in Zambia had management problems caused by too much reliance on a single founder. If anything happened to that person, or if a donor felt funds were being mismanaged, donors would withdraw funding. Sometimes the CSO founder would become political, or be closely allied to a political party. If the political landscape changed, support might evaporate. Founder syndrome is also characterized by CSOs employing relatives of the founder, financial mismanagement and poor oversight. FODEP, an organization originally stood up by NDI, has had six presidents and four executive directors. This history, Chipenze says, proves that FODEP does not have “Founder;s Syndrome”. FODEP has a constitution, bi-laws board manuals, and HR manuals, and actually expelled a president in 2011. According to Traci Cook of NDI, this was a conscious effort on the part of NDI as it set up FODEP and nurtured it over the years. “We didn’t dictate FODEP’s structure,” she said, “but we told them what a professional international NGO structure is.” A key feature for FODEP is the separation between the secretariat and the board. As Cook put it, “if there is no split, there is no one to fire an executive director.” Considering what is required for potential success, what are the characteristics of the local and regional organizations receiving CEPPS III support? In the case of Zambia in 2015, success depended on church organizations. USAID and NDI’s primary CSO partner for both the 2011, 2015 and 2016 election programs is CARITAS Zambia, a Catholic organization that is part of the Zambia Episcopal Conference. Caritas Zambia is a member of Caritas Internationalis is a confederation of 165 Catholic relief, development and social service groups operating in over 200 countries and territories worldwide. Caritas is in no way dependent on USAID. According to its website it receives funding from “Caritas Norway, CAFOD, Norwegian Church AID, OSISA, GIZ, Canadian Development and Peace, Caritas Internationalis, ZGF and USAID.” Caritas is a partner with no discernable sustainability issues. 88 USAID’s partnership with it was driven more by the imperative for a successful PVT than a need to provide CSO capacity. According to Samuel Mulafulafu, Executive Director at Caritas Zambia, successful coalitions require like-minded and structured organizations, such as church groups. Referring to the experience of 2011, when Caritas led a coalition of mostly non-church groups, he said there are a number of CSOs that can “only exist in the coalition,” and have no independent base or constituency. “Sustainable CSO need to really believe in the cause,” he said “not just be looking for funding.” Mulafulafu said that the CCMG volunteers have greater accountability than members of non￾church CSOs They are recruited at the local level,are members of known parishes and Caritas can check up on them through their parish leaders. QUESTION 5: WHAT EFFORTS ON BEHALF OF CEPPS HAVE BEEN MADE TO SUPPORT LOCAL ORGANIZATION OPPORTUNITIES FOR BECOMING DIRECT RECIPIENTS OF DONOR FUNDING? In Zambia, there is skepticism that direct funding would be successful in the long-run, mainly because of the difficulty required to manage such an arrangement. Sydney Watae of USAID, with the perspective of a small DG office in a small Mission, is skeptical of more direct funding to local CSOs. USAID already has to work through multiple mechanisms to support CSOs, he says. Direct funding would be a big management burden. “Is USAID willing to shoulder the risk if money is stolen or mismanaged,” he asks; “is it worth the goal of a larger number of CSOs?” Traci Cook of NDI warned against the temptation to assume that “once CSO capacity was built, everything will be fine.” There needs to be an acknowledgement, she said, that “money can be stolen and that groups can become political. Direct funding should be done with an understanding of those two risks. “USAID cannot do due diligence to keep track of all these CSOs,” Cook said. “USAID is not on the ground enough on a day-to-day basis to hear about rumors of political affiliation and other issues regarding CSOs.” To what extent has CEPPS made efforts to assist local organizations become prepared to apply and receive larger and more complicated awards? The sub-granting process is one of the primary CEPPS III instruments of building capacity, due to the rigorous reporting requirements,. Sub-grants have been a major tool for helping local organizations become familiar with the complexities of larger awards. Traci Cook of NDI maintains that much progress has been made through the sub-granting process, particularly in financial reporting. “Our reporting requirements through sub-grants are as strict as what USAID requires,” she says. Sub-grants, says Keith Jennings of NDI, may not be a specific part of a capacity building plan, “but it is what we try to do.” Once you master sub-grant process, Jennings says, “you should be able to submit a proposal, and a budget” directly to donors. Caritas, Jennings says, now has capacity to get direct funding. Despite its limitations, NDI’s Cook says “I don’t see an alternative to the sub-grant.” For an effort as specialized as a PVT, a single sub-grant is necessary. However, for less technical tasks, such as voter education, Cook 89 says, there is more leeway and multiple sub-grants for separate programs targeting women, youth, and others are possible. - What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative and project management practices)? One theme heard in Zambia, and in other case study countries, was a desire for CEPPS funding to cover administrative as well as technical costs. Currently, CEPPS provides only minimal funding for any overhead costs for its sub-grant recipients. Keith Jennings of NDI, stresses the need to “appreciate the intersection between the administrative and political”, addressing both political and organizational capacity building. In his view, administrative assistance is as important as technical ability in helping CSOs become sustainable. Discussions on Funding Overhead: A common theme among CSOs was the relative lack of funding from sub-grants that could be used for administrative, or overhead, purposes. According to Rev. Mwanza of the Evangelical Fellowship of Zambia, there were inadequate finances for the CCMG Steering Committee on which he served. Most funding, he said, was directed at field operations. There was also inadequate funding, he said, for incidentals and fuel. Sydney Watae of USAID points out that there is no process in place to establish a NICRA for local organizations – a mechanism that allows US-based partners to bill for overhead costs. Nonetheless, Traci Cook of NDI concedes that “to make a large-scale consortium work, we should cover some administrations costs of all organizations” not just the lead organization. There need to be more categories of administration allowed, she said. 90 APPENDIX I – PEOPLE INTERVIEWED USAID Greg Chitambo Saili, Civil Society Specialist Sydney Watae, Governance Advisor Implementer – National Democratic Institute Traci Cook - Senior Advisor, Southern and East Africa Keith Jennings - Senior Associate and Regional Director for Southern and East Africa Civil Society Organizations Caritas Martin Chileshe Sampa – Unit Head – Quality Assurance, M&E Eugene Kabilika – Unit Head – Programs Samuel Mulafulafu – Executive Director Ifoma Hamabele – Director of CCMG Episcopal Conference of Zambia (ECZ) Father Justin Matepa Evangelical Fellowship of Zambia (EFZ) Pukuta Mwanza – Executive Director Foundation for Democratic Process (FODEP) McDonald Chipenzi - Executive Director Peter Mwamangombe – Program Officer Southern African Centre for the Constructive Resolution of Disputes (SACCORD) Edgar Mainza, Program Officer Zambian National Women’s Lobby Juliet Kaira Chibuta, Executive Director ANNEXES ANNEX I: TASKING AND SOW USAID DRG-LER Tasking Request N019: Performance Evaluation of the Consortium for Elections and Political Processes Strengthening III Leader Award (CEPPS III LWA) Date of Request: (September 15, 2014; Base year) Type of Task: Performance Evaluation Services SOW: USAID seeks to better understand how the CEPPS III LWA mechanism has contributed to the capacity of local and regional organizations based in the developing world to support elections and political transitions in their own countries on an ongoing basis as well as to assist efforts in other countries. By reviewing work done under both the Leader Award and associate awards, this evaluation will inform USAID of achievement of results included as part of the CEPPS III LWA Objective 10, identify factors contributing to and hindering achievement of those results, and make recommendations for future program management and design. For further direction, please reference attached Scope of Work. Dates of performance and timeline: September 29, 2014: Proposal including team and workplan to DRG Center October 2014: In-briefing for DRG Center March 1, 2015: Draft report submitted March 31, 2015: Report Completed, out-briefing for the DRG Center Deliverables: Design Proposal/Workplan In-briefing Draft Evaluation Report Final Evaluation Report and out-briefing Budget: The budget for this task is not to exceed $300,000. Submission Instructions: You will have ten (10) business days to submit a brief proposal and budget. EVALUATION STATEMENT OF WORK Effectiveness of Support to Local and Regional Organizations (Objective 10) Through the CEPPS III LWA SUMMARY AND PURPOSE USAID seeks to better understand how the CEPPS III LWA mechanism has contributed to the capacity of local and regional organizations based in the developing world to support elections and political transitions in their own countries on an ongoing basis as well as to assist efforts in other countries. By reviewing work done under both the Leader Award and associate awards, this evaluation will inform USAID of achievement of results included as part of the CEPPS III LWA Objective 10, identify factors contributing to and hindering achievement of those results, and make recommendations for future program management and design. BACKGROUND In 2008, USAID awarded a Leader with Associates (LWA) award to the Consortium for Political Process Strengthening (CEPPS), a partnership of the International Republican Institute (IRI), the National Democratic Institute for International Affairs (NDI), and the International Foundation for Electoral Systems (IFES). This award, commonly referred to as CEPPS III because it followed a previous LWA awarded to the same group in 2001 known as CEPPS II and a cooperative agreement with CEPPS awarded earlier still, quickly became the most frequently utilized mechanism managed by USAID’s Center of Excellence on Democracy, Human Rights and Governance. The CEPPS III LWA (DFD-A-00-08-00350) is fully funded with a ceiling of $18,285,902. USAID Missions and other operating units have separately awarded 92 associate awards, as of the first quarter of FY 2014, with $527,708,095 in obligations to date and a total estimated cost of $594,497,955. CEPPS III’s original expiration date in September 2013 has been extended until September 2014. The period of performance for associate awards may extend up to five years past the expiration of the leader award. Therefore the predominant role of CEPPS in USAID’s global support in the elections and political transitions subsector over the last two decades will extend for several more years as work continues under country-specific associate awards. The ten objectives of the CEPPS III LWA supporting objectives related to elections and political processes on a global basis are summarized below: Objective 1 Developing an impartial legal framework for elections and political parties Objective 2 Building a sustainable, indigenous capacity to effectively administer elections Objective 3 Informing and empowering all members of society to become active participants in the political process Objective 4 Supporting local, regional and international efforts to monitor electoral processes Objective 5 Ensuring inclusive political participation wherein a greater range of groups, including women and other historically excluded populations, influence and participate in political processes Objective 6 Promoting consensus-building for peaceful agreement on democratic reform, rules and framework through peaceful, broad￾based participation in defining and negotiating changes to governing structures Objective 7 Strengthening political parties to ensure a competitive multi-party system that represents the diversity within a country Objective 8 Fostering the smooth transfer of power following elections Objective 9 Strengthening governance capacity of elected leaders and bodies Objective 10 Building capable and sustainable local and regional civil society organizations engaged in election assistance and in providing technical assistance to political parties and political processes Eight of these objectives are virtually identical to objectives of the CEPPS II LWA, but Objective 10 is unique to CEPPS III and represented a new priority in the democracy and governance sector. Although pre-dating the USAID Forward initiative for supporting local solutions, that is, making awards directly to capable organizations in host countries to address development needs, the CEPPS III LWA is consistent with this current USAID priority. The full description for Objective 10 as included in the program description of the CEPPS III award is: USAID aims to strengthen the capacity of indigenous local/regional organizations in developing countries to: 1) conduct elections related activities; and 2) provide technical assistance and training to other local organizations on elections and political processes. The rationale is that strengthened election-oriented organizations in developing countries will: 1) broaden the community of democracy promoters and advocates abroad; 2) demonstrate that USAID assistance leads directly to sustainable change among local organizations, without dependence on intermediary organizations; and 3) promote replication. The specific anticipated results incorporated into the award were Improved technical and organizational capacities of local civil society and regional organizations to implement and sustain election-related programs over the long term. This includes strengthened abilities to compete for, administer and account for resources from foundations, U.S. nonprofits, international organizations, and international financial institutions, and bilateral donors. and Local (non-U.S.) civil society organizations provide technical assistance and training to other local organizations on elections and political processes activities. [emphasis in original] To accomplish these objectives, the CEPPS III LWA offers the implementing consortium and its partners a variety of tools. Under the Leader Award, global field support may take the form of assessments and program design, state-of-the-art research in elections and political processes, evaluations, knowledge management and transfer of experience, and sub-grants to indigenous local/regional organizations. The Leader Award also contains a “rapid response” function to quickly provide key support in suddenly emerging country contexts unanticipated in the annual work plan when normal programming processes are insufficient. In 2011, the Global Women’s Leadership Fund was established as part of this Leader Award with the dedicated purpose of improving women’s participation in consensus-building processes. Through associate awards, CEPPS has provided technical assistance, sub-grants, international exchanges, and other means to involve local and regional partners in achieving specific results of each award. Associate awards have also included capacity building of local organizations as a specific activity. Exactly how CEPPS has utilized a combination of approaches to achieve the results specified under Objective 10 of the LWA, what those results are, and the sustainability of those results is partially obscured by the variety of means employed to achieve them under various agreements. USAID therefore seeks a better, evidence-based understanding of how the CEPPS III LWA mechanism has contributed to the capacity of local and regional organizations based in the developing world to support elections and political transitions in their own countries on an ongoing basis as well as to assist efforts in other countries. USAID seeks an evaluation of its work through the CEPPS III LWA mechanism to enable local and regional organizations not just 1) to conduct activities strengthening elections and political processes in their own countries but also 2) to become sustainable actors contributing to democratic change by providing technical and/or financial assistance to other organizations and 3) to becoming financial viable through donor and non-donor funding. The purpose of this evaluation is to inform USAID of successes and best practices in elevating organizations active in the sub-sector of elections and political transitions into sustainable leaders for democratic change in order to improve future program designs, to assess past performance, and to make recommendations for USAID Missions and CEPPS on how to accelerate achievement of results under Objective 10 under the ongoing associate awards in their final years, to the extent possible. EVALUATION DESIGN The primary programmatic subject of the evaluation will be work conducted by the Consortium for Election and Political Process Strengthening under the both the CEPPS III Leader Award and through CEPPS III associate awards. Evaluation Questions USAID seeks objective, evidence-based answers to the following key evaluation questions to inform its decision-making on future program design, assess DRG’s contribution towards USAID Forward’s local solutions initiative, and improve the achievement of results under the CEPPS III LWA. Local Capacity Building: Are there now, in CEPPS host countries, more local civil society organizations active in elections assistance, political processes, and in providing technical assistance to political parties than there were six years ago? If so, have many of the newer or newly involved organizations worked with CEPPS institutions or their local partners? CEPPS’s Strategic Approach: How did CEPPS alter or expand the programmatic strategies it developed under CEPPS II, its first LWA with nearly identical objectives, to effectively achieve the additional objective on local capacity building added in the CEPPS III LWA? What differences, if any, are evident in the consortium’s approach to activities under the Leader Award and associate awards, including the use of sub-grants to local organizations under each? Was CEPPS’s process for identifying and selecting local partner organizations optimal? How did CEPPS balance competitive selection processes against establishing ongoing relationships? Nature of Partnerships: To what degree did CEPPS involve local partners in the design of programs they were affiliated with? To what degree were partners involved in the management of awards, monitoring and evaluation processes, and the preparation of periodic progress reports (quarterlies)? Did CEPPS incorporate organizational capacity- building elements into their collaborations on technical activities or were they separated into a dedicated parallel track independent of the work on elections and political processes? To what extent have CEPPS’s local and regional partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? Factors Leading to Success: Which factors can be identified as contributing to the successful elevation of local organizations into sustainable actors able to deliver technical assistance on elections and political processes and/or conduct activities in (or outside of) their home countries? What are the characteristics or prerequisites of the local and regional organizations receiving support through the CEPPS LWA mechanism which align most clearly with potential for success? Which approaches among the many options in the CEPPS LWA “toolkit,” individually or in combination, show the greatest success rate for achieving the results specified for Objective 10? Sustainability and Financial Management: What efforts have been made to support local organizations in becoming direct recipients of donor funding? Have efforts been made to help local organizations’ planning in order to be prepared to apply for and receive larger and more complicated awards? What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative and project management practices)? Evaluation Products The Evaluation Team is responsible for ensuring the quality of the evaluation products consistent with USAID's Evaluation Policy. The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively assess what worked in the project, what did not, and why. Evaluation reports shall address all evaluation questions included in this Statement of Work. The evaluation report should include this Statement of Work as an annex. All modifications to the Statement of Work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the Contracting Officer’s Representative. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an annex in the final report. Evaluation findings will assess outcomes and impact on males and females. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. Sources of information need to be properly identified and listed in an annex. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. The following products are considered necessary deliverables for the success of this evaluation: 1) An in-briefing at USAID/DCHA/DRG at the beginning of the evaluation. During this briefing, the Evaluation Team will present its work plan. 2) An out-briefing at USAID/DCHA/DRG at the conclusion of research. During this briefing, the Evaluation Team will present its key findings and conclusions and draft recommendations. 3) A draft evaluation report in English of no more than 30 pages containing the findings, conclusions, and recommendations addressing the evaluation questions established in this Statement of Work and reflecting issues raised by USAID/DCHA/DRG during the in-briefing and out-briefing. The evaluation report should include a list of the local organizations found by the evaluation team to have applied for donor funding, have received donor funding, and of those who have received donor funds, which were USAID and which were achieved competitively. The draft evaluation report should be submitted to the Contracting Officer’s Representative in an acceptable electronic format within twenty working days of the completion of field research. Each page of the draft evaluation report should clearly be marked “DRAFT.” USAID will provide comments on the draft evaluation report within ten working days of receipt. 4) A final evaluation report in English of no more than 30 pages containing the findings, conclusions, and recommendations addressing the evaluation questions established in this Statement of Work and reflecting issues raised in USAID’s comments on the draft evaluation report. The final evaluation report should also include an executive summary of 2-4 pages, a table of contents, and the required annexes including a copy of this Statement of Work, a detailed description of evaluation design and methods, copies of actual data collection tools such as survey or interview questionnaires, a list of information sources (including documents reviewed, sites visited, and key informants), disclosure of any conflicts of interest, and additional annexes at the discretion of the Evaluation Team. The final evaluation report should be submitted to the Contracting Officer’s Representative in an acceptable electronic format (MS Word, PDF, Google Doc) within 10 working days of the completion of field research. The final evaluation report must conform with USAID’s Evaluation Policy, ADS 203, and the federal government’s Plain Language requirements. The Contracting Officer’s Representative will not accept as final any report which does not meet acceptable quality standards. It is the responsibility of the Evaluation Team to submit the final evaluation report to the Development Experience Clearinghouse (including any Statements of Differences which may be subsequently generated in response to the final evaluation report’s recommendations) within 30 days of the Contracting Officer’s Representative’s acceptance of the final evaluation report. Team Composition The Evaluation Team should be composed of individuals collectively qualified to competently answer the evaluation questions contained in this Statement of Work and to produce the required final evaluation report. The Evaluation Team will certify that there is no conflict of interest or potential conflict of interest with respect to the performance of this evaluation on the part of the Evaluation Team. AWARD PARAMETERS Period of Performance The final evaluation report should be submitted no later than March 31, 2015. Budget USAID/DCHA/DRG has up to $300,000 available for this award. ANNEX II.A: CEPPS AWARD MAP ANNEX II.B: COUNTRY CLASSIFICATION TABLE Country Political Environment Media training, financial support Training, advise, financial support Support for campaigns, research Financial and technical support Financial and technical support Support to CS promotion of civic education around elections Support to CS capability to monitor and observe elections to facilitate free and fair contests Increase and expand CS (and citizen) capability to participate in democratic processes and dialogue Strengthen CS capability to engage in advocacy and reform efforts Support CS assistance to political parties and establish linkages with parties Angola NF 1 Armenia PF 1 Azerbaijan NF 2 2 Bangladesh PF 1 Belarus NF 1 Bhutan* PF 1 BiH PF 1 1 1 Burkina Faso PF Burundi PF 1 1 Cambodia NF 1 Colombia PF 1 Egypt NF 1 Georgia PF 1 1 Ghana N/A 1 Guatemala PF 1 Guinea* PF Guyana F Haiti PF 2 Honduras PF 1 Indonesia F Iraq NF Jordan* NF 1 1 1 Kazakhstan NF 2 Kenya PF 1 1 1 1 Kosovo PF 1 1 Kyrgyzstan PF Lebanon PF 2 1 Libya NF* Macedonia PF Malawi PF 2 2 2 Maldives PF* 1 1 Mali PF 1 Mauritania NF Mexico* PF 1 Morocco PF 1 Nepal PF Nicaragua PF Niger PF 1 1 1 Sierra Leone* PF South Sudan NF Sri Lanka PF 1 1 Sudan NF Syria NF Tajikistan NF 1 1 Tunisia* PF Togo PF 1 Uganda PF Ukraine* PF Uzbekistan NF 1 1 Yemen* NF 1 1 1 Zambia PF 1 1 Zimbabwe NF TABLE 2. NO CSO-RELATED LOCAL CAPACITY BUILDING OBJECTIVE Country Political Environment Conflict prevention and resolution Improving institutional capacity for effective governance International observation for elections, political processes, and governing institutions Gauging public perceptions through conduct of research Promoting public awareness Inclusion of minorities, IDPs, youth, women, the disabled in political processes Angola NF 1 Armenia PF Azerbaijan NF 1 Belarus NF Bhutan* PF BiH PF Burkina Faso PF Burundi PF Cambodia NF Colombia PF 1 Georgia PF 3 2 Ghana N/A Guatemala PF 1 1 1 1 Guinea* PF 1 Guyana F Haiti PF 1 1 Honduras PF Indonesia F 1 Iraq NF 1 Jordan* NF Kazakhstan NF 1 2 Kenya PF 1 1 Kosovo PF 1 Kyrgyzstan PF Lebanon PF Libya NF* 1 Macedonia PF Malawi PF 1 Maldives PF* Mali PF 1 1 Mauritania NF Mexico* PF 1 Morocco PF 1 1 1 1 Nepal PF 1 1 Nicaragua PF Niger PF 1 Sierra Leone* PF 1 1 Somalia NF South Sudan NF Sri Lanka PF Sudan NF 1 1 Syria NF Tajikistan NF Togo PF Tunisia* PF Uganda PF Ukraine* PF 1 Uzbekistan NF Yemen* NF Zimbabwe NF 1 The scores for “Political Environment” reflect a composite Freedom House rating that measures whether countries are “Free” (F), “Partially Free” (PF), or “Not Free” (NF). Table 4.1.4. Regressions for the average age of CSOs between 2009 and 2014 CSOs Electorally focused CSOs Politically focused CSOs both electorally and politically focused VARIABLES Average CSO Age (a) Average CSO Age, (Country Fixed-Effects) Average CSO Age (a) Average CSO Age (Country Fixed-Effects) Average CSO Age (a) Average CSO Age (Country Fixed-Effects) Year 1.519 3.811*** -0.316 4.879*** -1.467 5.209*** (2.434) (0.698) (2.386) (0.812) (2.925) (0.822) Number of CEPPS Electoral CSOs 2.933*** -2.590*** (0.789) (0.593) Number of CEPPS Political CSOs 3.602*** -2.516*** (0.762) (0.691) Number of CEPPS grantees 2.280** -1.098*** (0.923) (0.276) Constant 10.86* 18.78*** 10.66* 18.44*** 11.82* 14.62*** (5.828) (1.872) (5.583) (2.047) (6.120) (1.390) Observations 95 95 95 95 95 95 R-squared 0.238 0.983 0.300 0.981 0.172 0.982 Notes: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. (a) Covariates include GDP per-capita; Economist Intelligence Unit democracy index (free; not-free; partially free); number of NGOs per country, per year; whether or not IFES, NDI, or IRI implemented in-country; whether or not they had an election the previous year; average age of CSOs in-country. An important caveat here is that the total number of CSOs remains constant over time. ANNEX III: METHODOLOGY Informed Consent and Confidentiality Each survey questionnaire began with an informative introductory statement that describes to respondents the subject of the survey and some basic details about the confidential and voluntary nature of their participation. For example, the introduction informed respondents about CEPPS and the purpose of the survey, and a statement that their participation is voluntary, that their responses will remain confidential and used in aggregated summaries only, that they may skip questions they do not feel comfortable answering, and time required to complete the survey. We provide this information to respondents so that they may give an informed consent to participate, which is consistent with NORC’s professional commitment as members of the American Association for Public Opinion Research (AAPOR). Survey instruments and the interview protocol are included in Annexes III and IV. Implementation The final survey instruments were programmed into NORC’s web survey program (Liberty) and pretested in English by NORC staff to ensure all question text, response options, logic skips and validations were working properly. After finalizing the CSO survey platform in English, the Arabic, French, Russian, and Spanish versions of the questionnaire were uploaded to Liberty and pretested by the evaluation team and NORC staff. All typos, glitches, and bugs were amended. Prior to launching the survey, the evaluation team sent out an initial email message to all respondents in the sample in order to 1) introduce the evaluation to respondents and let them know when they could expect to receive the survey links, and 2) to capture a list of respondents with invalid email addresses. For CSO respondents, the initial letter came from the presidents of IRI, NDI, and IFES; for the implementers, this letter came directly from the president of their respective organization; and for USAID staff respondents the letter came from Jeff Vanness, Democracy and Governance Field Advisor. For those respondents who were unreachable by email either due to having an outdated or invalid email, the evaluation team reached out to USAID/CEPPS or elsewhere to obtain current email addresses; those for whom we were not able to obtain valid email addresses were dropped from the sample. All email campaigns were sent through SendGrid, a service which automatically tracks all invalid emails, bounce-backs (i.e. emails that are returned to sender because the email addresses are no longer in service or have automatic spam-blocking software), and otherwise undeliverable emails. One benefit of SendGrid is the ability to track the “unique open rate,” which tracks the number and percentage of unique respondents that have opened each email send via SendGrid. For the initial email, NORC found that on average, of the five introductory emails sent out in each language an average of 45% of them were opened by respondents. This suggests that only around 55% of the CSO emails NORC received from the implementers were actively monitored, and 45% of respondents never opened the email. The sample was then updated with current email addresses and by deleting individuals who could not be located due to invalid or outdated emails, or who were considered “out of scope,” such as secretarial staff from the organization or others who had little knowledge of organization activities. On 8/06/15, the production server went live and the CSO and AOR surveys were launched; because the implementer survey instrument took longer to finalize, the survey for implementers was not launched until 8/27/15. When each survey was launched, respondents received a second email, this time addressed from Katharine Mark at NORC, containing a unique survey link for each respondent to complete the questionnaire online. The respondents were able to click on the link and gain access to the survey. They were able to complete any part of it and then resume where they left off. The CSO and AOR survey were live for approximately 7 weeks due to the delay of the implementer survey, and the implementer survey was live for 4 weeks. During this period, respondents of each stakeholder group received two additional automated e-mail reminders from NORC encouraging them to complete the survey. The evaluation team received and responded to questions from respondents about the evaluation including issues with links or other logistical issues, concerns about the confidentiality or legitimacy of the survey, and confirming that questionnaires had been properly submitted. For USAID staff, an additional email of encouragement was sent via USAID in an attempt to boost low response rates. Response rates for are presented below in Table 3.1. Note that response rates were calculated as the number of partial and fully completed surveys divided by the sample size. Table 3.1: Response rate Sample Size Non￾Responder Partial Completion Fully Completed Response Rate CSO Survey 1,018 608 129 281 40.3% Implementer Survey 128 36 29 63 71.9% AOR Survey 62 27 5 29 54.8% The web survey resulted in a response rate of 40.3% for the CSO survey; 56.4% for the AOR survey; and 71.9% for the implementer survey. The response rate for the CSO survey is considered especially high because of the low unique open rate we had for CSO respondents: while only around 55% of total respondents received and opened the email (the unique open rate), 73.3% of those respondents actually completed or partially completed the survey, bringing the total CSO response rate to 40.3%. (This implies that the major problem was that email addresses may have been out of date or incorrect, but that respondents were relatively positive about responding.) The implementer response rate was similarly impressive, with 71.9% of all implementers providing partial or complete response data despite having the shortest field period of 4 weeks. The response rate for the AOR survey, however, only received 34 usable responses, despite sending out multiple automated reminders from NORC and targeted reminders from USAID. ANNEX IV: DRAFT KII PROTOCOLS Key Informant Interview Questions (CEPPS Field Staff) Introduction: Thanks for agreeing to be a key informant to provide insight regarding the perspective of CEPPS implementers – NDI, IRI, and IFES – on the CEPPS mechanism. We have been contracted by USAID’s DRG bureau to evaluate a limited number of aspects of their CEPPS3 program. The evaluation is interested in the degree to which the capacity of civil-society organizations (CSOs) participating in CEPPS awards has evolved, both substantively and organizationally. It is emphatically not focused on whether the CEPPS implementers – NRI, IRI, IFES – could have done a better job or whether they fulfilled their contractual targets, to the extent any were set. Toward this end the questions focus on the relationship between CEPPS implementer and their local CSO partners. Your responses will help us to: inform the contents of the CSO and local CEPPS office surveys, which are the main sources of our analysis; And acquire critical background information regarding the overall CEPPS program, including leader-level, cross-country award activities. If you prefer, you can choose anonymity and the option of speaking “off the record,” If you chose to do so, data will be coded and stored according to NORC’s strict IRB procedures so that your identity cannot be matched to the interview content, except by the interviewer. IDENTIFYING QUESTIONS What is the nature of your involvement with the CEPPS3 mechanism and its partner CSOs? How long have you been involved with CEPPS3? Were you involved in CEPPS2? What CEPPS awards or programs have you worked on? Have you visited them? QUESTION 1 – NUMBERS OF ACTIVE CSOs 1.1 a.What are the main categories of elections assistance you’d say [IMPLEMENTER NAME] has provided? [IE election administration, voter education/information, EOMs, etc] In which of these main categories are CSOs active? b. Are more CSOs more active today in elections assistance than there were five years ago why do you think this is? To what extent would you say that CEPPS played a role? 1.2a Now let’s think assistance with strengthening political processes. Are a. What are the main categories of political process assistance you’d say [IMPLEMENTER NAME] has provided? [constituent outreach, issue-based ideology, polling, etc] In which of these main categories are CSOs active? b. Are there more CSOs active today in political process assistance than there were five years ago? If so, why do you think this is? To what extent would you say that CEPPS played a role? WHAT WOULD THIS INCLUDE? Would this include CSOs that are acting like political parties? 1.3. Given the broad latitude that each CEPPS associate award provides to [IMPLEMENTER NAME], to what extent does [IMPLEMENTER NAME] endeavor to: increase the number of CSOs with which it has worked within a CEPPS country? involve newer CSOs? 1.4 What country-level characteristics do you think most influence CSO formation? i.e. permissive environment, adequate funding opportunities? QUESTION 2 – CHANGES TO STRATEGIES 2.1.1 What factors determine the way in which capacity building of local and regional CSOs figures into how [IMPLEMENTER NAME] implements an award? i.e. as partners, subcontractors, subgrantees, etc. Publicly versus underground 2.1.2.What type of guidance has USAID provided regarding CSO capacity-building activity? (whether in the cooperative agreement, program descriptions, or other directives) What programmatic strategies were identified to accomplish those capacity-building aims? Please distinguish between substantive capacity – e.g., carrying out election assistance – and organizational capacity – e.g., financial and staff management. 2.1.3. What programmatic strategies were/are actually used under CEPPS III to build CSO capacity? Again, please distinguish between substantive capacity – e.g., carrying out election assistance – and organizational capacity – e.g., financial and staff management. 2.1.4. Are those capacity-building strategies different from strategies that were used under CEPPS II (altered, expanded, using subgrants, etc.)? I.E. NOT JUST BUILDING UPON PAST ACTIVITIES, BUT A MATERIAL CHANGE TO THE STRATEGIC APPROACH, BASED UPON LESSONS LEARNED, ETC? 2.2.1. How are local partner organizations selected in CEPPSIII, where not dictated by the PD? (They can mention several, perhaps used in different contexts). 2.2.2. Is there a tension between competitive selection processes and the need for CEPPS to support ongoing CSO relationships? If so, how were these balanced? 2.3.1 To what extent were awards that [IMPLEMENTER NAME] received appropriate for building local capacity? 2.3.2 To what extent did USAID request/recommend specific approaches for CSO capacity building? Are there instances where you disagreed with the recommended approach? 2.3.3 To what extent did [IMPLEMENTER NAME] follow through on the USAID requested/recommended capacity-building approaches? 2.3.4 What were the reasons for following through or not following through with the USAID requested approaches? 2.4.1 Did [IMPLEMENTER NAME] develop and implement any capacity-building beyond those explicitly requested/recommended by USAID in an award? If so, what were these approaches, what were their rationales for their development? 2.4.2 Where capacity-building was provided beyond that explicitly requested/recommended by USAID in an award what was it about the underlying award that made it conducive to such capacity building? QUESTION 3 – INVOLVEMENT OF LOCAL PARTNERS 3.1.1 How and to what degree does/did [IMPLEMENTER NAME] involve local CSOs in the design of programs within CEPPS awards with which they were affiliated? 3.1.2 To what degree did [IMPLEMENTER NAME] involve local CS5O partners in the awards management of programs within CEPPS awards with which they were affiliated? 3.1.3 To what degree did [IMPLEMENTER NAME] involve local CSO partners in the M&E processes of programs within CEPPS awards which they were affiliated? 3.1.4 To what degree did [IMPLEMENTER NAME] involve local CSO partners in the reporting of programs within CEPPS awards which they were affiliated? 3.2.1. If you were designing a CEPPS-like instrument, what might you say about CSO capacity development? 3.2.2. What types of capacity building does CEPPS provide (differentiate by Study Area as appropriate)? NOTE THAT WE ARE ASKING ABOUT “CEPPS”, NOT THE IMPLEMENTER] 3.2.3 Did [IMPLEMENTER NAME] integrate organizational capacity-building activities into their collaborations on elections and political processes, or did capacity building occur on more of a parallel track? 3.2.4 To what extent have CEPPS local and regional CSO partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? [PROBE: PLEASE NAME SOME OF THE MAIN “BEST PRACTICE” CAPACITY BUILDING OPPORTUNITIES IN EACH STUDY AREA.] What would you consider, based upon your experience, as “best practices” in capacity development opportunities in electoral processes? In political party processes? QUESTION 4 – FACTORS FOR LOCAL ORGANIZATION CAPACITY 4.1.1 Please name what you see as the key factors (e.g., CSO capacity along various dimensions —governance, administrative systems, finances, subject matter expertise, quality control) that contribute to the successful elevation of local organizations into sustainable providers of technical assistance related to elections and political processes. By sustainable we mean the ability to carry on without supervision from organizations such as yours and to raise their own resources 4.1.2 Please name some of the outcomes of CEPPS assistance (e.g., the number/type of CSO collaborations, awards, clients, and networks) that contribute to the successful elevation of local organizations into sustainable providers of technical assistance related to elections and political processes. [PROBE: YOU MIGHT WANT TO THINK SEPARATELY ABOUT SUCCESS IN (I) PROVIDING TECHNICAL ASSISTANCE, AND (II) BECOMING SUSTAINABLE]. 4.1.3. Does [IMPLEMENTER NAME] track any of the outcomes listed above? Which ones? How? 4.2.1. What are some of the key factors besides the substantive expertise and experience of its staff that might affect the likelihood of a CSO successfully improving organizational capacity (e.g., size, location, founder characteristics, age of organization)? 4.3.1. What CEPPS activities have contributed to this “success”? 4.3.2. What capacity-building/development modalities do you use, if any? ( i.e. NGO sustainability index, IDF, STAR, whatever PACT’s tool is named, etc. ) 4.3.3. What country characteristics have the greatest effect on a CSO's successful adoption of capacity building? 4.4. Are you yourself familiar with any exceptionally effective CSOs in the countries where you have worked? Specifically, we are asking about independent, financially sound, well-managed organizations who are active in either elections assistance or providing TA in political processes? Could you describe them? QUESTION 5 – FACTORS FOR CAPACITY BUILDING 5.1 What specific efforts – and how often – under CEPPS has [IMPLEMENTER NAME] explicitly supported local CSO opportunities for becoming direct recipients of donor funding? [PROBE: DIFFERENTIATE BY TYPE OF SUPPORT AND STUDY AREA] 5.2.1 To what extent has [IMPLEMENTER NAME] made efforts to assist local CSOs to become better prepared to apply and receive larger and more complicated awards? 5.2.2 What other efforts has [IMPLEMENTER NAME] made through CEPPS3 to increase sustainability of local CSO partners (such as efforts to build management, administrative and project management practices)? Key Informant Interview Questions (CEPPS Headquarters Staff) Introduction: Thanks for agreeing to be a key informant to provide insight regarding the perspective of CEPPS implementers – NDI, IRI, and IFES – on the CEPPS mechanism. We have been contracted by USAID’s DRG bureau to evaluate a limited number of aspects of their CEPPS3 program. The evaluation is interested in the degree to which the capacity of civil-society organizations (CSOs) participating in CEPPS awards has evolved, both substantively and organizationally. It is emphatically not focused on whether the CEPPS implementers – NRI, IRI, IFES – could have done a better job or whether they fulfilled their contractual targets, to the extent any were set. Toward this end the questions focus on the relationship between CEPPS implementer and their local CSO partners. Your responses will help us to: identify case study countries inform the contents of the CSO and local CEPPS office surveys, which are the main sources of our analysis; And acquire critical background information regarding the overall CEPPS program, including leader-level, cross-country award activities. If you prefer, you can choose anonymity and the option of speaking “off the record,” If you chose to do so, data will be coded and stored according to NORC’s strict IRB procedures so that your identity cannot be matched to the interview content, except by the interviewer. IDENTIFYING QUESTIONS What is the nature of your involvement with the CEPPS3 mechanism and its partner CSOs? How long have you been involved with CEPPS3? Were you involved in CEPPS2? What CEPPS awards or programs have you worked on? Have you visited them? Do your CEPPS offices abroad compile a list of CSOs working in electoral and/or political processes in their host country? [PAUSE BEFORE ASKING:] Would that be something that we could access? Does [IMPLEMENTER NAME] have similar CSO lists in non-CEPPS countries? Would that be something that we could access (but not share with any other organization)? Are you aware of any third-party databases with CSO lists that might cover election and political processes support? QUESTION 1 – NUMBERS OF ACTIVE CSOs 1 1.1 Let’s think about the countries in which [IMPLEMENTER NAME] provides elections assistance under CEPPS. Are you familiar with some of those awards? [IF NO, GO TO Question 1.2] Which ones? (may know this from Identifying questions above) a. What are the main categories of elections assistance you’d say [IMPLEMENTER NAME] has provided? [IE election administration, voter education/information, EOMs, etc] In which of these main categories are CSOs active? b. In those countries where there are more CSOs active today in elections assistance than there were five years ago why do you think this is? To what extent would you say that CEPPS played a role? c. In those countries where there are fewer CSOs active today in elections assistance than there were five years ago why do you think this is? To what extent would you say that there would have been even fewer had CEPPS played a role? 1.2a Now let’s think about the CEPPS countries in which [IMPLEMENTER NAME] provides assistance with strengthening political processes. Are you familiar with some of those awards? [IF NO, GO TO RESEARCH QUESTION 2] Which ones? (may know this from Identifying questions above) a. What are the main categories of political process assistance you’d say [IMPLEMENTER NAME] has provided? [constituent outreach, issue-based ideology, polling, etc] In which of these main categories are CSOs active? b. In those countries where there are more CSOs active today in political process assistance than there were five years ago why do you think this is? To what extent would you say that CEPPS played a role? WHAT WOULD THIS INCLUDE? Would this include CSOs that are acting like political parties? c. In those countries where there are fewer CSOs active today in political process assistance than there were five years ago why do you think this is? To what extent would you say that there would have been even fewer had CEPPS played a role? 1.3. Given the broad latitude that each CEPPS associate award provides to [IMPLEMENTER NAME], to what extent does [IMPLEMENTER NAME] endeavor to: a. increase the number of CSOs with which it has worked within a CEPPS country? b. involve newer CSOs? 1.4 What country-level characteristics do you think most influence CSO formation? i.e. permissive environment, adequate funding opportunities? 1 Headings are for the information of the interviewer. No need to inform interviewees. QUESTION 2 – CHANGES TO STRATEGIES 2.1.1 What factors determine the way in which capacity building of local and regional CSOs figures into how [IMPLEMENTER NAME] implements an award? i.e. as partners, subcontractors, subgrantees, etc. Publicly versus underground 2.1.2.What type of guidance has USAID provided regarding CSO capacity-building activity? (whether in the cooperative agreement, program descriptions, or other directives) What programmatic strategies were identified to accomplish those capacity-building aims? Please distinguish between substantive capacity – e.g., carrying out election assistance – and organizational capacity – e.g., financial and staff management. 2.1.3. What programmatic strategies were/are actually used under CEPPS III to build CSO capacity? Again, please distinguish between substantive capacity – e.g., carrying out election assistance – and organizational capacity – e.g., financial and staff management. THIS WILL VARY FROM COUNTRY TO COUNTRY, SO SHOULD WE DEFINE THOSE OF WHICH THE INTERVIEWER SPEAKS? 2.1.4. Are those capacity-building strategies different from strategies that were used under CEPPS II (altered, expanded, using subgrants, etc.)? I.E. NOT JUST BUILDING UPON PAST ACTIVITIES, BUT A MATERIAL CHANGE TO THE STRATEGIC APPROACH, BASED UPON LESSONS LEARNED, ETC? 2.2.1. How are local partner organizations selected in CEPPSIII, where not dictated by the PD? (They can mention several, perhaps used in different contexts). 2.2.2. Is there a tension between competitive selection processes and the need for CEPPS to support ongoing CSO relationships? If so, how were these balanced? 2.3.1 To what extent were awards that [IMPLEMENTER NAME] received appropriate for building local capacity? We are asking several things: how to quantify “to what extent”; what is “appropriate”; and thinking about building local capacity of CSOs—in the skills sense and in the OD sense 2.3.2 To what extent did USAID request/recommend specific approaches for CSO capacity building? Are there instances where you disagreed with the recommended approach? 2.3.3 To what extent did [IMPLEMENTER NAME] follow through on the USAID requested/recommended capacity-building approaches? 2.3.4 What were the reasons for following through or not following through with the USAID requested approaches? 2.4.1 Did [IMPLEMENTER NAME] develop and implement any capacity-building beyond those explicitly requested/recommended by USAID in an award? If so, what were these approaches, what were their rationales for their development? 2.4.2 Where capacity-building was provided beyond that explicitly requested/recommended by USAID in an award what was it about the underlying award that made it conducive to such capacity building? QUESTION 3 – INVOLVEMENT OF LOCAL PARTNERS 3.1.1 How and to what degree does/did [IMPLEMENTER NAME] involve local CSOs in the design of programs within CEPPS awards with which they were affiliated? 3.1.2 To what degree did [IMPLEMENTER NAME] involve local CSO partners in the awards management of programs within CEPPS awards with which they were affiliated? 3.1.3 To what degree did [IMPLEMENTER NAME] involve local CSO partners in the M&E processes of programs within CEPPS awards which they were affiliated? 3.1.4 To what degree did [IMPLEMENTER NAME] involve local CSO partners in the reporting of programs within CEPPS awards which they were affiliated? 3.2.1. If you were designing a CEPPS-like instrument, what might you say about CSO capacity development? 3.2.2. What types of capacity building does CEPPS provide (differentiate by Study Area as appropriate)? [INTERVIEWER: NOTE THAT WE ARE ASKING ABOUT “CEPPS”, NOT THE IMPLEMENTER] 3.2.3 Did [IMPLEMENTER NAME] integrate organizational capacity-building activities into their collaborations on elections and political processes, or did capacity building occur on more of a parallel track? 3.2.4 To what extent have CEPPS local and regional CSO partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? [PROBE: PLEASE NAME SOME OF THE MAIN “BEST PRACTICE” CAPACITY BUILDING OPPORTUNITIES IN EACH STUDY AREA.] What would you consider, based upon your experience, as “best practices” in capacity development opportunities in electoral processes? In political party processes? QUESTION 4 – FACTORS FOR LOCAL ORGANIZATION CAPACITY 4.1.1 Please name what you see as the key internal organizational factors (e.g., CSO capacity along various dimensions —governance, administrative systems, finances, subject matter expertise, quality control) that contribute to the successful elevation of local organizations into sustainable providers of technical assistance related to elections and political processes. By sustainable we mean the ability to carry on without supervision from organizations such as yours and to raise their own resources. [PROBE: YOU MIGHT WANT TO THINK SEPARATELY ABOUT SUCCESS IN (I) PROVIDING TECHNICAL ASSISTANCE, AND (II) BECOMING SUSTAINABLE.] 4.1.2 Please name some of the outcomes of CEPPS assistance (e.g., the number/type of CSO collaborations, awards, clients, and networks) that contribute to the successful elevation of local organizations into sustainable providers of technical assistance related to elections and political processes. [PROBE: YOU MIGHT WANT TO THINK SEPARATELY ABOUT SUCCESS IN (I) PROVIDING TECHNICAL ASSISTANCE, AND (II) BECOMING SUSTAINABLE]. 4.1.3. Does [IMPLEMENTER NAME] track any of the outcomes listed above? Which ones? How? 4.2.1. What are some of the key factors besides the substantive expertise and experience of its staff that might affect the likelihood of a CSO successfully Improving organizational capacity (e.g., size, location, founder characteristics, age of organization)? 4.3.1. What CEPPS activities have contributed to this “success”? [ADD IF APPROPRIATE TO THE CAPACITY-BUILDING ACTIVITIES ALREADY DESCRIBED ABOVE] 4.3.2. What capacity-building/development modalities do you use, if any? i.e. NGO sustainability index, IDF, STAR, whatever PACT’s tool is named, etc 4.3.3. What country characteristics have the greatest effect on a CSO's successful adoption of capacity building? 4.4. Are you yourself familiar with any exceptionally effective CSOs in the countries where you have worked? Specifically, we are asking about independent, financially sound, well-managed organizations who are active in either elections assistance or providing TA in political processes? Could you describe them? QUESTION 5 – FACTORS FOR CAPACITY BUILDING 5.1 What specific efforts – and how often – under CEPPS has [IMPLEMENTER NAME] explicitly supported local CSO opportunities for becoming direct recipients of donor funding? [PROBE: DIFFERENTIATE BY TYPE OF SUPPORT AND STUDY AREA] 5.2.1 To what extent has [IMPLEMENTER NAME] made efforts to assist local CSOs to become better prepared to apply and receive larger and more complicated awards? 5.2.2 What other efforts has [IMPLEMENTER NAME] made through CEPPS3 to increase sustainability of local CSO partners (such as efforts to build management, administrative and project management practices)? INFORMING THE CASE STUDIES 6.1 What criteria would you suggest us to use in picking case-study countries of [IMPLEMENTER NAME] CEPPS activities related to capacity building? 6.2 What countries do you think your criteria would select? ANNEX V: DATA COLLECTION INSTRUMENTS 29 FINAL USAID Evaluation of the Consortium for Elections and Political Process Strengthening (CEPPS) Draft Questionnaire for CSOs INTRO. PROG: PLEASE DISPLAY THE FOLLOWING MESSAGE ON ITS OWN PAGE: NORC, a research center associated with the University of Chicago, is conducting an independent evaluation of a program funded by the United States Agency for International Development (USAID), the Consortium for Elections and Political Process Strengthening (CEPPS) in ##P-COUNTRY##. As part of this evaluation, we are collecting information from program staff, implementers, and beneficiaries to help us better understand the CEPPS’s program activities and how the program can be improved to better meet the needs of CSOs like yours and the public you ultimately serve. Hence, your participation in this survey is important because it is one of the few ways available for getting direct feedback on program outcomes. The survey should take about 20-30 minutes to complete. Your participation in this survey is voluntary. All of your responses will be kept strictly confidential, and will only be used in a statistical summary. By clicking the ‘Begin Survey’ button at the bottom of the screen, you acknowledge that you understand that your participation in this survey is completely voluntary. If you have any questions about the survey, please e-mail Katharine Mark, Principal Research Scientist and Survey Manager, at USAIDpartnersurvey@norc.org so that we may assist you. PROG: INSERT NEXT, PREVIOUS, STOP BUTTONS ON THIS AND SUBSEQUENT PAGES PROG: END MESSAGE 30 PROG: START RESPONDENT SORTING/FILTERING SECTION Q0.0a. Which of the following best describes your current position at your organization? 1 Leadership of the organization 2 Board Member 3 Staff 4 Volunteer 96 Other, please specify: ________________ PROG: OPEN-ENDED Q0.0b1. Is there another individual who you think would be able to provide additional insight about your organization for this survey? 1 Yes 2 No PROG: SKIP TO Q0.1 Q0.0b2. PROG: ASK IF Q0.0b1==1. Please provide the contact information for this individual. If you do not know any of the below information, please leave that space blank. 1M First and last name: _________________________ PROG: OPEN-ENDED 2M E-mail address: _________________________ PROG: OPEN-ENDED 3M Telephone number: _________________________ PROG: OPEN-ENDED Q0.1. Does your organization currently carry out activities in the area of election processes for an upcoming election? By “election processes,” we refer to a range of activities including election observation, voter education, and electoral legal reform. 1 Yes 2 No, but did in the past 3 No PROG: SKIP TO Q0.3 Q0.2a. PROG: ASK IF Q0.1==1 OR Q0.1==2. In which year did your organization first engage in electoral-process activities? __________ PROG: 4-DIGIT YEAR 98 Do not know Q0.2b. PROG: ASK IF Q0.1==2. PROG: SELECT ALL THAT APPLY. Why doesn’t your organization any longer participate in electoral-process activities or initiatives? Please select all that apply. 1M No need for such activities 2M Unable to raise funding 3M Not enough experience or training in this area 4M Lost critical staff 5M Decided to concentrate on other areas 96M Other, please specify: ______________________ PROG: OPEN-ENDED 98 Do not know Q0.3. Does your organization currently conduct programs in the area of political processes? By “political processes,” we refer to a range of activities including training for political party leadership and 31 activists, civil society or media, building citizen awareness or government responsiveness, conducting political surveys, and strengthening civil society advocacy. Yes No, but did in the past No PROG: SKIP TO Q0.4a PROG: UPON ENTRY IF Q0.1==3 & Q0.3==3, SKIP TO Q1.3 Q0.3a. PROG: ASK IF Q0.3==1 OR Q0.3==2. In which year did your organization first engage in political-process activities? 1 _________ PROG: 4-DIGIT YEAR 98 Do not know Q0.3b. PROG: ASK IF Q0.3==2. PROG: SELECT ALL THAT APPLY. Why doesn’t your organization any longer participate in political-process activities or initiatives? Please select all that apply. 1M No need for such activities 2M Unable to raise funding 3M Not enough experience or training in this area 4M Lost critical staff 5M Decided to concentrate on other areas 96M Other, please specify: ______________________ PROG: OPEN-ENDED 98 Do not know Q0.4a. In which year was your organization first established? 1 _______ PROG: 4-DIGIT YEAR 98 Do not know Q0.4b. Please fill out the following table with estimates regarding the number of staff working for your organization in both the current period and in 2009 (or first year of operation if after 2009). 1 In Current period 2 In 2009 A. Permanent staff __ __ __ PROG: INTEGER 98 Do not know __ __ __ PROG: INTEGER 98 Do not know B. Temporary staff __ __ __ PROG: INTEGER 98 Do not know __ __ __ PROG: INTEGER 98 Do not know C. Board members __ __ __ PROG: INTEGER 98 Do not know __ __ __ PROG: INTEGER 98 Do not know D. Volunteers __ __ __ PROG: INTEGER 98 Do not know __ __ __ PROG: INTEGER 98 Do not know 32 PROG: START SECTION ON CSO ATTRIBUTES QA.1. PROG: SELECT ALL THAT APPLY. Please indicate the main activities your organization engages in. Please select all that apply. 1M Training of political party activists 2M Conducting election observation 3M Conducting advocacy campaigns 4M Media training 5M Public opinion polling and/or focus group discussions 6M Parallel vote tabulation (PVT) 7M Training of election management bodies 8M Political dialogue and/or debates 9M Voter education campaigns 10M Advocacy on elections law, political party law, and/or CSO law 96M Other, please specify: _______________ PROG: OPEN-ENDED QA.2. Is the work of your organization mostly concentrated in the months before and during the actual election campaigns and voting, or does it work more continuously? 1 Months directly before and during election 2 Continuously QA.2a. PROG: SELECT ALL THAT APPLY. Which of the following does your organization have? Please select all that apply. 1M Logo 2M Mission statement 3M Brochures or pamphlets describing the work of your organization 4M Building, office, or suite with functioning utilities 5M At least one transportation vehicle 6M At least one working computer 7M Website or Facebook Page 8M Up-to-date NGO registration or NGO license 9M Budget 10M Specialized staff (including accountants, M&E staff) 11M Support staff (such as secretary) QA.2b. To the best of your knowledge, what percent of your staff has completed university? 1 0-25% 2 26-50% 3 51-75% 4 76-90% 5 Over 90% QA.3. Has your organization ever supported or conducted programs in another country? 1 Yes 33 2 No 98 Do not know QA.4a. Has your organization ever received a contract or grant from a CEPPS III implementer such as IRI, NDI, or IFES? 1 Yes 2 No PROG: SKIP TO QB.1 QA.4b. PROG: ASK IF QA.4a==1. How many contracts or grants did your organization receive from a CEPPS III implementer (IRI, NDI, and/or IFES) during the period 2005-2008? If your organization did not receive any contracts or grants from a CEPPS III implementer during this period, please enter a value of “0”. _______ PROG: INTEGER PROG: IF 0, SKIP TO QA.4c PROG: START CONTRACT/GRANT LOOP FOR PERIOD 2006-2009, TO BE COMPLETED FOR UP TO THREE ITERATIONS DEPENDING ON VALUE OF QA.4b QA.4b1. PROG: ASK IF QA.4b>0. We would like to learn a little about the contracts/grants your organization received during the period 2005-2008. First, please think about the [most recent, second most recent, third most recent] contract/grant in that period. What type of contractual arrangement was this, a contract or a grant? 1 Contract 2 Grant QA.4b2. PROG: ASK IF QA.4b>0. How was funding obtained? 1 Competitive procurement 2 Required by USAID 3 Other 98 Don’t know PROG: RESTART OR END CONTRACT/GRANT LOOP FOR PERIOD 2006-2009. QA.4c.PROG: ASK IF QA.4a==1. How many contracts or grants did your organization receive from a CEPPS III implementer (IRI, NDI, and/or IFES) during the period 2009-Present? If your organization did not receive any contracts or grants from a CEPPS III implementer during this period, please enter a value of “0”. _______ PROG: INTEGER PROG: IF 0, SKIP TO QB.1 PROG: START CONTRACT/GRANT LOOP FOR PERIOD 2009-PRESENT, TO BE COMPLETED FOR UP TO THREE ITERATIONS DEPENDING ON VALUE OF QA.4c. QA.4c1. PROG: ASK IF QA.4c>0. We would like to learn a little more about the contracts/grants your organization received during the period 2009-Present. First, please think about the [most recent, second most recent, third most recent] contract/grant in that period. What type of contractual arrangement was this, a contract or a grant? 34 1 Contract 2 Grant QA.4c2. PROG: ASK IF QA.4c>0. How was funding obtained? 1 Competitive procurement 2 Required by USAID 3 Other 98 Don’t know PROG: RESTART OR END CONTRACT/GRANT LOOP FOR PERIOD 2009-PRESENT QB.1. The next questions refer to the most recent major elections your organization may have worked on over the 2009-2015 period. By “major elections,” we refer to presidential, parliamentary, and nationwide local or municipal elections. With that in mind, how many major elections did your organization work on during the 2009-2015 period? Please provide your best estimate. If your organization did not work on any major election during this period, please enter a value of “0”. _______ PROG: INTEGER PROG: IF 0, SKIP TO Q1.1 PROG: START CSO ACTIVITY LOOP, TO BE COMPLETED UP TO 3 TIMES DEPENDING ON RESPONSE IN QB.1. QB.1a.PROG: ASK IF QB.1>0. Please indicate the type of election your organization worked on [most recently, prior to the one just described]. 1 Presidential 2 Parliamentary 3 Local / Municipal QB.1b. PROG: ASK IF QB.1>0. In which year did this election take place? 1 2009 2 2010 3 2011 4 2012 5 2013 6 2014 7 2015 QB.2a. PROG: ASK IF QB.1>0. PROG: SELECT ALL THAT APPLY. Please select the types of activities your organization undertook for this election. Include both those specific to the election itself and work your organization may have done with political parties. Please select all that apply. 1M Election observation 2M Voter education 3M Electoral legal reform 4M Training of election observers 35 5M Parallel vote tabulation (PVT) 6M Media monitoring 7M Voter mobilization 8M Training for election commissions 9M Campaigning 10M Political debates 96M Other, please specify: _______________ PROG: OPEN-ENDED QB.2b. PROG: ASK IF QB.1>0. Using the table below, please estimate the total level of effort (total work time) that your organization staff worked during this election, including all staff and volunteers. Total time spent working on election 1 Less than one staff￾month (1-20 staff￾days) 2 Between one and 3 staff￾months (21-60 staff￾days) 3 Between 3 and 6 staff￾months (61- 120 staff￾days) 4 Between 6 and 10 staff￾months (120 – 200 staff-days) 5 More than 10 staff￾months (more than 200 staff -days) 97 Not applicable (No staff of this type or 0 staff￾days) 98 Do not know A. Permanent staff o O o o o O o B. Temporary staff o O o o o O O C. Volunteers o O o o o O O QB.2c. PROG: ASK IF QB.1>0. Was CEPPS (IRI, NDI, and/or IFES) involved with this election/activity? 1 Yes 2 No PROG: SKIP TO QB.2e QB.2d. PROG: ASK IF QB.2c==1. Did IRI, NDI, and/or IFES provide your organization with some funding for the activity, only technical assistance, or both? 1 Funding only 2 Technical assistance only 3 Both funding and technical assistance QB.2e. PROG: ASK IF QB.1>0. Please indicate the number of other international organizations/agencies (other than IRI, NDI, and IFES) that provided funding to your organization during this election. 1 ________ PROG: INTEGER 98 Do not know 36 QB.2f. PROG: ASK IF QB.1>0. Please indicate the number of local organizations/agencies that provided funding to your organization during this election. ________ PROG: INTEGER 98 Do not know PROG: RESTART OR END CSO ACTIVITY LOOP PROG: START SECTION ON RESEARCH QUESTION 1 (NUMBER OF ACTIVE CSOs) Q1.1. Do you think there are more local civil-society organizations (CSOs) working in the areas of elections and political processes active today than there were in 2009? 1 Yes 2 No PROG: SKIP TO Q1.2b 98 Do not know PROG: SKIP TO Q1.2c1 Q1.2a. PROG: ASK IF Q1.1==1. PROG: SELECT UP TO TWO. Why do you think there are more local civil-society organizations (CSOs) active today than there were in 2009? Please select no more than two (2) of the most important reasons. 1M Greater availability of funding 2M More open political environment for civil society 3M More frequent elections 4M More competitive elections 5M Changes in laws allowing for domestic observation 6M Greater capacity within civil society 96M Other, please specify: _______________ PROG: OPEN-ENDED 98 Do not know Q1.2b. PROG: ASK IF Q1.1==2. PROG: SELECT UP TO TWO. Why do you think there are fewer local civil-society organizations (CSOs) active today than there were in 2009? Please select no more than two (2) of the most important reasons. 1M Lack of funding 2M Better electoral processes 3M Worse electoral processes 4M Legislative changes prohibiting activity 5M Intimidation of civil society 6M More important activities 96M Other, please specify: __________ PROG: OPEN-ENDED 98 Do not know Q1.2c1. Please estimate the number of other local CSOs your organization had substantial collaborations with in 2009, or first year of operation if after 2009. 1 ____ PROG: INTEGER 98 Do not know Q1.2c2. Please estimate the number of other local CSOs your organization had substantial collaborations with in 2014. 37 1 ____ PROG: INTEGER 98 Do not know PROG: START SECTION ON RESEARCH QUESTION 2 (CHANGES TO STRATEGIES) Q2.1a. Did you ever lose a formal bidding competition for a grant or contract from CEPPS (IRI, NDI, and/or IFES)? 1 Yes 2 No PROG: SKIP TO Q2.2a Q2.1b. PROG: ASK IF Q2.1a==1. Did you feel that the selection process was fair? If you lost more than one formal bidding competition, please think about the most recent one. 1 Yes, it was a fair competition PROG: SKIP TO Q2.2a 2 No Q2.1c. PROG: ASK IF Q2.1b==2. Why do you feel that the selection process was not fair? Please select the statement below that most closely matches your view. 1 A CSO with a previous contract or grant was given preference 2 A less qualified CSO was selected 3 A politically connected CSO was selected 4 The scope of work or the rules of the procurement were not clear 96 Other, please specify__________________________ PROG: OPEN-ENDED FIELD Q2.2a. PROG: SELECT ALL THAT APPLY. PROG: ASK IF Q0.4a<=2008. Thinking back to the period 2005-2008, in what areas would you say that your organization most needed capacity building at that time? Consider both the organizational capacity and the technical capacity of your organization. Please select all topics which were needed. Organizational Capacity: 1M Financial management 2M Organizational management 3M Fundraising 4M Database management 5M Internal communication 96M Other, please specify: ____________ PROG: OPEN-ENDED Q2.2b. PROG: SELECT ALL THAT APPLY. PROG: ASK IF Q0.4a<=2008. Technical Capacity: 1M Survey and research skills 2M Methodology 3M Parallel vote tabulation (PVT) 38 4M Applying contextual analysis to elections and political processes 5M Advocacy/oversight 6M Public information campaigns 96M Other, please specify: ____________ PROG: OPEN-ENDED PROG: UPON ENTRY IF QA.4a==2 OR QA.4b==0, SKIP TO Q2.4a Q2.3a. PROG: ASK IF QA.4b>0 AND Q0.4a<=2008. What types of capacity building, if any, did your organization receive from IRI, NDI, and/or IFES (either through joint work with CEPPS or through stand-alone events) over the period 2005-2008? Please select all that apply. 1M Organizational capacity building 2M Technical capacity building 3 Neither organizational nor technical capacity building PROG: SKIP TO Q2.4a Q2.3b. PROG: ASK IF Q2.3a==1. What was the primary form of the organizational capacity building your organization received from IRI, NDI, and/or IFES over the period 2005-2008? 1 Technical assistance through joint work on a grant or contract 2 Stand￾alone Technical assistance 3 Training or workshop 4 Other 97 Not applicable A. Financial management O o O o O B. Organizational management O o O o O C. Fundraising O o O o O D. Database management O o O o O E. Internal communication O o O o O Q2.3c. PROG: ASK IF Q2.3a==1. Please give your assessment of the organizational capacity building your organization received from IRI, NDI, and/or IFES during the period 2005-2008. For each aspect below, please select the option that most closely matches your view. 1 Less than needed 2 About right 3 More than needed 97 Not applicable 98 Do not know A. Amount of Training o o o o o B. Amount of Hands-on assistance o o o o o C. Quality of experts o o o o o D. Range of topics covered o o o o o 39 E. Other/Specify:______________ PROG: OPEN-ENDED o o o o o Q2.3d. PROG: ASK IF Q2.3a==2. What was the primary form of the technical capacity building your organization received from IRI, NDI, and/or IFES over the period 2005-2008? 1 Technical assistance through joint work on a grant or contract 2 Stand￾alone Technical assistance 3 Training or workshop 4 Other 97 Not applicable A. Survey and research skills O o o o O B. Methodology O o o o O C. Parallel voter tabulation (PVT) O o o o O D. Voter mobilization O o o o O E. Applying contextual analysis to elections and political processes O o o o O F. Advocacy / oversight O o o o O G. Public information campaigns O o o o O Q2.3e. PROG: ASK IF Q2.3a==2. Please give your assessment of the technical capacity building your organization received from IRI, NDI, and/or IFES during the period 2005-2008. For each aspect below, please indicate the option that most closely matches your view. 1 Less than needed 2 About right 3 More than needed 97 Not applicable 98 Do not know A. Amount of Training o o o o o B. Amount of Hands-on assistance o o o o o C. Quality of experts o o o o o D. Range of topics covered o o o o o E. Other/Specify:______________ PROG: OPEN-ENDED o o o o o 40 Q2.4a. PROG: SELECT ALL THAT APPLY. In what areas would you say that your organization most needed capacity building during the period 2009-Present? Consider both the organizational capacity and the technical capacity of your organization. Please select all the topics which were needed. Organizational Capacity: 1M Financial management 2M Organizational management 3M Fundraising 4M Database management 5M Internal communication 96M Other, please specify: ____________ PROG: OPEN-ENDED Q2.4b. PROG: SELECT ALL THAT APPLY. Technical Capacity: 1M Survey and research skills 2M Methodology 3M Parallel Vote Tabulation (PVT) 4M Applying contextual analysis to elections and political processes 5M Advocacy/oversight 6M Public information campaigns 96M Other, please specify: ____________ PROG: OPEN-ENDED PROG: UPON ENTRY IF QA.4a==2, SKIP TO Q3.10a PROG: UPON ENTRY IF QA.4c==0, SKIP TO Q2.7 Q2.5a. PROG: ASK IF QA.4c>0. PROG: SELECT ALL THAT APPLY. What types of capacity building, if any, did your organization receive from IRI, NDI, and/or IFES (either through joint work with CEPPS or through stand-alone events) over the period 2009-Present? Please select all that apply. 1M Organizational capacity building 2M Technical capacity building 3 Neither organizational nor technical capacity building PROG: SKIP TO Q2.6 Q2.5b. PROG: ASK IF Q2.5a==1. What was the primary form of the organizational capacity building your organization received from IRI, NDI, and/or IFES over the period 2009-Present? 1 Technical assistance through joint work on a grant or contract 2 Stand￾alone Technical assistance 3 Training or workshop 4 Other 97 Not applicable A. Financial management O o o o O 41 B. Organizational management O o o o O C. Fundraising O o o o O D. Database management O o o o O E. Internal communication O o o o O Q2.5c. PROG: ASK IF Q2.5a==1. Please give your assessment of the organizational capacity building your organization received from IRI, NDI, and/or IFES during the period 2009-Present. For each aspect below, please indicate the option that most closely matches your view. 1 Less than needed 2 About right 3 More than needed 97 Not applicable 98 Do not know A. Amount of Training o o o o o B. Amount of Hands-on assistance o o o o o C. Quality of experts o o o o o D. Range of topics covered o o o o o E. Other/Specify:______________ PROG: OPEN-ENDED o o o o o Q2.5d. PROG: ASK IF Q2.5a==2. What was the primary form of the technical capacity building your organization received from IRI, NDI, and/or IFES over the period 2009-Present? 1 Technical assistance through joint work on a grant or contract 2 Stand￾alone Technical assistance 3 Training or workshop 4 Other 97 Not applicable A. Survey and research skills O o o o O B. Methodology O o o o O C. Parallel voter tabulation (PVT) O o o o O D. Voter mobilization O o o o O E. Applying contextual analysis to elections and political processes O o o o O 42 F. Advocacy / oversight O o o o O G. Public information campaigns O o o o O Q2.5e. PROG: ASK IF Q2.5a==2. Please give your assessment of the technical capacity building your organization received from IRI, NDI, and/or IFES during the period 2009-Present. For each aspect below, please indicate the option that most closely matches your view. 1 Less than needed 2 About right 3 More than needed 97 Not applicable 98 Do not know A. Amount of Training o o o o o B. Amount of Hands-on assistance o o o o o C. Quality of experts o o o o o D. Range of topics covered o o o o o E. Other/Specify:______________ PROG: OPEN-ENDED o o o o o Q2.6. PROG: ASK IF (QA.4b>0 AND QA.4c>0). Would you say that the capacity building needs of your organization were better addressed by IRI, NDI, and/or IFES in the period 2005-2008, or in the period 2009-Present? 1 Better addressed in the period 2005-2008 2 Addressed about the same in both periods 3 Better addressed during the period 2009-Present 98 Do not know 99 No response Q2.7. PROG: ASK IF QA.4a==1. To what extent have the capacity-building approaches used by IRI, NDI, and/or IFES been useful? 4 Very useful 3 Somewhat useful 2 Not very useful 1 Not at all useful 98 Do not know 99 No response Q2.8. PROG: ASK IF QA.4a==1. To what extent have the practices introduced by IRI, NDI, and/or IFES actually been adopted and used by your organization? 1 Most practices introduced have been adopted and used 2 Some practices have been adopted and used 3 Very few practices has been adopted and used 43 4 No practice has been adopted and used 98 Do not know PROG: START SECTION ON RESEARCH QUESTION 3 (INVOLVEMENT OF LOCAL PARTNERS) Q3.1. PROG: ASK IF QA.4a==1. In your work with CEPPS (IRI, NDI, and/or IFES), how was it determined which activities your organization would undertake? If your organization has worked on multiple CEPPS programs, please consider the most recent one. Please select the sentence that is closest to your organization’s experience. 1 Implementer had a clear idea of what your organization’s role should be and your organization agreed 2 Your organization had planning sessions with the implementer to determine what needed to be done and what your organization could best do 3 Implementer approached your organization with options 4 Your organization negotiated with the implementer to define activities 5 Your organization rejected the implementer’s proposed activities, but suggested alternatives 6 Your organization proposed the activities and they were accepted 98 Do not know Q3.2a. PROG: ASK IF QA.4a==1. The design of program activities within the award on which your organization and CEPPS III implementer (IRI, NDI, and/or IFES) are collaborating typically entailed elaboration of detailed work plans, timelines, and budgets. Did IRI, NDI, and/or IFES give any specific training or mentoring to your organization with regard to how to design the programs of the award on which your organizations collaborated? 1 Yes 2 No 98 Do not know Q3.2b. PROG: ASK IF QA.4a==1. To what extent did IRI, NDI, and/or IFES involve your organization in the design of programs on which your organizations collaborated? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know Q3.3a. PROG: ASK IF QA.4a==1. The award on which your organization and CEPPS III implementer (IRI, NDI, and/or IFES) are collaborating requires management of its program activities. Such management typically entails overseeing activities, ensuring quality of deliverables, and tracking of budgets. Did IRI, NDI, and/or IFES give any specific training or mentoring to your organization with regard to how to provide the management associated with the programs of the award on which your organizations collaborated? 44 1 Yes 2 No 98 Do not know Q3.3b. PROG: ASK IF QA.4a==1. To what extent did IRI, NDI, and/or IFES involve your organization in the awards management of programs on which your organizations collaborated? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know Q3.4a. PROG: ASK IF QA.4a==1. The award on which your organization and CEPPS III implementer (IRI, NDI, and/or IFES) are collaborating is funded by USAID. USAID requires that program activities within these awards be subject to monitoring and evaluation (M&E) processes. These processes typically entail the establishment of operational indicators as well as targets, milestones, and a host of related data collection. Did IRI, NDI, and/or IFES give any specific training or mentoring to your organization with regard to how to conduct the M&E processes associated with the programs of the award on which your organizations collaborated? 1 Yes 2 No 98 Do not know Q3.4b. PROG: ASK IF QA.4a==1. To what extent did your IRI, NDI, and/or IFES involve your organization in the monitoring and evaluation (M&E) processes of programs on which your organizations collaborated? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know Q3.5a. PROG: ASK IF QA.4a==1. In addition to the important services both your organization and CEPPS III implementer (IRI, NDI, and/or IFES) provide under the award, USAID, the donor requires that program activities and outcomes be described and written up in a series of quarterly and annual reports and that management, M&E, and financial reports be submitted. Did IRI, NDI, and/or IFES give any specific training or mentoring to your organization with regard to how to carry out the reporting associated with the programs of the award on which your organizations collaborated? 1 Yes 2 No 98 Do not know 45 Q3.5b. PROG: ASK IF QA.4a==1. To what extent did IRI, NDI, and/or IFES involve your organization in the reporting requirements of programs on which your organizations collaborated? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know Q3.6a. PROG: ASK IF QA.4a==1. One aspect of the democratization process relates to the empowerment of females, both women and girls. Hence, some projects begin with a “gender analysis”. This analysis entails examining the differential impact of the projects on women and men. Did IRI, NDI, and/or IFES give any specific training or mentoring to your organization with regard to how to conduct the gender analysis associated with the programs of the award on which your organizations collaborated? 1 Yes 2 No 98 Do not know Q3.6b. PROG: ASK IF QA.4a==1. To what extent has your organization incorporated gender analysis into its work with IRI, NDI, and/or IFES? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know Q3.6c. PROG: ASK IF QA.4a==1. Did your organization normally incorporate gender analysis into its work prior to collaborating with IRI, NDI, and/or IFES? 4 Usually 3 Sometimes 2 Rarely 1 Never 98 Do not know Q3.7. PROG: ASK IF QA.4c>0. During the period from 2009 through 2014, to what extent do you think that the responsibilities of your organization expanded in your work with IRI, NDI, and/or IFES? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know 46 Q3.9. PROG: ASK IF Q2.3a!=3 OR Q2.5a!=3. Did IRI, NDI, and/or IFES ever consult your organization on what capacity building it required, or was the assistance provided part of a standard package aimed to serve all CSOs in the country that were likely to attend? 1 Organization was consulted 2 Workshops were not specially tailored to your organization Q3.10a. Does your organization have a timesheet system, where staff record the hours they work on projects and various overhead functions? 1 Yes 2 No PROG: GO TO Q3.11a Q3.10b. PROG: ASK IF Q3.10a==1. Who fills out the timesheet, each staff person or someone else? Each full-time staff member completes his or her own timesheet A department head or project director The head of your organization Someone else Q3.11a. In the budgets you prepare, does your organization use a consistent overhead rate that is accepted by USAID contractors or another international donors? 1 Yes 2 No PROG: GO TO Q3.12 98 Do not know PROG: GO TO Q3.12 Q3.11b. PROG: ASK IF Q3.11a==1. In which year did you first prepare an overhead rate that USAID or another international donor accepted? ________ PROG: INTEGER 98 Do not know Q3.12. Please fill in the following table with the estimated number of grants and contracts that your organization received in 2005-2008 and 2010 - 2014. Year Number of grants/contracts from local clients Number of grants/contracts from international clients 2005-2008 |___| PROG: INTEGER 98 Do not know |___| PROG: INTEGER 98 Do not know 2010-2014 |___| PROG: INTEGER 98 Do not know |___| PROG: INTEGER 98 Do not know Q3.13. PROG: SELECT ALL THAT APPLY. Please identify below any areas in organization administration or project management (as opposed to the substantive service you provide related to election and/or political processes) in which you think your organization still could use help. Please select all that apply. 1M Financial management 47 2M Hiring procedures 3M Communications / public relations 4M Fundraising 5M Database management 6M Monitoring and evaluation (M&E) 7M Report writing 96M Other, please specify: _________________ PROG: OPEN-ENDED PROG: START SECTION ON EVALUATION RESEARCH QUESTION 4 (FACTORS AFFECTING LOCAL ORGANIZATION CAPACITY) Q4.1. PROG: SELECT UP TO 3. The ability to survive and flourish in the long term is clearly an important objective for any CSO. This is generally referred to as “sustainability”. There are no doubt particular obstacles to its sustainability that your organization faces. From the following factors – or those you add – please identify up to three (3) principal obstacles that you believe threaten the sustainability of your organization in the coming two years. Please select up to 3 Lack of endowment or financial reserves to maintain core staff between grant and contract awards |__| Little interest from citizens in our activities |__| Lack of specialized staff and/or staff turnover |__| Lack of funding sources |__| Lack of sufficient knowledge of what international clients need in order to hire us |__| Lack of sufficient knowledge on how to obtain funding from international clients |__| Failure of international donors to adapt to local and cultural contexts |__| Our administrative systems do not meet requirements of international clients. |__| Lack of subject-matter expertise |__| Intimidation from governmental or political influences. |__| Difficulty in mobilizing a sufficient number of temporary additional personnel |__| Difficulties with our service quality control |__| 48 Please select up to 3 Lack of consistency from international donors of services required over time |__| Lack of adequate funding from international donors |__| Lack of appropriate capacity building from international donors |__| Lack of knowledge on how to access funds from international donors |__| Lack of knowledge/experience in fundraising |__| Our organization does not have a command of the languages of key donors to submit proposals |__| Political and/or economic instability |__| Other/Specify: ______________________________________ PROG: OPEN-ENDED |__| Other/Specify: ______________________________________ PROG: OPEN-ENDED |__| Q4.2. PROG: SELECT UP TO 3. The results of the work of your organization may not depend solely on your organization. In your organization’s operating environment, what would you identify as the key factors increasing the likelihood that outcomes are successful once your organization has begun an activity? Please identify from among the following factors up to three (3) – or those you add – that you believe are the most critical to your organization for ensuring successful outcomes. Factor Please select up to 3 Collaborations with other local CSOs |__| Absence of government or political intimidation |__| Strong internal administrative systems |__| Expertise of senior staff and supervisors |__| Adequate funding from international funders |__| Capacity building assistance, mentoring, and advice from international organizations |__| 49 PROG: UPON ENTRY IF QA.4a=2 GO TO Q1.3 PROG: START SECTION ON RESEARCH QUESTION 5 (FACTORS FOR LOCAL ORGANIZATION CAPACITY) Q5.1a. PROG: ASK IF QA.4a==1. Did IRI, NDI, and/or IFES ever introduce your organization to a new potential funder – either via email, a telephone call, or in-person – in order to facilitate establishing a relationship with the donor? Yes No 98. Do not know Q5.1b. PROG: ASK IF QA.4a==1. Did IRI, NDI, and/or IFES work with your organization to prepare customized materials to give certain potential donors? Yes No 98. Do not know Q5.1c. PROG: ASK IF QA.4a==1. Did IRI, NDI, and/or IFES work with your organization to identify what activities your organization could provide that would be of strong interest to other donors? Yes No Access to skilled staff and volunteers |__| Other, please specify: _______________________________________ PROG: OPEN-ENDED |__| Other, please specify: ________________________________________ PROG: OPEN-ENDED |__| 50 98. Do not know Q5.1d. PROG: ASK IF QA.4a==1. Did the assistance of IRI, NDI, and/or IFES directly lead to your organization successfully obtaining new funds from another donor? Yes No 98. Do not know Q5.2a. PROG: ASK IF QA.4a==1. Overall, to what extent has IRI, NDI, and/or IFES made efforts to support your organization in becoming direct recipients of donor funding? 4 To a great extent 3 To some extent 2 To a small extent PROG: GO TO Q5.3 1 Not at all PROG: GO TO Q5.3 98 Do not know PROG: GO TO Q5.3 Q5.2b. PROG: ASK IF Q5.2a==3 OR Q5.2a==4. Could you please describe in a few words some of what IRI, NDI, and/or IFES did that was helpful in supporting your organization in becoming a direct recipient of donor funding? _______________________________ PROG: OPEN-ENDED FIELD Q5.3. PROG: ASK IF QA.4a==1. To what extent has IRI, NDI, and/or IFES made efforts to help your organization to become capable of applying for and receiving larger and more complicated awards? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know Q5.4. PROG: ASK IF QA.4a==1. What other efforts have IRI, NDI, and/or IFES made to increase the sustainability of your organization? 1 _______________________________ PROG: OPEN-ENDED FIELD 2 None 98 Do not know PROG: START SECTION ON OBTAINING INFORMATION ON OTHER ORGANIZATIONS Q1.3. We are interested in learning more about CSOs who work in this area and would like your help in identifying other CSOs active in your country. In the following table, please provide the names and contact information (if you have it) of three of the newest CSOs you are aware of that conduct activities in the areas of national elections and/or political processes. If you cannot think of any CSOs to list, please leave this question blank. If you know fewer than three, leave the remaining rows blank in the table. 51 A B C D CSO name and city with main office Contact name Email Phone number 1 ______________________ PROG: OPEN-ENDED _______________ PROG: OPEN￾ENDED 98 Do not know _______________ PROG: OPEN￾ENDED 98 Do not know _______________ PROG: OPEN￾ENDED 98 Do not know 2 ______________________ PROG: OPEN-ENDED _______________ PROG: OPEN￾ENDED 98 Do not know _______________ PROG: OPEN￾ENDED 98 Do not know _______________ PROG: OPEN￾ENDED 98 Do not know 3 ______________________ PROG: OPEN-ENDED _______________ PROG: OPEN￾ENDED 98 Do not know _______________ PROG: OPEN￾ENDED 98 Do not know _______________ PROG: OPEN￾ENDED 98 Do not know Q1.4. In the following table, please provide the names and locations of main offices (if you have them) of up to three CSOs you are aware of that conducted activities in the areas of national elections and/or political processes but no longer exist, and the reason that most closely matches why these CSOs no longer exists. If you do not know of any CSOs that worked in this area and no longer exist, please leave this question blank. If you know fewer than three, leave blank the remaining rows in the table. 52 A B C CSO name City with main office Reason for demise 1 ___________________ __ PROG: OPEN-ENDED _______________ PROG: OPEN￾ENDED 98 Do not know 1 No more funding 2 Poor management 3 Merged or was absorbed by another organization 4 Poor skills or training 5 Founder no longer with CSO 6 High staff turnover/attrition 7 Shut down by government body 8 Lack of office space or equipment 96 Other, please specify: _______ PROG: OPEN-ENDED 98 Do not know 2 ___________________ __ PROG: OPEN-ENDED _______________ PROG: OPEN￾ENDED 98 Do not know 1 No more funding 2 Poor management 3 Merged or was absorbed by another organization 4 Poor skills or training 5 Founder no longer with CSO 6 High staff turnover/attrition 7 Shut down by government body 8 Lack of office space or equipment 96 Other, please specify: _______ PROG: OPEN-ENDED 98 Do not know 3 ___________________ __ PROG: OPEN-ENDED _______________ PROG: OPEN￾ENDED 98 Do not know 1 No more funding 2 Poor management 3 Merged or was absorbed by another organization 4 Poor skills or training 5 Founder no longer with CSO 6 High staff turnover/attrition 7 Shut down by government body 8 Lack of office space or equipment 96 Other, please specify: _______ PROG: OPEN-ENDED 98 Do not know PROG: DISPLAY CLOSURE MESSAGE: Thank you for participating in our survey! PROG: END MESSAGE 53 USAID Evaluation of the Consortium for Elections and Political Process Strengthening (CEPPS) Draft Questionnaire for USAID AORs INTRO. PROG: PLEASE DISPLAY THE FOLLOWING MESSAGE ON ITS OWN PAGE: USAID has retained NORC, a research center associated with the University of Chicago, to conduct an independent evaluation of its CEPPS program. As part of this evaluation, we are collecting information from USAID technical staff, implementers, and beneficiaries to better understand your perspective on the results, outcomes, issues and challenges of the CEPPS programming in ##P-country##. Hence, your participation in this survey is important because it is one of the few ways available for getting direct feedback on program outcomes. The survey should take about 20 minutes to complete. All of your responses will be kept strictly confidential, and will only be used in a statistical summary. By clicking the ‘Begin Survey’ button at the bottom of the screen, you acknowledge that you understand that your participation in this survey is completely voluntary. If you have any questions about the survey, please e-mail Katharine Mark, Principal Research Scientist and Survey Manager, at USAIDpartnersurvey@norc.org so that we may assist you. PROG: INSERT NEXT, PREVIOUS, STOP BUTTONS ON THIS AND SUBSEQUENT PAGES PROG: END MESSAGE 54 PROG: START SECTION ON RESPONDENT SORTING/FILTERING Q0.1. What is (or was) your role with the CEPPS III program in ##P-country##? If you worked on multiple CEPPS III programs in ##P-country##, please think of the most recent CEPPS III program you worked on. 1 AOR 2 Alternate AOR 96 Other, Please specify: ___________________ PROG: OPEN-ENDED 97 I have not been involved with that program PROG: SKIP TO CLOSURE Q0.2a. In which year did you first have responsibility for any CEPPS program (either CEPPS II or CEPPS III) in ##P-country##? _______________ PROG: INTEGER Q0.2b. In which year did your role for your most recent CEPPS III project in ##P-country## end? 1 _______________ PROG: INTEGER 97 Still ongoing Q0.3. In which year did you first begin working for USAID? _______________ PROG: INTEGER Q0.4. What is your status with USAID? 1 FSN (foreign service national) PROG: SKIP TO Q1.1 2 FSO (foreign service officer) 3 PSC (personal services contractor) 4 TCN (third country national) Q0.5. PROG: ASK IF Q0.4≠1. For how many years have you been in ##P-country##? _______________ PROG: INTEGER PROG: START SECTION ON RESEARCH QUESTION 1 (NUMBER OF ACTIVE CSOs) Q1.1. According to USAID’s CSO sustainability index: “For a CSO sector to be sustainable, the legal and regulatory environment should support the needs of CSOs. It should facilitate new entrants, help prevent governmental interference, and give CSOs the necessary legal basis to engage in appropriate fundraising activities and legitimate income-producing ventures.” With this in mind, how supportive to CSOs do you believe the legal and regulatory environment is in ##P-country##? 4 Very supportive 3 Somewhat supportive 55 2 Not very supportive 1 Not at all supportive 98 Do not know Q1.2. Do you know of any local CSOs that conduct (or conducted) activities in election processes (such as election observation, voter education, or electoral law/regulatory reform) or political processes (such as training of political party leadership, monitoring political parties, or conducting advocacy) in ##P-country##? 1 Yes 2 No PROG: SKIP TO Q1.6 Q1.3. PROG: ASK IF Q1.2==1. PROG: SELECT ALL THAT APPLY. Please indicate the activities such CSOs engage in. Please select all that apply. 1M Training of political party activists 2M Conducting election observation 3M Conducting advocacy campaigns 4M Media training 5M Public opinion polling and/or focus group discussions 6M Parallel vote tabulation (PVT) 7M Training of election management bodies 8M Political dialogue and/or debates 9M Voter education campaigns 10M Advocacy on elections law, political party law, and/or CSO law 96M Other, please specify: ___________________ PROG: OPEN-ENDED Q1.4. PROG: ASK IF Q1.2==1. If you can, please provide the names and contact information of up to three of the primary CSOs you are aware of that currently conduct activities in the areas of elections and/or political processes in ##P-country##. If you cannot think of any of the below information, please leave that space blank. If you cannot think of a CSO to list, please move on to the next question. 1 CSO name 2 City with main office 3 Contact name 4 E-mail 5 Phone number A ______________ _ PROG: OPEN￾ENDED _____________ __ PROG: OPEN￾ENDED 98 Do not know _____________ __ PROG: OPEN￾ENDED 98 Do not know ______________ _ PROG: OPEN￾ENDED 98 Do not know ______________ _ PROG: OPEN￾ENDED 98 Do not know B ______________ _ PROG: OPEN￾ENDED _____________ __ PROG: OPEN￾ENDED 98 Do not know _____________ __ PROG: OPEN￾ENDED 98 Do not know ______________ _ PROG: OPEN￾ENDED 98 Do not know ______________ _ PROG: OPEN￾ENDED 98 Do not know C ______________ _ _____________ __ PROG: OPEN- _____________ __ PROG: OPEN- ______________ _ PROG: OPEN- ______________ _ PROG: OPEN- 56 PROG: OPEN￾ENDED ENDED 98 Do not know ENDED 98 Do not know ENDED 98 Do not know ENDED 98 Do not know Q1.5. PROG: ASK IF Q1.2==1. In the following table, please provide the names of up to three CSOs that were active in the areas of national elections and/or political processes in ##P-country## but no longer exist. Please list the city where the CSO’s main office was located, and the reason why this CSO no longer exists if you know it. If you do not know of any CSOs that worked in this area and no longer exist, please leave this question blank. 1 CSO name 2 City where main office was located 3 Reason for demise A ___________________ __PROG: OPEN￾ENDED _______________ PROG: OPEN￾ENDED 98 Do not know 1 Lack of funding 2 Poor management 3 Merged or was absorbed by another organization 4 No demand for their services 5 Poor skills or training 6 Founder no longer with CSO 7 Shut down by government body 8 Other, please specify: _______ PROG: OPEN-ENDED 98 Do not know B ___________________ __PROG: OPEN￾ENDED _______________ PROG: OPEN￾ENDED 98 Do not know 1 No more funding 2 Poor management 3 Merged or was absorbed by another organization 4 No demand for their services 5 Poor skills or training 6 Founder no longer with CSO 7 Shut down by government body 8 Other, please specify: _______ PROG: OPEN-ENDED 98 Do not know C ___________________ __PROG: OPEN￾ENDED _______________ PROG: OPEN￾ENDED 98 Do not know 1 No more funding 2 Poor management 3 Merged or was absorbed by another organization 4 No demand for their services 5 Poor skills or training 6 Founder no longer with CSO 7 Shut down by government body 8 Other, please specify: _______ 57 PROG: OPEN-ENDED 98 Do not know Q1.6. Do you think there are more local civil-society organizations (CSOs) working in the areas of elections and political processes active today than there were in 2009? 1 Yes 2 No PROG: SKIP TO Q1.7b 98 Do not know PROG: SKIP TO Q2.1 Q1.7a. PROG: ASK IF Q1.6==1. PROG: SELECT UP TO TWO. In your opinion, why are there more local civil-society organizations active in election/political processes today than there were in 2009? Please select no more than two (2) of the most important reasons. 1M Greater availability of funding 2M More open political environment for civil society 3M More frequent elections 4M More competitive elections 5M Legislative changes allowing different activities 6M Greater capacity within civil society 96M Other, please specify: _______________ PROG: OPEN-ENDED Q1.7b. PROG: ASK IF Q1.6==2. PROG: SELECT UP TO TWO. In your opinion, why are there fewer local civil-society organizations active in election/political processes today than there were in 2009? Please select no more than two (2) of the most important reasons. 1M Lack of funding 2M Better electoral processes 3M Worse electoral processes 4M Legislative changes prohibiting activity 5M Intimidation of civil society 6M Closed operating environment for CSOs 7M More important activities 96M Other, please specify: __________ PROG: OPEN-ENDED PROG: START SECTION ON RESEARCH QUESTION 2 (CHANGES TO STRATEGIES) Q2.1. Were local CSOs included in any way in the ##P-country## project? 1 Yes 2 No PROG: SKIP TO Q4.2 98 Do not know PROG: SKIP TO Q4.2 Q2.2. PROG: ASK IF Q2.1==1. PROG: SELECT ALL THAT APPLY. To your knowledge, in what ways were local CSOs included in this project? Please select all that apply. 1M Grants to local CSOs 2M Subcontracts with local CSOs 58 3M Collaboration with local CSOs 4M Training and/or other capacity building provided to CSOs 96M Other, please specify: __________ PROG: OPEN-ENDED Q2.3. PROG: ASK IF Q2.2==1. PROG: SELECT ALL THAT APPLY. What was the purpose of giving grants to local CSOs? Please select all that apply. 1M To support their provision of desired services 2M To build CSO organizational capacity 3M To build CSO technical skills 96M Other, please specify: __________ PROG: OPEN-ENDED 98 Do not know Q2.4. PROG: ASK IF Q2.1==1. From USAID’s perspective, how important was the inclusion of local CSOs in this particular project? 1 Essential 2 Important 3 Not very important 4 Not necessary to include CSOs 98 Do not know Q2.5. PROG: ASK IF Q2.1==1. Now we would like to talk about the selection process behind selecting local partner organizations to work with CEPPS III and/or to receive CEPPS III benefits. Are you familiar with this selection process? 1 Yes 2 No PROG: SKIP TO Q2.7a Q2.6a1. PROG: ASK IF Q2.5==1. When selecting local CSOs to work with USAID, there can be different objectives, such as emphasizing a long term partnership or ensuring a transparent competitive process. In selecting local CSOs for the CEPPS III project in your country context, how important do you believe each of the following criteria should have been? 1.) Maintaining a consistent long-term partnership with specific CSOs in order to build capacity and sustainability of those organizations 4 Very important 3 Somewhat important 2 Not very important 1 Not important Q2.6a2. PROG: ASK IF Q2.5==1. 2.) Ensuring that there is a transparent and objective competitive process so that all CSOs have an equal chance 4 Very important 3 Somewhat important 2 Not very important 1 Not important 59 Q2.6b. PROG: ASK IF Q2.5==1. In the actual selection of local CSOs to participate in the CEPPS III program, how appropriate do you believe that the balance between the need for establishing stable continuous partnerships and ensuring an objective competitive selection processes was? 1 Optimal balance 2 Fairly good balance 3 Not very well balanced 4 Do not know Q2.6c. PROG: ASK IF Q2.5==1. How were CSOs selected? A competitive process was held CSOs were identified in the program description 3 CSOs were selected without competition but not listed in the program description 98 Do not know Q2.6d. PROG: ASK IF Q2.5==1. How important were each of the following criteria in selecting a local partner organization? For each of the criteria, please indicate how important it was. 1 Not Important 2 Not very important 3 Somewhat important 4 Very Important A. Quality of CSO Technical Skills o o o o B. Quality of CSO management o o o o C. Past performance o o o o D. Length of existence o o o o E. Past experience with CEPPS or USAID o o o o F. Name recognition o o o o G. Familiarity with main actors in the field o o o o H. Cost o o o o I. Other; please specify: __________ PROG: OPEN-ENDED o o o o Q2.6e. PROG: ASK IF Q2.5==1. To the best of your knowledge, were any of the CSOs selected active with CEPPS programming in the past? 1 Yes 2 No 98 Do not know Q2.7a. PROG: ASK IF Q2.1==1. PROG: ASK IF Q0.2a<2009. PROG: SELECT ALL THAT APPLY. Think back to the period 2005-2008. To the best of your knowledge, in what areas would you say that local CSOs most needed capacity building during that time? Please select all the topics which were needed. 60 Organizational Capacity: 1M Financial management 2M Organizational management 3M Fundraising 4M Database management 5M Internal communication 96M Other, please specify: ____________ PROG: OPEN-ENDED 98 Do not know Q2.7b. PROG: ASK IF Q2.1==1. PROG: ASK IF Q0.2a<2009. Technical Capacity: 1M Survey and research skills 2M Methodology 3M Parallel Vote Tabulation (PVT) 4M Applying contextual analysis to elections and political processes 5M Advocacy/oversight 6M Public information campaigns 96M Other, please specify: ____________ PROG: OPEN-ENDED 98 Do not know Q2.8. PROG: ASK IF Q2.1==1. PROG: ASK IF Q0.2a<2009. To your knowledge, did local CSOs in ##P-country## receive any capacity building from a CEPPS III implementer (IRI, NDI, and/or IFES) over the period 2005-2008? 1 Yes 2 No PROG: SKIP TO Q2.11a Q2.9. PROG: ASK IF Q2.8==1. PROG: SELECT ALL THAT APPLY. In what areas did CSOs receive capacity building in the period 2005-2008? Please select all that apply. 1M Organizational management 2M Technical skills 3 Do not know PROG: SKIP TO Q2.11a Q2.10a. PROG: ASK IF Q2.9==1. In what form was the organizational capacity building provided over the period 2005-2008? 1 Technical assistance through joint work on a grant or contract 2 Stand-alone Technical assistance 3 Training or workshop 97 Capacity building on this topic was not provided A. Financial management O o O O B. Organizational management O o O O C. Fundraising O o O O 61 D. Database management O o O O E. Internal communication o o o o Q2.10b. PROG: ASK IF Q2.9==2. In what form was the technical capacity building provided over the period 2005-2008? 1 Technical assistance through joint work on a grant or contract 2 Stand-alone Technical assistance 3 Training or workshop 97 Capacity building on this topic was not provided A. Survey and research skills O o O O B. Methodology O o O O C. Parallel votetabulation (PVT) O o O O D. Applying contextual analysis to elections and political processes O o O O E. Advocacy / oversight O o o O F. Public information campaigns O o o O Q2.11a. PROG: ASK IF Q2.1==1. PROG: SELECT ALL THAT APPLY. Now think about the more recent period, 2009 – Present. In what areas would you say local CSOs most needed capacity building during the period 2009-Present? Please select all the topics which were needed. Organizational Capacity: 1M Financial management 2M Organizational management 3M Fundraising 4M Database management 5M Internal communication 96M Other, please specify: ____________ PROG: OPEN-ENDED 98 Do not know Q2.11b. PROG: ASK IF Q2.1==1. Technical Capacity: 62 1M Survey and research skills 2M Methodology 3M Parallel Vote Tabulation (PVT) 4M Applying contextual analysis to elections and political processes 5M Advocacy/oversight 6M Public information campaigns 96M Other, please specify: ____________ PROG: OPEN-ENDED 98 Do not know Q2.12. PROG: ASK IF Q2.1==1. PROG: SELECT ALL THAT APPLY. To your knowledge, did local CSOs receive any capacity building from a CEPPS III implementer over the period 2009 - Present? 1 Yes 2 No PROG: SKIP TO Q3.1 Q2.13. PROG: ASK IF Q2.12=1. PROG: SELECT ALL THAT APPLY. In what areas did CSOs receive capacity building in the period 2009-Present? Please select all that apply. 1M Organizational management 2M Technical skills 3 Do not know PROG: SKIP TO Q2.15 Q2.14a. PROG: ASK IF Q2.13==1. In what form was the organizational capacity building provided over the period 2009-Present? Topic 1 Technical assistance through joint work on a grant or contract 2 Stand-alone Technical assistance 3 Training or workshop 97 Capacity building on this topic was not provided A. Financial management O o o o B. Organizational management O o o o C. Fundraising O o o o D. Database management O o o o E. Internal communication O o o o Q2.14b. PROG: ASK IF Q2.13==2. In what form was the technical capacity building provided over the period 2009 - Present? 1 Technical assistance through joint work on a 2 Stand-alone Technical assistance 3 Training or workshop 97 Capacity building on this topic was not provided 63 grant or contract A. Survey and research skills O o o o B. Methodology O o o o C. Parallel vote tabulation (PVT) O o o o D. Applying contextual analysis to elections and political processes O o o o E. Advocacy / oversight O o o o F. Public information campaigns O O o o Q2.15. PROG: ASK IF Q2.8==1 AND Q2.12==1. Overall would you say that the capacity building needs of CSOs were better addressed by CEPPS III implementer(s) in ##P-country## in the period 2005-2008, or in the period 2009-Present? 1 Better addressed prior to 2009 2 Addressed about the same in both periods 3 Better addressed during 2009-present 98 Do not know 99 No response PROG: START SECTION ON RESEARCH QUESTION 3 (INVOLVEMENT OF LOCAL PARTNERS) Q3.1. PROG: ASK IF Q2.1==1. At the outset of the program, what approach did you envision the implementer and the local CSOs using to determine what activities, among those programmed into the award design, local CSOs would work on with the CEPPS program in ##P-country##? The implementer would develop a clear idea of activities the local CSOs should undertake and they would comply The implementer would hold planning sessions with CSOs to determine what needed to be done and what activities the CSOs could best do The implementer would approach CSOs with options of what activities to undertake. CSOs would negotiate with the implementer to define activities CSOs would propose the activities Other, please specify: _________________ PROG: OPEN-ENDED Do not know Q3.2. PROG: ASK IF Q2.1==1. In actuality, what approach do you believe was used to determine what activities local CSOs would work on for the CEPPS program in ##P-country##? The implementer developed a clear idea of what the role of local CSOs should be and they complied 64 The implementer held planning sessions with CSOs to determine what needed to be done and what activities the CSOs could best do The implementer approached CSOs with options for activities to undertake CSOs negotiated with the Implementer to define activities CSOs proposed the activities Other, please specify: __________________ PROG: OPEN-ENDED Do not know Q3.3. PROG: ASK IF Q2.1==1. To the best of your knowledge, to what extent were local CSOs involved by CEPPS III implementer(s) in developing the work plan of the ##P-country## program? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know 99 No response Q3.4. PROG: ASK IF Q2.1==1. To the best of your knowledge, to what extent did the CEPPS III implementer(s) involve local CSO(s) in the reporting requirements for the ##P-country## project? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know 99 No response Q3.5. PROG: ASK IF Q2.1==1. Over the period from 2010 through 2015, do you feel that the responsibilities of local CSOs expanded within the collaboration they had with the CEPPS III implementer(s) in ##P-country##? 4 Yes, to a significant extent 3 Yes, to some extent 2 No, the responsibilities remained about the same 1 No, the responsibilities were reduced 98 Do not know 99 No response PROG: START SECTION ON EVALUATION RESEARCH QUESTION 4 (FACTORS AFFECTING LOCAL ORGANIZATION CAPACITY) Q4.1. PROG: ASK IF Q2.1==1. Thinking of the local CSOs that CEPPS has worked with, have you seen a change in local CSO capacity in each of the following areas in the period 2009 - Present? 1 Have seen improvement 2 No change 3 Have gotten weaker 98 Do not know A. Financial management O o o o 65 B. Organizational management O o o o C. Fundraising O o o o D. Communications / public relations O o o o E. Monitoring & evaluation (M&E) O o o o Q4.2. PROG: SELECT UP TO THREE. Please identify up to three (3) principal obstacles that you believe are an impediment to the sustainability of local CSOs in the coming two years. 1M Lack of endowment or financial reserves to maintain core staff between grant and contract awards 2M Lack of demand for services 3M Lack of consistency in demand for services 4M Lack of funding sources 5M Lack of sufficient knowledge of what international donors need 6M The under-developed state of administrative systems relative to what international donors require 7M Lack of subject-matter expertise 8M Intimidation from governmental or political influences. 9M Difficulty in mobilizing a sufficient number of temporary additional personnel 10M Difficulties with service quality control 11M Lack of knowledge/experience in fundraising 12M Other, please specify: ________________________ PROG: OPEN-ENDED PROG: START SECTION ON RESEARCH QUESTION 5 (FACTORS FOR LOCAL ORGANIZATION CAPACITY) Q5.1. Has your USAID Mission directly funded any local organizations? 1 Yes 2 No PROG: SKIP TO Q5.4 Q5.2. PROG: ASK IF Q5.1==1. Have any of those local organizations performed services related to elections and/or political processes? 1 Yes 2 No PROG: SKIP TO Q5.4 Q5.3. PROG: ASK IF Q5.2==1. Please list up to three of those CSOs. Also indicate which of them, if any, were at some point associated with the CEPPS III program. Name of CSO Email Telephone Associated with CEPPS at some time? A ___________ ___________ ___________ 1 Yes 2 No 66 PROG: OPEN￾ENDED PROG: OPEN￾ENDED PROG: OPEN￾ENDED B ___________ PROG: OPEN￾ENDED ___________ PROG: OPEN￾ENDED ___________ PROG: OPEN￾ENDED 1 Yes 2 No C ___________ PROG: OPEN￾ENDED ___________ PROG: OPEN￾ENDED ___________ PROG: OPEN￾ENDED 1 Yes 2 No Q5.4. How many local CSOs in ##P-country## would you estimate have the capacity to handle USAID contracts or grants? 0 None 1 One or two 2 3-10 3 More than ten 98 Do not know Q5.5. PROG: SELECT ALL THAT APPLY. To the best of your knowledge, which areas do you believe would create the biggest challenges for local CSOs wishing to manage USAID contracts or grants? Please select up to three (3). 1M Technical knowledge 2M Lack of sufficient knowledge on how to prepare proposals 3M Inadequate administrative systems 4M Inadequate financial controls 5M Uneven service quality control 6M Lack of experience 96M Other, please specify: _______________ PROG: OPEN-ENDED Q5.6. Would you be willing to be contacted to talk on the telephone to answer additional questions? 1 Yes 2 No PROG: SKIP TO CLOSURE Q5.7. PROG: ASK IF Q5.6==1. Please provide your phone number. ____________________ PROG: TELEPHONE NUMBER PROG: DISPLAY CLOSURE MESSAGE: Thank you for completing our survey! PROG: END MESSAGE 67 USAID Evaluation of the Consortium for Elections and Political Process Strengthening (CEPPS) FINAL Questionnaire for Implementers INTRO. PROG: PLEASE DISPLAY THE FOLLOWING MESSAGE ON ITS OWN PAGE: NORC, a research center associated with the University of Chicago, is conducting an independent evaluation of the Consortium for Elections and Political Process Strengthening (CEPPS) program, a project funded by the United States Agency for International Development (USAID) and implemented by IRI, NDI, and IFES. As part of this evaluation, we are collecting information from program staff, implementers, and beneficiaries to help us better understand the CEPPS’s program activities and how the program can be improved to better meet the needs of its intended audience. Your participation in this survey is important because it is one of the few ways available for getting truly representative opinions. Your responses will help us to evaluate the effectiveness of the CEPPS, and how it can be improved to better serve its intended beneficiaries. This survey is designed to ask you questions about the CEPPS III programs you have helped implement with ##P-organization##. If you have worked in more than one country on CEPPS III programs, you will have the opportunity to respond on each of those countries separately. If you have worked on multiple CEPPS projects within a single country, you will be asked to respond for the most recent project. The survey should take no more than 20-30 minutes to complete. Your participation in this survey is voluntary. If you are unable to answer a question, please leave it blank. All of your responses will be kept strictly confidential, and will only be used in a statistical summary. If you have any questions about the survey, please e-mail Katharine Mark, Principal Research Scientist and Survey Manager, at USAIDpartnersurvey@norc.org so that we may assist you. By clicking the ‘Begin Survey’ button at the bottom right of the screen, you acknowledge that your participation in this survey is completely voluntary. PROG: INSERT NEXT, PREVIOUS, STOP BUTTONS ON THIS AND SUBSEQUENT PAGES PROG: END MESSAGE 68 PROG: START SECTION ON RESPONDENT SORTING/FILTERING Q0.0. In how many countries have you helped implement CEPPS III projects for ##P-organization##? Please provide the number of countries in which you have helped implement CEPPS III projects with ##P-organization## rather than the number of CEPPS III projects you have worked on. If you have never had a role in implementing any CEPPS III projects with ##P-organization##, please enter a value of “0”. _______ PROG: INTEGER PROG: START PROJECT LOOP, TO BE COMPLETED FOR X ITERATIONS, WHERE X IS EQUAL TO THE VALUE OF THE RESPONSE IN Q0.0. Q0.1. PROG: DROPDOWN BOX. Please select the CEPPS III country/program you worked on [most recently, prior to the one just described]. 1 African Union 2 Angola 3 Armenia 4 Azerbaijan 5 Bangladesh 6 Belarus 7 Bhutan 8 Bosnia & Herzegovina 9 Burkina Faso 10 Burma 11 Burundi 12 Cambodia 13 Central African Republic 14 Colombia 15 Côte d’Ivoire 16 Ecuador 17 Egypt 18 Fiji 19 Georgia 20 Guatemala 21 Guinea 22 Guyana 23 Haiti 24 Honduras 25 Indonesia 26 Iraq 27 Jordan 28 Kazakhstan 69 29 Kenya 30 Kosovo 31 Kyrgyzstan 32 Lebanon 33 Liberia 34 Libya 35 Macedonia 36 Malawi 37 Maldives 38 Mali 39 Mauritania 40 Mexico 41 Morocco 42 Nepal 43 Nicaragua 44 Niger 45 Nigeria 46 Pakistan 47 Papua New Guinea 48 REAPPS (Eurasia Regional) 49 Serbia 50 Sierra Leone 51 Somalia 52 Sri Lanka 53 Sudan or South Sudan 54 Syria 55 Tajikistan 56 Togo 57 Tunisia 58 Uganda 59 Ukraine 60 Uzbekistan 61 Yemen 62 Zambia 63 Zimbabwe Q0.2. What is (or was) your role for the CEPPS III project in ##Q0_1##? If your role changed over the course of the project, please select the role with the most responsibility. If you were involved in more than one CEPPS III project in ##Q0_1##, please answer this and subsequent questions for the most recent project. 1 COP 2 Deputy COP 3 Project Director 4 Project back-stop 70 96 Other, please specify: __________________ PROG: OPEN-ENDED Q0.3a. In which year did your involvement with this project begin? _______________ PROG: INTEGER, RANGE 1995-2015 Q0.3b. In which year did your involvement with this project end? 1 _______________ PROG: INTEGER, RANGE 2009-2015 97 Still ongoing PROG: START SECTION 1: NUMBER OF ACTIVE CSOs Q1.0. According to USAID’s CSO sustainability index: “For a CSO sector to be sustainable, the legal and regulatory environment should support the needs of CSOs. It should facilitate new entrants, help prevent governmental interference, and give CSOs the necessary legal basis to engage in appropriate fundraising activities and legitimate income-producing ventures.” With this in mind, how supportive or hostile to CSOs do you believe the legal and regulatory environment is in ##Q0_1##? 1 Very supportive 2 Somewhat supportive 3 Somewhat hostile 4 Very hostile 98 Do not know Q1.1. Do you know of any local CSOs that currently conduct (or conducted in the past) activities in election processes (such as election observation, voter education, or electoral law/regulatory reform) or political processes (such as training of political party leadership, monitoring political parties, or conducting advocacy) in ##Q0_1##? 1 Yes 2 No PROG: SKIP TO Q2.1a 98 Do not know PROG: SKIP TO Q2.1a Q1.2. PROG: ASK IF Q1.1==1. PROG: SELECT ALL THAT APPLY. Please indicate the activities in which such CSOs in ##Q0_1## are active. Please select all that apply. 1M Training of political party activists 2M Conducting election observation 3M Conducting advocacy campaigns 4M Media training 5M Public opinion polling and/or focus group discussions 6M Parallel vote tabulation (PVT) 7M Training of election management bodies 71 8M Political dialogue and/or debates 9M Voter education campaigns 10M Advocacy on elections law, political party law, and/or CSO law 96M Other, please specify: _____________ PROG: OPEN-ENDED Q1.3. PROG: ASK IF Q1.1==1. If you can, please provide the names and emails of up to three of the primary CSOs you are aware of that provide technical assistance in the areas of elections and/or political processes in ##Q0_1##. If you cannot think of any of the below information, please leave that space blank. If you cannot think of a CSO to list, please move on to the next question. A B CSO name E-mail 1 _________________ __ PROG: OPEN￾ENDED _______________ PROG: OPEN￾ENDED 98 Do not know 2 _________________ _ PROG: OPEN￾ENDED _______________ PROG: OPEN￾ENDED 98 Do not know 3 _________________ _ PROG: OPEN￾ENDED _______________ PROG: OPEN￾ENDED 98 Do not know Q1.4a. PROG: ASK IF Q1.1==1. Please provide the names of up to three CSOs that were active in the areas of national elections and/or political processes in ##Q0_1## but no longer exist. If you do not know of any CSOs that worked in this area and no longer exist, please move on to the next question. CSO Name 1: _____________ PROG: OPEN-ENDED CSO Name 2: _____________ PROG: OPEN-ENDED CSO Name 3: _____________ PROG: OPEN-ENDED Q1.4b. Please select the reason(s) for the demise of the CSOs above that were active in the areas of national elections and/or political processes in ##Q0_1## but no longer exist. Please select all that apply. If you do not know of any such CSOs, please leave this question blank. 72 1M Lack of funding 2M Poor management 3M Merged or was absorbed by another organization 4M No demand for their services 5M Poor skills or training 6M Founder no longer with CSO 7M Shut down by government body 8M Other, please specify: ______________ PROG: OPEN-ENDED 98 Do not know Q1.5. PROG: ASK IF Q1.1==1. Do you think there are more CSOs in ##Q0_1## active today than there were in the start of 2009 working in the areas of elections and political processes? 1 Yes 2 No PROG: SKIP TO Q1.6b 98 Do not know PROG: SKIP TO Q2.1a Q1.6a. PROG: ASK IF Q1.5==1. PROG: SELECT UP TO TWO. In your opinion, why are there more local CSOs active in election/political processes today than there in 2009? Please select no more than two (2) of the most important reasons. 1M Greater availability of funding 2M More open political environment for civil society 3M More frequent elections 4M More competitive elections 5M Legislative changes allowing different activities 6M Greater capacity within civil society 96M Other, please specify: _______________ PROG: OPEN-ENDED Q1.6b. PROG: ASK IF Q1.5==2. PROG: SELECT UP TO TWO. In your opinion, why are there fewer local civil-society organizations active in election/political processes today than there were in 2009? Please select no more than two (2) of the most important reasons. 1M Lack of funding 2M Better electoral processes 3M Worse electoral processes 4M Legislative changes prohibiting activity 5M Intimidation of civil society 6M Closed operating environment for CSOs 7M More important activities 96M Other, please specify: __________ PROG: OPEN-ENDED PROG: START SECTION 2: CHANGES TO STRATEGIES Q2.1a. PROG: SELECT ALL THAT APPLY. In what ways were local CSOs involved in the CEPPS III program in ##Q0_1##? Please select all that apply. 73 1M Grants for CSOs 2M Subcontracts for CSOs 3M Collaboration with CSOs 4M Training for CSOs 96M Other, please specify: ________________ PROG: OPEN-ENDED 97 No CSO involvement PROG: SKIP TO Q2.7 Q2.1b. PROG: ASK IF Q2.1a==1. PROG: SELECT ALL THAT APPLY. What was the purpose of giving grants to local CSOs? Please select all that apply. 1M To support their provision of desired services 2M To build CSO organizational capacity 3M To build CSO technical skills 96M Other, please specify: __________ PROG: OPEN-ENDED 98 Do not know Q2.1c. PROG: ASK IF Q2.1a!=97. Were the CSOs on the CEPPS III in ##Q0_1## selected via some sort of selection process? 1 Yes, there was a selection process 2 No, partner CSOs were pre-selected by USAID PROG: SKIP TO Q2.2a Q2.1c1. PROG: ASK IF Q2.1c==1. When selecting local CSOs, there can be different objectives. In selecting local CSOs for the CEPPS project in ##Q0_1##, how important do you believe each of the following criteria were? 1.) Maintaining a consistent long-term partnership with specific CSOs in order to build capacity and sustainability of those organizations. 4 Very important 3 Somewhat important 2 Not very important 1 Not important Q2.1c2. PROG: ASK IF Q2.1c==1. 2.) Ensuring that there is a transparent and objective competitive process so that all CSOs have an equal chance. 4 Very important 3 Somewhat important 2 Not very important 1 Not important Q2.1d. PROG: ASK IF Q2.1c==1. In the actual selection of local CSOs to participate in the CEPPS program in ##Q0_1##, what was the balance between the two above objectives? 1 Leaned towards establishing ongoing relationships 2 Leaned towards competitive selection process 74 3 Optimal balance 98 Do not know Q2.2a. PROG: ASK IF Q2.1a≠97. PROG: SELECT ALL THAT APPLY. In what areas would you say that local CSOs in ##Q0_1## most needed capacity building at the start of this CEPP III program? Please select all the topics which were needed. Organizational Capacity: 1M Financial management 2M Organizational management 3M Fundraising 4M Database management 5M Internal communication 96M Other, please specify: ____________ PROG: OPEN-ENDED 97 No organizational capacity building needed Q2.2b. PROG: ASK IF Q2.1a≠97. Technical Capacity: 1M Survey and research skills 2M Methodology 3M Parallel vote tabulation (PVT) 4M Applying contextual analysis to elections and political processes 5M Advocacy/oversight 6M Public information campaigns 96M Other, please specify: ____________ PROG: OPEN-ENDED 97 No technical capacity building needed Q2.3a. PROG: ASK IF Q2.1a≠97. Was building the capacity of local CSOs an explicit part of this program? Yes No Q2.3b. PROG: ASK IF Q2.1a≠97. How much of this program involved building capacity of local CSOs? 1 A large part of program activities 2 A substantial part of program activities 3 A small part of program activities 97 No capacity building of CSOs PROG: SKIP TO Q2.7 98 Do not know PROG: SKIP TO Q2.7 75 Q2.4. PROG: ASK IF Q2.3b≠97 or 98. PROG: SELECT ALL THAT APPLY. In what areas did CSOs in ##Q0_1## receive capacity building under the CEPPS III program? Please select all that apply. 1M Organizational management 2M Technical skills 98 Do not know PROG: SKIP TO Q2.7 Q2.5a. PROG: ASK IF Q2.4==1. In what form was the organizational capacity building provided to local CSOs in ##Q0_1## under CEPPS III? 1 Technical assistance through joint work on a grant or contract 2 Stand-alone technical assistance 3 Training or workshop 97 Capacity building on this topic was not provided A. Financial management O o O O B. Organizational management O o O O C. Fundraising O o O O D. Database management O o O O E. Internal communication O o O O Q2.5b. PROG: ASK IF Q2.4==2. In what form was the technical capacity building provided to local CSOs in ##Q0_1## under CEPPS III? 1 Technical assistance through joint work on a grant or contract 2 Stand-alone technical assistance 3 Training or workshop 97 Capacity building on this topic was not provided A. Survey and research skills O o O O B. Methodology O o O O C. Parallel vote tabulation (PVT) O o O O D. Applying contextual analysis to elections and political processes O o O O 76 E. Advocacy / oversight O o O O F. Public information campaigns O o O O Q2.6a. PROG: ASK IF Q2.3b≠97. Did the CEPPS program develop and implement any capacity￾building activities beyond those requested/recommended by USAID? 1 Yes 2 No PROG: SKIP TO Q2.7 98 Do not know PROG: SKIP TO Q2.7 Q2.6b. PROG: ASK IF Q2.6a==1. Please describe the capacity-building activities ##P￾organization## developed. ______________________________________ PROG: OPEN-ENDED FIELD 99 No response Q2.6c. PROG: ASK IF Q2.6a==1. What was the rationale for their adoption? ______________________________________ PROG: OPEN-ENDED FIELD 99 No response Q2.7. Were you involved in any CEPPS II programs in ##Q0_1##? Please note CEPPS II projects were generally implemented during the period 2003-2008. 1 Yes 2 No PROG: SKIP TO Q3.1 Q2.8a. PROG: ASK IF Q2.7==1. PROG: SELECT ALL THAT APPLY. In what ways were local CSOs involved in the CEPPS II program in ##Q0_1##? Please select all that apply. 1M Grants for CSOs 2M Subcontracts for CSOs 3M Collaboration with CSOs 4M Training for CSOs 96M Other, please specify: ________________ PROG: OPEN-ENDED 97 No CSO involvement PROG: SKIP TO Q3.1 Q2.8b. PROG: ASK IF Q2.8a==1. PROG: SELECT ALL THAT APPLY. What was the purpose of giving grants to local CSOs during the CEPPS II project in ##Q0_1##? Please select all that apply. 1M To support their provision of desired services 2M To build CSO organizational capacity 3M To build CSO technical skills 96M Other, please specify: __________ PROG: OPEN-ENDED 77 98 Do not know Q2.9a. PROG: ASK IF Q2.8a≠97. PROG: SELECT ALL THAT APPLY. Now think about the program you worked on under CEPPS II, during the period 2003-2008. In what areas would you say local CSOs in ##Q0_1## most needed capacity building during the period 2003-2008? Please select all the topics which were needed. Organizational Capacity: 1M Financial management 2M Organizational management 3M Fundraising 4M Database management 5M Internal communication 96M Other, please specify: ____________ PROG: OPEN-ENDED 97 No organizational capacity building needed Q2.9b. PROG: ASK IF Q2.8a≠97. Technical Capacity: 1M Survey and research skills 2M Methodology 3M Parallel Vote Tabulation (PVT) 4M Applying contextual analysis to elections and political processes 5M Advocacy/oversight 6M Public information campaigns 96M Other, please specify: ____________ PROG: OPEN-ENDED 97 No technical capacity building needed Q2.10a. PROG: ASK IF Q2.8a≠97. Was building the capacity of local CSOs an explicit part of the CEPPS II program in ##Q0_1##? 1 Yes 2 No Q2.10b. PROG: ASK IF Q2.8a≠97. How much of the CEPPS II program involved building capacity of local CSOs? A large part of program activities A substantial part of program activities A small part of program activities No capacity building PROG: SKIP TO Q3.1 Do not know PROG: SKIP TO Q3.1 Q2.11. PROG: ASK IF Q2.10b≠97 or 98. PROG: SELECT ALL THAT APPLY. In what areas did CSOs in ##Q0_1## receive capacity building under CEPPS II? Please select all that apply. 1M Organizational management 2M Technical skills 98 Do not know PROG: SKIP TO Q2.13a 78 Q2.12a. PROG: ASK IF Q2.11==1. In what form was the organizational capacity building provided to local CSOs in ##Q0_1## under CEPPS II? Topic 1 Technical assistance through joint work on a grant or contract 2 Stand-alone technical assistance 3 Training or workshop 97 Capacity building on this topic was not provided A. Financial management O o O o B. Organizational management O o O o C. Fundraising O o O o D. Database management O o O o E. Internal communication O o O o Q2.12b. PROG: ASK IF Q2.11==2. In what form was the technical capacity building provided to local CSOs in ##Q0_1## under CEPPS II? 1 Technical assistance through joint work on a grant or contract 2 Stand-alone technical assistance 3 Training or workshop 97 Capacity building on this topic was not provided A. Survey and research skills o o O o B. Methodology o o O o C. Parallel vote tabulation (PVT) o o O o D. Applying contextual analysis to elections and political processes o o O o E. Advocacy / oversight o o O o F. Public information campaigns o o O o 79 Q2.13a. PROG: ASK IF Q2.3b≠97 or 98 AND Q2.10b≠97 or 98. In your opinion, was there a change in the programmatic strategies for building local CSO capacity in ##Q0_1## from CEPPS II to CEPPS III? 1 Yes 2 No PROG: SKIP TO Q2.14 98 Do not know PROG: SKIP TO Q2.14 Q2.13b. PROG: ASK IF Q2.13a==1. How have these strategies changed? Please be as specific as possible, mentioning specific strategies if possible. _______________ PROG: OPEN-ENDED FIELD 99 No response Q2.14. PROG: ASK IF Q2.3b≠97 AND Q2.10b≠97. Overall, would you say that the capacity building needs of CSOs in ##Q0_1## were better addressed in this country under CEPPS II or under CEPPS III? 1 Better addressed under CEPPS II 2 Addressed about the same in both periods 3 Better addressed under CEPPS III 98 Do not know 99 No response PROG: START SECTION ON RESEARCH QUESTION 3 (INVOLVEMENT OF LOCAL PARTNERS) Q3.1. PROG: ASK IF Q2.1a==1. Thinking again about the most recent CEPPS III program in ##Q0_1##, to the best of your knowledge, to what extent were local CSOs involved by ##P￾organization## in developing the work plan of the CEPPS III program? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know 99 No response Q3.2. PROG: ASK IF Q2.1a==1. To the best of your knowledge, to what extent did ##P￾organization## involve local CSO(s) in the reporting requirements for the CEPPS III program in ##Q0_1##? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 80 98 Do not know 99 No response Q3.3. PROG: ASK IF Q2.1a==1. To what extent did ##P-organization## involve local partners in ##Q0_1## in the design of programs with which it was affiliated? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know 99 No response Q3.4. PROG: ASK IF Q2.1a==1. To what extent did ##P-organization## involve local partners in ##Q0_1## in the awards management of programs with which it was affiliated? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know 99 No response Q3.5. PROG: ASK IF Q2.1a==1. To what extent did ##P-organization## involve local partners in ##Q0_1## in the M&E processes of programs with which it was affiliated? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all 98 Do not know 99 No response Q3.6. Did ##P-organization## integrate organizational capacity-building activities into your collaborations on elections and political processes, or did the activities occur on a separate parallel track? 1 Integrated capacity building 2 Parallel track 98 Do not know Q3.7. To what extent have local and regional partners in ##Q0_1## incorporated gender analysis – for their donors as well as for non-donor work? 4 To a great extent 3 To some extent 2 To a small extent 81 1 Not at all 98 Do not know 99 No response Q3.8a. PROG: ASK IF Q2.1a==1. Over the period from 2010 through 2015, do you feel that the responsibilities of local CSOs expanded within the collaboration they had with ##P-organization## in ##Q0_1##? 4 Yes, to a significant extent 3 Yes, to some extent 2 No, their responsibilities remained about the same 1 No, their responsibilities were reduced 98 Do not know 99 No response Q3.8b. PROG: ASK IF Q2.1a==1. What approach do you believe was used to determine what activities local CSOs would work on with the CEPPS program in ##Q0_1##? Please select the sentence that is closest to ##P-organization##’s experience. ##P-organization## developed a clear idea of what the role of local CSOs should be and they complied ##P-organization## held planning sessions with CSOs to determine what needed to be done and what activities the CSOs could best do ##P-organization## approached CSOs with options for activities to undertake CSOs negotiated with ##P-organization## to define activities CSOs proposed the activities Other, please specify: __________________ PROG: OPEN-ENDED Q3.9. What best practices have local and regional partners in ##Q0_1## adopted subsequent to your starting to work with them? Please be as specific as possible. 1 _______________________________ PROG: OPEN-ENDED FIELD 97 Not applicable 98 Do not know PROG: START SECTION 4: FACTORS FOR LOCAL ORGANIZATION CAPACITY Q4.1. PROG: SELECT UP TO 3. The ability to survive and flourish in the long term is clearly an important objective for any CSO. From the following factors – or those you add – please identify up to three (3) principal obstacles that you believe threaten the sustainability of local CSOs working in electoral and political process in ##Q0_1##. 82 Please select up to 3 Lack of endowment or financial reserves to maintain core staff between grant and contract awards |__| Little interest from citizens in activities |__| Lack of specialized staff and/or staff turnover |__| Lack of funding sources |__| Lack of sufficient knowledge of what international clients need |__| Lack of sufficient knowledge on how to obtain funding from international clients |__| Failure of international donors to adapt to local and cultural contexts |__| Administrative systems do not meet requirements of international clients. |__| Lack of subject-matter expertise |__| Intimidation from governmental or political influences. |__| Difficulty in mobilizing a sufficient number of temporary additional personnel |__| Difficulties with our service quality control |__| Lack of consistency from international donors of services required over time |__| Lack of adequate funding from international donors |__| Lack of appropriate capacity building from international donors |__| Lack of knowledge/experience in fundraising |__| Lack of command of the languages of key donors to submit proposals |__| Political and/or economic instability |__| Other/Specify: ______________________________________ PROG: OPEN-ENDED |__| Other/Specify: ______________________________________ PROG: OPEN-ENDED |__| 83 Q4.2. PROG: SELECT UP TO 3. The results of the work of the CSO may not depend solely on the organization itself. What would you identify as the key factors in the CSOs’ operating environment increasing the likelihood that outcomes are successful? Please identify from among the following factors – or those you add – up to three (3) that you believe are the most critical to CSOs in ##Q0_1## for ensuring successful outcomes. PROG: START SECTION 5: EFFORTS FOR LOCAL ORGANIZATION JOINT FUNDING Q5.1. PROG: ASK IF Q2.1≠97. Was helping local CSOs in ##Q0_1## to become direct recipients of donor funding an objective for your program? 1 Yes, it was an explicit objective 2 It was not explicit, but it was one of the project’s objectives 3 It was not a major objective, but it was viewed as a positive parallel benefit 4 No, it was not an objective at all Q5.2a. PROG: ASK IF Q2.1≠97. To what extent would you say your program helped local CSOs in ##Q0_1## to become direct recipients of donor funding? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all Factor Please select up to 3 Collaborations with other local CSOs |__| Absence of governmental or political intimidation |__| Strong internal administrative systems |__| Expertise of senior staff and supervisors |__| Adequate funding from international funders |__| Capacity building assistance, mentoring, and advice from international organizations |__| Access to skilled staff and volunteers |__| Other, please specify: _______________________________________ PROG: OPEN-ENDED |__| Other, please specify: ________________________________________ PROG: OPEN-ENDED |__| 84 Q5.2b. PROG: ASK IF Q2.1≠97. To what extent would you say your program helped local CSOs in ##Q0_1## to become capable of applying for and receiving larger and more complicated awards? 4 To a great extent 3 To some extent 2 To a small extent 1 Not at all Q5.3. PROG: ASK IF Q2.1≠97. PROG: SELECT ALL THAT APPLY. Which of the following actions did your program take? Please select all that apply. 1M Introduce local CSOs to potential funders (via email, telephone, or in person) 2M Help CSOs prepare customized materials to give potential donors 3M Work with the CSO to identify activities they could provide that would be of interest to donors 4 None of the above Q5.4. PROG: ASK IF Q2.1≠97. What other efforts has ##P-organization## made to increase sustainability of local partners in ##Q0_1##? 1 _________________________________________ PROG: OPEN-ENDED 97 Not applicable 98 Do not know Q5.5. PROG: ASK IF Q2.1≠97.How many of the local CSOs your program worked with do you think will still be active five years from now? 1 Most of them 2 Some of them 3 A few of them 4 One of them 5 None of them Q5.6a. PROG: ASK IF Q2.1≠97. How many of the local CSOs your program worked with do you think are now capable of becoming direct recipients of donor funding? 1 Most of them 2 Some of them 3 A few of them 4 One of them 5 None of them Q5.6b. How many local CSOs (of any type) in this country would you estimate have the capacity to handle USAID contracts or grants? 85 0 None 1 One or two 2 3-10 3 More than 10 98 Do not know Q5.7. PROG: SELECT ALL THAT APPLY. To the best of your knowledge, which areas do you believe would create the biggest challenges for local CSOs in ##Q0_1## wishing to manage USAID contracts or grants? Please select up to three (3) responses. 1M Lack of sufficient technical knowledge 2M Lack of sufficient knowledge on how to prepare proposals 3M Inadequate administrative systems 4M Inadequate financial controls 5M Uneven service quality control 6M Lack of experience 96M Other, please specify: _______________ PROG: OPEN-ENDED PROG: RESTART PROJECT LOOP AT Q0.1 OR, IF LAST ITERATION, CONTINUE TO SURVEY CLOSURE Q5.8. Would you be willing to be contacted via telephone to answer additional questions? 1 Yes 2 No PROG: SKIP TO CLOSURE Q5.9. PROG: ASK IF Q5.8==1. Please provide your phone number. ____________________ PROG: TELEPHONE NUMBER PROG: DISPLAY CLOSURE MESSAGE: Thank you for participating in our survey! PROG: END MESSAGE 86 87 ANNEX VI: MATRIX OF RESEARCH QUESTIONS AND DATA SOURCES Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other Create CSO sampling frame Names of international NGOs providing C3SA services(b) Names of data￾bases(a) X X X X X Web search Names of multilateral, bilateral, and international-NGO donors funding C3SA services Names of local user organizations of CSO C3SA services Generic type of organizations: EMBs, political parties, candidates, media, other NGOs. Ditto X X X X X Name of national NGO registry X Govt. sources CSO names Taxonomy of sub￾activities by C3SA(b) X X X X X Names generated from above rows Local Capacity Building Question 1.1.A: Are there more CSOs active in each C3SA now than five years ago? Question 1.1.B: Has the life expectancy of CSOs increased compared to what it was five years ago? Outcomes: 88 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other 1.1.A: Number of CSOs 2009 and 2014, by C3SA Definition of an “active” CSO, possibly by C3SA NORC sampling frame (above) 1.1.B: CSO age, by C3SA: - in 2009, or whether non-existent - in 2014, or whether defunct X X National NGO registries Covariates CEPPS value of each award and local CSO sub￾grant, by C3SA X X Number of CSOs CEPPS￾II awardees, by C3SA X X Number of CSOs CEPPS￾II non-awardee participants, by C3SA X X X Real GDP or PPP per￾capita, by country World Tables (Penn, World Bank) Degree-free or Democracy Indicator by country Other cross-coun￾try datasets of elec￾toral and political process quality2 Heritage Foundation; Freedom House; 2 Examples include: quality Freedom House “Freedom in the World”; Cato Institute “Index of Freedom in the World”; Economist Intelligence Unit (from the Economist) Democracy Index (specifically includes political participation/political culture); Political Instability Task Force; Polity Data Series-very relevant as it measures electoral competition, political participation, etc; current series only through 2013) 89 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other Transparency International Whether CSO is partnered with a non￾CEPPS3 international NGO X X Funding source distribution (pct. CEPPS, international NGOs, donors, domestic, ser￾vices) X X Region dummies (Africa, Post-communist, Asia/n.e.s., Americas) NORC Dummies for country’s participation in previous CEPPS programs X X X Key country covariates (e.g., percent government revenues from extraction industries Propose others? IFS Yearbook of Gov￾ernment Sta￾tistics Type of capacity building received over period, by C3SA Taxonomy of capacity building, by From CEPPS C3SA From other donors X X X X X Per capita USAID budget in country X USG 90 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other Question 1.2: Does a greater share of CSOs interact with CEPPS/their local partners in 2014 compared to five years back? Outcomes: Pct. of CSOs that interacted with CEPPS or partners 2009 and 2014, by C3SA NORC sampling frame (above) Covariates CEPPS value of each award and local CSO sub￾grant, by C3SA X (See above) Real GDP or PPP per￾capita, by country (See above) Degree-free or Democracy Indicator, by country (See above) Whether CSO is partnered with another International NGO (See above) Funding source distribution (pct. CEPPS, international NGOs, donors, domestic, ser￾vices) (See above) 91 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other Degree of deepening of CEPPS from Stage II to Stage III Question 2.1: How were programmatic strategies developed under CEPPS altered or expanded for Objective 10? [We can compare distributions] Capacity-building strategies, by CEPPS phase, implementer, and country According to CEPPS Taxonomy of capacity-building strategies, by C3SA X X X According to CSO X Programmatic strategy, by award and CEPPS phase Can we code each award? X X X Sub-award characteristics (value, duration, repeat award, etc.) , by sub￾award and CEPPS phase Do these make sense? Are there others? X Question 2.2: Was CEPPS’s process for selecting local partner organizations optimal in the sense that competitive selection processes were balanced against the need for establishing ongoing relationships? [We can infer tradeoff and ask USAID the degree to which they agree with it] Method of selection according to USAID CEPPS Awarded CSOs, by sub￾award Non-awarded CSOs who bid, by sub-award Other CSOs Taxonomy of award selection X X X X X X X X CSOs perception of whether process was transparent and fair, by sub-award Suggestions to concretize this X X 92 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other USAID preferences on the tradeoff, by C3SA Infer revealed pre￾ference from award choices X Local CEPPS office documents justifying CSO choice CSO bidder characteristics (age, size, past experience, geographic location, etc.) Past experience with CEPPS partners, CEPPS subgrants, others. Could also disaggregate by mission (youth, women, etc) This information will likely be on the score sheets Country characteristics (See Section I covariates) Question 2.3: To what degree did receiving a CEPPS grant pre 2009 change the probability of receiving an award in 2009-2014, all else held equal? Type of sub-award Taxonomy of sub￾award purpose & activity X X X Dummy if CSO operated in 2009 Dummy if CSO operated in 2014 X X (See above) (See above) Did CSO receive sub￾award in 2009 Did CSO receive sub￾award in 2014 X X X X Did CSO participate in 2009 CEPPS activity Did CSO participated in 2014 CEPPS activity X X X X X X 93 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other The nature of CEPPS3 partnerships Q.3.1: To what degree did CEPPS involve local partners in design, awards, management etc.? By award, CEPPS assessment score of SAB involvement in award’s:(e) Implementation design Award management M&E processes Awards reporting X X By award, SAB assessment score of involvement in award’s: Implementation design Award management M&E processes Awards reporting X Country, award, CSO covariates Other ideas? (See above and Section V) Q.3.2: Did CEPPS incorporate CSO organizational capacity building as an integral part of collaboration or undertake it as separate task? Type of capacity building desirable (whether explicit or not) , by C3SA (Use taxonomy from above) X Implementer scopes of work Types of capacity building provided to SAB according to CEPPS, by award X X Types of capacity building provided to sub-awardee according to awardee, by award X 94 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other Characteristics of award activities, country, awardee X X (See above) Q.3.3: To what extent were best practices adopted by local partners? Types of best practices List by C3SA X X X KIIs of international NGOs implementatio n literature Extent best practices were requested by award, sub-award, and/or via training Can we code each award and sub￾award? X X X Local implementer perception of whether best practices were adopted X X SAB perception of whether best practices were adopted X Characteristics of award activities, country, awardee X X (See above) 95 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other Factors leading to CEPPS3 success Question 4.1: Which factors contribute to the successful elevation of local organizations into sustainable providers of TA? Question 4.2: What are the characteristics or prerequisites of the local and regional organizations receiving CEPPS3 support which most align with potential for success? Question 4.3: Which CEPPS LWA “toolkit” approaches individually or in combination, show the greatest success rate for achieving the results specified for Objective 10? Outcomes (by C3SA, if possible) Concrete metrics of output (revenues, staff size, productivity indicators, skills acquisition, etc.) Which are local and regional? Concrete dimen￾sions of CSO out￾put X X X X Concrete measures of impact (increase in non￾CEPPS awards, greater demand for its services, etc.) Concrete dimen￾sions of CSO sustainability Define “success” X X X X Literature on NGO sustainability Covariates (by C3SA, if possible) CSO organizational characteristics (baseline, endline), and importance for local vs. region service provision CSO organizational characteristics X X X X Literature on NGO sustainability Award activity programmatic character￾istics (service components) Programmatic characteristics X X X 96 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other Degree and type of CEPPS implementation modalities of interaction with CSOs and the tool￾kits used) Types of CEPPS toolkits used: IRI scorecard X X X X Amount of training, technical assistance, learning by doing received by CSO from CEPPS implementer Taxonomy of TA and training (IRI was to provide list of training modules.) X X X X Country characteristics Which are key? X X X X (See above) Assessing CSO sustainability and financial management due to CEPPS3 assistance Q.5.1: What CEPPS efforts have supported local org opportunities for becoming direct recipients of local funding? Implementer perceptions of dichotomous or continuous variables on TA given to CSO to become a direct recipient of donor funding, by type of support and C3SA Taxonomy of CSO support to become direct recipients of donor funding Criteria from NUPAS3 X X X X Award SOW CSO perceptions of dichotomous or continuous variables on TA received to become a direct recipient of donor X X 3 NUPAS criteria are: 1) Legal Structure 2) Financial Management and Internal Control Systems 3) Procurement Systems 4) Human Resource Systems 5) Project Performance Management and 6) Sustainability. Each has several sub-elements. http://www.usaid.gov/sites/default/files/documents/1868/303sam.pdf 97 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other funding, by type of support and C3SA Country characteristics CSO organizational characteristics Which are key? Country: Freedom/ Democracy/ Political index score CSO: Financial Mgmt X X X X X (See above) Q.5.2: Have efforts been made to help local organizations’ planning in order to be prepared to apply for and receive larger and more complicated awards? Dichotomous or continuous variables of implementer perceptions on planning TA given to CSO to receive larger or more complex awards, by type of support and C3SA Taxonomy of planning support for CSO to receive larger / more complex awards X X X X Award SOW Dichotomous or continuous variables of CSO perceptions on planning TA received to increase CSO’s ability to handle larger or more complex awards, by type of support and C3SA X X Country characteristics CSO organizational characteristics Which are key? Country: Freedom/Democrac y/Politic index score CSO: NUPA sustainability sub￾X X X X X (See above) 98 Research Questions / variables Definitions / Comments KIIs Web surveys Other sources DRG / DC CEPP S DC office AO Rs Local CEPPS imple￾menter CSO bene￾ficiary Other CSOs USAID / CEPPS reports & PMPs Other elements Cash Flow Management and Absorptive Capacity. Q.5.3: What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative and project management practices)?(c) Dichotomous or continuous variables of implementer perceptions of efforts, by type of sup￾port and C3SA, to strengthen CSO sustainability through: Organizational management Administrative practices Project management Practical taxon￾omy to score implementer CSO support in terms of stated three areas X X X X Award SOW Literature on NGO sustainability Ditto from the CSO perspective X Country characteristics CSO organizational characteristics Which are key? X X X X X (See above) (a) Consider the following: http://www.wango.org/resources.aspx?section=ngodir http://esango.un.org/civilsociety/login.do. http://www.uia.org/international-organizations-research (b) C3SA = CEPPS3 study area (elections processes and political processes) (c) Note that this question is not on the effectiveness of such efforts, an issue addressed in Question IV. 99 (d) SAB = Sub-awardee or (substantial) beneficiary (e) This reflects the USAID research question’s interest in the sub-award selection tradeoff between ensuring a competitive selection process and supporting ongoing relationships. 100 ANNEX VII: ANALYSIS PLAN This document outlines a series of feasible analysis strategies to address with statistical rigor the research questions posed by USAID with regard to its CEPPS3 program. The plan explicitly draws on a CSO survey, implementer survey, USAID staff survey, focus groups, CEPPS3 implementer reports,4 budget data from DRG, third-party country-level databases, and key informant interviews and is based on the document “CEPPS Evaluation – Research Questions and Data Sources” (14 September 2015), as found in Annex XXX. The following sections are organized by USAID research question. Research Question 1 A. Question 1.1: Has the number and life expectancy of active CSOs in each C3SA increased since five years ago? CSO-survey questions most applicable are: Q0.1. Does CSO currently provide assistance in election processes? Q0.3. Does CSO currently provide assistance in political processes? Q0.2.1. Year CSO first engage in electoral-process activities Q0.3.2. Year CSO first engage in political-process activities Q0.5.1. Year CSO was first established. Q1.1. Are there more local civil-society organizations active today than there were five years ago (i.e., since the start of 2009) in the areas of elections and political assistance? Q1.3. Contact information of three of the newest CSOs you are aware of that provide technical assistance in the areas of national elections and/or political processes Q1.4. Names of three CSOs you are aware of that provided technical assistance in the areas of national elections and/or political processes but no longer exist. Step 1: Use CSO survey data to create a new country-level dataset using the record format given in Table 1. 4 The sheer number of these meant that the NORC team only examined a representative sample, not all. 101 Table 1: Dataset variables used in analysis and their sources Variable name Variable code Source 1 Country {𝐷𝐷𝑐𝑐 C} [Index key of file] 2 Year dummy [t={2009,2014}] 𝑇𝑇𝑡𝑡 [Index key of file] Equal to 0 if t=2009 and 1 o.w. 3 Region {𝐷𝐷𝑐𝑐 R} CIA Fact Book 4a* Number of CSOs in electoral processes 𝑁𝑁𝑐𝑐 E CSO survey: 2014=Q0.1 + report, 2009 derive from Q0.2.1. Be sure to cross-check against lists from implementer and AOR surveys 4b* Number of CSOs in political processes 𝑁𝑁𝑐𝑐 P CSO survey: 2014=Q0.3 + report, 2009 derive from Q0.3.2. Be sure to cross-check against lists from implementer and AOR surveys to avoid double-counting and for other candidates. 5* Number of CSOs 𝑁𝑁𝑐𝑐 Responding CSOs from country c for year t 6 Number of CEPPS3 CSOs in electoral processes 𝑀𝑀𝑐𝑐 E CSO survey: 2014= QA.4c, 2009 =QA.4b; then refer to CEPPS3 reports since survey doesn’t say area of focus. Be sure to cross-check against lists from implementer and AOR surveys to avoid double-counting and for other candidates. 7 Number of CEPPS3 CSOs in political processes 𝑀𝑀𝑐𝑐 P CSO survey: 2014= QA.4c, 2009 =QA.4b; then refer to CEPPS3 reports since survey doesn’t say area of focus. Be sure to cross-check against lists from implementer and AOR surveys to avoid double-counting and for other candidates. 8a Number of CEPPS3 CSOs 𝑀𝑀𝑐𝑐 Responding CSOs from country c for year t=2009, 2014. 8b Number of CEPPS2 CSOs 𝑀𝑀𝑐𝑐 (2) Number of CSO saying they got CEPPS2 benefits. 9a Real GDP (or PPP) per￾capita, year t 𝑌𝑌𝑐𝑐 World Tables, World Bank 9b Real GDP (or PPP) per￾capita, year t 𝑌𝑌𝑐𝑐,𝑡𝑡−1 World Tables, World Bank 9c Real GDP (or PPP) per￾capita, year t-2 𝑌𝑌𝑐𝑐,𝑡𝑡−2 World Tables, World Bank 9d Real GDP (or PPP) per￾capita, year t-3 𝑌𝑌𝑐𝑐,𝑡𝑡−3 World Tables, World Bank 9e Real GDP (or PPP) per￾capita, year t-4 𝑌𝑌𝑐𝑐,𝑡𝑡−4 World Tables, World Bank 10a Democracy Indicator, year t 𝐼𝐼𝑐𝑐,𝑡𝑡 H Heritage Foundation 10b Degree free 𝐼𝐼𝑐𝑐,𝑡𝑡 F Freedom House 102 Variable name Variable code Source 11 Number of NGOs, all types in 2015 𝐺𝐺𝑐𝑐 http://www.wango.org/resources.aspx?section=ngodir http://esango.un.org/civilsociety/login.do. http://www.uia.org/international-organizations￾research 12a USAID CEPPS country budget, year 2014 𝐵𝐵𝑐𝑐4 DRG 12b USAID CEPPS country budget, year 2013 𝐵𝐵𝑐𝑐,3 DRG 12c USAID CEPPS country budget, year 2012 𝐵𝐵𝑐𝑐,2 DRG 12d USAID CEPPS country budget, year 2011 𝐵𝐵𝑐𝑐,1 DRG 12e USAID CEPPS country budget, year 2010 𝐵𝐵𝑐𝑐,0 DRG 12f USAID CEPPS country budget, year 2009 𝐵𝐵𝑐𝑐,9 DRG 13a Population, year 2014 𝑃𝑃𝑐𝑐4 World Tables, World Bank 13b Population, year 2009 𝑃𝑃𝑐𝑐,9 World Tables, World Bank 14a IRI present 𝐷𝐷𝑐𝑐 I CEPPS 14b NDI present 𝐷𝐷𝑐𝑐 N CEPPS 14c IFES present 𝐷𝐷𝑐𝑐 F CEPPS 15 Number of distinct imple￾menters in country 𝐷𝐷𝑐𝑐 # A number from 1 to 3 16 Employees in CSOs 𝐿𝐿𝑐𝑐,𝑡𝑡 pt 𝐿𝐿𝑐𝑐,𝑡𝑡 ft 𝐿𝐿𝑐𝑐,𝑡𝑡 Part-time Full-time Total 17 Per capita CEPPS country budget, year 2014 𝐵𝐵𝑐𝑐,4 K [Calculated] 17b Per capita CEPPS country budget, year 2013 𝐵𝐵𝑐𝑐,3 K [Calculated] 103 Variable name Variable code Source 17c Per capita CEPPS country budget, year 2012 𝐵𝐵𝑐𝑐,2 K [Calculated] 17d Per capita CEPPS country budget, year 2011 𝐵𝐵𝑐𝑐,1 K [Calculated] 17e Per capita CEPPS country budget, year 2010 𝐵𝐵𝑐𝑐,0 K [Calculated] 17f Per capita CEPPS country budget, year 2009 𝐵𝐵𝑐𝑐,9 K [Calculated] 18a CEPPS country budget per unit GDP, year 2014 𝐵𝐵𝑐𝑐,4 G [Calculated] 18b CEPPS country budget per unit GDP, year 2013 𝐵𝐵𝑐𝑐,3 G [Calculated] 18c CEPPS country budget per unit GDP, year 2012 𝐵𝐵𝑐𝑐,2 G [Calculated] 18d CEPPS country budget per unit GDP, year 2011 𝐵𝐵𝑐𝑐,1 G [Calculated] 18e CEPPS country budget per unit GDP, year 2010 𝐵𝐵𝑐𝑐,0 G [Calculated] 18f CEPPS country budget per unit GDP, year 2009 𝐵𝐵𝑐𝑐,9 G [Calculated] 19 No national election since January t-1 𝐷𝐷𝑐𝑐,𝑡𝑡 V [CIA Fact Book? web? Ask experts?] 20 Number of CSOs that failed 𝑁𝑁𝑐𝑐 F CSO survey Q1.4. Note that the value will be the same in 2014 and 2009. Be sure to cross-check against lists from implementer and AOR surveys to avoid double-counting and for other candidates. 21 CSO’s opinion of whether number of CSOs has 𝑅𝑅𝑐𝑐 CSO survey: Q1.1 104 Variable name Variable code Source increased from 2009-14 22 Region￾relative number of CSOs that failed 𝑛𝑛𝑐𝑐 F CSO survey Q1.4. Note that the value will be the same in 2014 and 2009. 23 TA/training received by CSO 𝑇𝑇𝑐𝑐 (1) CSO survey: Q2.5c(A), Q2.6. Note that the superscript permits multiple variable options. 24* Average CSO age 𝐴𝐴𝑐𝑐 CSO survey: Q0.5.1 *Dependent variable. Step 2: Compute variables in 17. Compute Variable 20, that is, the number of CSOs said to have failed (Q1.4) by country. (You could also create an alternative version where you weigh the number of each country’s CSOs by the country’s share of CSO full-time employees.) Step 3a: Estimate the following regressions: 𝑁𝑁𝑐𝑐,𝑡𝑡 E = 𝛽𝛽0 E + 𝛽𝛽1 E𝑀𝑀𝑐𝑐,𝑡𝑡 E + 𝛽𝛽2 E𝑌𝑌𝑐𝑐,𝑡𝑡 + 𝛽𝛽3 E𝑀𝑀𝑐𝑐 (2) + 𝛽𝛽4 E𝐼𝐼𝑐𝑐,𝑡𝑡 F + 𝛽𝛽5 E𝐺𝐺𝑐𝑐,4 + 𝛽𝛽6 E𝐷𝐷𝑐𝑐,𝑡𝑡 V + 𝛽𝛽7 E𝑇𝑇𝑡𝑡 + 𝛽𝛽8 E𝐷𝐷𝑐𝑐 F + 𝛽𝛽9 E𝐷𝐷𝑐𝑐 N + 𝛽𝛽10 E 𝐷𝐷𝑐𝑐 I + 𝛽𝛽11 E 𝐷𝐷𝑐𝑐 # + {𝐷𝐷𝑐𝑐 R} + 𝜖𝜖𝑐𝑐,𝑡𝑡 E 𝑁𝑁𝑐𝑐,𝑡𝑡 P = 𝛽𝛽0 P + 𝛽𝛽1 P𝑀𝑀𝑐𝑐,𝑡𝑡 P + 𝛽𝛽2 P𝑌𝑌𝑐𝑐,𝑡𝑡 + 𝛽𝛽3 P𝑀𝑀𝑐𝑐 (2) + 𝛽𝛽4 P𝐼𝐼𝑐𝑐,𝑡𝑡 F + 𝛽𝛽5 P𝐺𝐺𝑐𝑐,4 + 𝛽𝛽6 P𝐷𝐷𝑐𝑐,𝑡𝑡 V + 𝛽𝛽7 P𝑇𝑇𝑡𝑡 + 𝛽𝛽8 P𝐷𝐷𝑐𝑐 F + 𝛽𝛽9 P𝐷𝐷𝑐𝑐 N + 𝛽𝛽10 P 𝐷𝐷𝑐𝑐 I + 𝛽𝛽11 P 𝐷𝐷𝑐𝑐 # + {𝐷𝐷𝑐𝑐 R} + 𝜖𝜖𝑐𝑐,𝑡𝑡 P 𝑁𝑁𝑐𝑐,𝑡𝑡 = 𝛽𝛽0 + 𝛽𝛽1 𝑀𝑀𝑐𝑐,𝑡𝑡 + 𝛽𝛽2 𝑌𝑌𝑐𝑐,𝑡𝑡 + 𝛽𝛽3 𝑀𝑀𝑐𝑐 (2) + 𝛽𝛽4 𝐼𝐼𝑐𝑐,𝑡𝑡 F + 𝛽𝛽5 𝐺𝐺𝑐𝑐,4 + 𝛽𝛽6 𝐷𝐷𝑐𝑐,𝑡𝑡 V + 𝛽𝛽7 𝑇𝑇𝑡𝑡 + 𝛽𝛽8 𝐷𝐷𝑐𝑐 F + 𝛽𝛽9 𝐷𝐷𝑐𝑐 N + 𝛽𝛽10𝐷𝐷𝑐𝑐 I + 𝛽𝛽11𝐷𝐷𝑐𝑐 # + {𝐷𝐷𝑐𝑐 R} + 𝜖𝜖𝑐𝑐,𝑡𝑡 One challenge with the above models is the need for 𝐵𝐵𝑐𝑐,8 K (i.e., the CEPPS2 country budgets) as the appro￾priate value for 𝐵𝐵𝑐𝑐,9 K . This may be hard to get and, so, the variable has been dropped. One unsatisfactory alternative would be to estimate the equations without 𝐵𝐵𝑐𝑐,9 K ; in this case the hypothesis test would be the significance and sign of 𝑇𝑇𝑡𝑡. Concerning 𝐺𝐺𝑐𝑐,4, I believe I told Alex to set it to missing for 2009; if so (and please check) then please set 𝐺𝐺𝑐𝑐,9 = 𝐺𝐺𝑐𝑐,4 so that the 2009 observations are not dropped. Step 3b: Estimate these regressions again but with country fixed effects instead of country variables. This requires dropping all covariates that are time-invariant. Step 4: Estimate the following regressions, where ∆𝑥𝑥 refers to the change from 2009 through 2014: ∆𝑁𝑁𝑐𝑐 E = 𝜃𝜃0 E + 𝜃𝜃1 E𝑀𝑀𝑐𝑐 (2) + 𝜃𝜃2 E𝑁𝑁𝑐𝑐,9 E + 𝜃𝜃3 E�𝑁𝑁𝑐𝑐,9 E � 2 + 𝜃𝜃4 E∆𝑌𝑌𝑐𝑐 + 𝜃𝜃5 E𝐵𝐵𝑐𝑐 K + 𝜃𝜃6 E � 𝐵𝐵𝑐𝑐 K 𝑀𝑀𝑐𝑐,4 � + 𝜃𝜃7 E𝐼𝐼𝑐𝑐,9 F + 𝜃𝜃8 E𝐺𝐺𝑐𝑐 + 𝜃𝜃9 E𝐷𝐷𝑐𝑐 F + 𝜃𝜃10 E 𝐷𝐷𝑐𝑐 N + 𝜃𝜃11 E 𝐷𝐷𝑐𝑐 I + 𝜃𝜃12 E 𝐷𝐷𝑐𝑐 # + {𝐷𝐷𝑐𝑐 R} + 𝜂𝜂𝑐𝑐 E ∆𝑁𝑁𝑐𝑐 P = 𝜃𝜃0 P + 𝜃𝜃1 P𝑀𝑀𝑐𝑐 (2) + 𝜃𝜃2 P𝑁𝑁𝑐𝑐,9 P + 𝜃𝜃3 P�𝑁𝑁𝑐𝑐,9 P � 2 + 𝜃𝜃4 P∆𝑌𝑌𝑐𝑐 + 𝜃𝜃5 P𝐵𝐵𝑐𝑐 K + 𝜃𝜃6 P � 𝐵𝐵𝑐𝑐 K 𝑀𝑀𝑐𝑐,4 � + 𝜃𝜃7 P𝐼𝐼𝑐𝑐,9 F + 𝜃𝜃8 P𝐺𝐺𝑐𝑐 + 𝜃𝜃9 P𝐷𝐷𝑐𝑐 F + 𝜃𝜃10 P 𝐷𝐷𝑐𝑐 N + 𝜃𝜃11 P 𝐷𝐷𝑐𝑐 I + 𝜃𝜃12 P 𝐷𝐷𝑐𝑐 # + {𝐷𝐷𝑐𝑐 R} + 𝜂𝜂𝑐𝑐 P 105 ∆𝑁𝑁𝑐𝑐 = 𝜃𝜃0 + 𝜃𝜃1 𝑀𝑀𝑐𝑐 (2) + 𝜃𝜃2 𝑁𝑁𝑐𝑐,9 + 𝜃𝜃3 �𝑁𝑁𝑐𝑐,9� 2 + 𝜃𝜃4 ∆𝑌𝑌𝑐𝑐 + 𝜃𝜃5 𝐵𝐵𝑐𝑐 K + 𝜃𝜃6 � 𝐵𝐵𝑐𝑐 K 𝑀𝑀𝑐𝑐,4 � + 𝜃𝜃7 𝐼𝐼𝑐𝑐,9 F + 𝜃𝜃8 𝐺𝐺𝑐𝑐 + 𝜃𝜃9 𝐷𝐷𝑐𝑐 F + 𝜃𝜃10𝐷𝐷𝑐𝑐 N + 𝜃𝜃11𝐷𝐷𝑐𝑐 I + 𝜃𝜃12𝐷𝐷𝑐𝑐 # + {𝐷𝐷𝑐𝑐 R} + 𝜂𝜂𝑐𝑐 There may be too few degrees of freedom for this to de well. If so, we can consider dropping terms, such as the squared term and the implementer dummies (but not 𝐷𝐷𝑐𝑐 #); if that is insufficient then drop the {𝐷𝐷𝑐𝑐 R}. Step 5: Compute 𝑅𝑅𝑐𝑐, the percent of respondents per country in 2014 answering “yes” to Opinion on CSO Growth Since 2009 (Q1.1.). Present the results by region and as a whole. Step 6: Estimate a Probit regression with 𝑅𝑅𝑐𝑐 as the dependent variable and the covariates from ∆𝑁𝑁𝑐𝑐 regression. Step 7: Use Age as: a. the dependent variable for the regressions, above. b. an independent variable in the regressions, above. Step 8: Run t-tests on the above dependent variables. Compare either 2009 to 2014 values for the panel variables or compare to zero (0) for the differenced variables. B. Question 1.2: Does a greater share of CSOs interact with CEPPS or their local partners in 2014 compared to five years ago? Step 1: Separately for CSOs who claim to interact with CEPPS or their local partners (QA.4a, QB.2c, QB.2d) and those that do not: a. Compute means and variance of Q1.2c1 and Q1.2c2 by country and region. b. Compute Q1.2c1-Q1.2c2 by CSO and then compute mean and variance by country and region. c. For 2014 and 2009, count the number of CSOs who claim to have interacted with CEPPS or their local partners. Divide these by the total number of CSOs we computed, above, for the respectively year. Research Question 2 A. Question 2.1: How were programmatic strategies developed under CEPPS II altered or expanded (e.g., dif￾ferent approaches to activities, the use of sub-grants) under CEPPS III to address the local capacity building objective? 1. Using Q2.3a and Q2.5a, compute for the two CEPPS periods the share of non-missing responses for each of the three response options. a. Create a table with the proportions of CSOs for each of the three response options and for each of the two CEPPS periods. Note that these will not add to 100 percent since a CSO can say that they receive both types of capacity building. 106 b. Determine whether there is statistical evidence for a change in the reported amount (distribution) of capacity building from CEPPS2 and CEPPS3, by using separate Chi-Squared tests with two degrees of freedom on those CSOs claiming they received capacity building to compare between the two CEPPS periods the share indicating receipt of: i. organizational capacity building; ii. technical capacity building; iii. both technical and organizational capacity building c. For the full sample claiming participation in either CEPPS2, CEPPS3 or both determine using a Chi-Squared test with two degrees of freedom whether there is statistical evidence for a change in the proportion of capacity building occurring from CEPPS2 and CEPPS3. d. Add a column to the table with the results of the hypothesis tests. 2. Number of grants/contracts in each evaluation period Using QA.4b and QA.4c, determine whether there was a shift in emphasis between grants and contracts in each evaluation period. Toward this end, a. Compute the values for the CEPPS2 and CEPPS3 periods of the share of contracts each CSO received. i. There could be up to three contracts/grants reported per period. Hence, for each CSO having a positive answer to QA.4.b or QA.4c, compute the proportion with the answer “Contract” (1). ii. Compute the average, by region and by implementer-supported countries, for each of the two CEPPS periods. b. Conduct a Chi-Squared test with two degrees of freedom on whether there is statistical evidence for a change in the proportion of joint work occurring from CEPPS2 and CEPPS3 across the countries. 3. Repeat an analogous analysis to Q.A2 on whether funding obtained (i.e., each of the contracts or grants in the previous calculations) was through competitive procurement or not. B. Question 2.2: Was CEPPS’s process for selecting local partner organizations optimal in the sense that com￾petitive selection processes were balanced against the need for establishing ongoing relationships? 1. Create a dataset with one observation per award j of CSO i in country c and create a. a dichotomous variable 𝐹𝐹𝑖𝑖,𝑐𝑐 (2) equal 1 if QA.4b is greater than zero and 0 otherwise. b. a dichotomous variable 𝐶𝐶𝑖𝑖,𝑗𝑗,𝑐𝑐 equal 1 if QA.4c2 is greater than 1 and 0 otherwise. 2. Determine whether the probability of competitive procurement during CEPPS3 is lower for a CEPPS2 rather than a CEPPS3 CSO: a. Estimate the logit regression 𝒫𝒫 �𝐶𝐶𝑖𝑖,𝑗𝑗,𝑐𝑐 = 1|𝐹𝐹𝑖𝑖,𝑐𝑐 (2) , {𝑐𝑐𝑖𝑖}� b. Run a t-test on the coefficient of 𝐹𝐹𝑖𝑖,𝑐𝑐 (2) 107 3. Redo QB.2 but also including the independent covariates, 𝐷𝐷𝑖𝑖,𝑐𝑐 I , 𝐷𝐷𝑖𝑖,𝑐𝑐 N , and 𝐷𝐷𝑖𝑖,𝑐𝑐 R , and their interac￾tion terms with 𝐹𝐹𝑖𝑖,𝑐𝑐 (2) . Now test whether the interaction terms are statistically significantly dif￾ferent from zero, first, individually and then as a group (F-test). 4. Redo QB.2 and QB.3 but dropping 𝐹𝐹𝑖𝑖,𝑐𝑐 (2) and including the two dichotomous independent covari￾ates, 𝐹𝐹=1,𝑖𝑖,𝑐𝑐 (2) and 𝐹𝐹>1,𝑖𝑖,𝑐𝑐 (2) . The latter two are also dummies, with the first equal to 1 only if the answer to QA4.b is exactly 1 (and zero otherwise) and with the second equal to 1 only if the answer to QA4.b is greater than 1 (and zero otherwise). 5. Degree procurement was considered fair by CSOs, as a whole and by implementer. a. For those CSOs answering “Yes” to Q2.1a, create a dichotomous variable, 𝐷𝐷𝑖𝑖,𝑗𝑗,𝑐𝑐 Fair that is equal to one only if the answer to Q2.1b is “Yes” and zero otherwise. b. This is a binomial population so, we use a standard Normal to determine whether the proportion is different from zero, i.e., the procurements were considered “fair”. c. Conduct a Chi-Squared test with three degrees of freedom on whether there is statistical evidence that the degree “fair” is equal across the three implementers. C. Question 2.3: To what degree did receiving a CEPPS grant in 2009 change the probability of receiving an award in 2014, all else held equal? 1. Add to the CSO-level dataset a dichotomous variable, 𝐹𝐹𝑖𝑖,𝑐𝑐 (3) equal 1 if QA.4a is greater than 1 and 0 otherwise. 2. Determine whether the probability of getting an award during CEPPS2 the probability of getting one in CEPPS3: i. Estimate the logit regression 𝒫𝒫 �𝐹𝐹𝑖𝑖,𝑐𝑐 (3) = 1|𝐹𝐹𝑖𝑖,𝑐𝑐 (2) , {𝑐𝑐𝑖𝑖}� ii. Run a t-test on the coefficient of 𝐹𝐹𝑖𝑖,𝑐𝑐 (2) 3. Redo C.2, above, but now also including the independent covariates, 𝐷𝐷𝑖𝑖,𝑐𝑐 I , 𝐷𝐷𝑖𝑖,𝑐𝑐 N , and 𝐷𝐷𝑖𝑖,𝑐𝑐 R , and their interaction terms with 𝐹𝐹𝑖𝑖,𝑐𝑐 (2) . Now test whether the interaction terms are statistically significantly different from zero, first, individually and then as a group (F-test). Research Question 3 A. Question 3.1: To what degree did CEPPS involve local partners in the design, awards management, M&E processes, and reporting of programs with which they were affiliated? 1. Compute the share of CSOs answering “1” (Implementer decided) to Q3.1 This is a binomial population so, if the sample size is greater than 30 we used a standard Normal to determine whether there is statistical evidence that the implementers dictated 100 percent of the time, 50 percent of the time, or never. 108 2. Compute for each implementer the share of CSOs answering “Implementer decided” (1) to Q3.1. Conduct a Chi-Squared test with three degrees of freedom on whether there is statistical evidence that the implementers dictated is equal across the three implementers. 3. For each of the following questions whose answers are not missing (skipped) or with a “Do Not Know” (98) answer, compute the share of CSOs answering “To a great extent” (4). This is a binomial population so, if the sample size is greater than 30 we should be able to use a standard Normal (otherwise use a binominal distribution) to determine whether there is statistical evi￾dence that the share was 100 percent, 50 percent, or zero. a. Q3.2b b. Q3.3b c. Q3.4b d. Q3.5b 4. Compute the share of CSOs answering “To a great extent” (4) for each implementer. For each question in QA.3, conduct a Chi-Squared test with three degrees of freedom on whether the hypothesis that the shares are equal across the three implementers can be rejected. B. Question 3.2: Did CEPPS incorporate organizational capacity-building into their collaborations on elections and political processes or were these separated into a parallel tracks? 1. Using Q2.3b and Q2.3d, determine whether there was a shift in emphasis on technical assistance provision through joint work or through direct assistance. Toward this end, i. Compute the values for the CEPPS2 and CEPPS3 periods of the share of technical assistance each CSO receives on average from joint work. 1. Count the number responses to A to E that are “Technical assistance through joint work” (1). Include in computation only the A-E responses that are not “N/A (97)”. Hence, the calculation should yield a number between 0 and 5. 2. Divide the answer by the number of A-E responses that are not “N/A (97)”. ii. Conduct a Chi-Squared test with two degrees of freedom on whether there is statistical evidence for a change in the proportion of joint work occurring from CEPPS2 and CEPPS3 C. Question 3.3: To what extent have CEPPS’s local and regional partners adopted best practices, such as using tools such as gender analysis, for their donor as well as non-donor funded work? 1. For answers to Q3.6b that are not missing (skipped) or “Do Not Know” (98), compute the share of CSOs answering that the degree they incorporate gender analysis is “To a great extent” (4). This is a binomial population so we used a standard Normal to determine whether there is statistical evidence that the share was 100 percent, 50 percent, or zero. 2. Repeat the previous analysis but with Q3.6c. 3. Does training have an effect? 109 a. Create the dichotomous variable, 𝑇𝑇𝑖𝑖,𝑐𝑐 Gen that is equal to 1 if the answer to Q3.6a is “Yes” (1), and zero otherwise. b. Compute the dichotomous variable, Ei,c that equals 1 only if CSO i’s answer to Q3.6b is greater than that of Q3.c, and zero otherwise. Set the variable to missing if Q3.6a, Q3.6b, or Q3.6c “Do not know” (98). c. Estimate the probit model, regression 𝒫𝒫�𝐸𝐸𝑖𝑖,𝑐𝑐 = 1|𝑇𝑇𝑖𝑖,𝑐𝑐 Gen�. d. Test the hypothesis that the coefficient of 𝑇𝑇𝑖𝑖,𝑐𝑐 Gen is not statistically significantly different from zero. e. Estimate the probit model, regression: 𝒫𝒫�𝐸𝐸𝑖𝑖,𝑐𝑐 = 1|𝑇𝑇𝑖𝑖,𝑐𝑐 Gen,𝐷𝐷𝑖𝑖,𝑐𝑐 I ,𝐷𝐷𝑖𝑖,𝑐𝑐 N ,𝐷𝐷𝑖𝑖,𝑐𝑐 R , 𝑇𝑇𝑖𝑖,𝑐𝑐 Gen𝐷𝐷𝑖𝑖,𝑐𝑐 I , 𝑇𝑇𝑖𝑖,𝑐𝑐 Gen𝐷𝐷𝑖𝑖.𝑐𝑐 N , 𝑇𝑇𝑖𝑖,𝑐𝑐 Gen𝐷𝐷𝑖𝑖,𝑐𝑐 R � and test the interaction terms on whether we can reject the hypothesis that each is independently is zero and then that they are jointly zero. 4. For answers to Q3.10a that are not missing (skipped) or “Do Not Know” (98), compute the share of CSOs answering that, “Yes” (1), they utilize a time sheet system. This is a binomial population so we used a standard Normal to determine whether there is statistical evidence that the share was 100 percent, 50 percent, or zero. Research Question 4 A. Question 4.1: Which factors contribute to the successful elevation of local organizations into sustainable providers of TA? 1. Construct measures of the degree a CSO is a “sustainable provider of TA” and of factors. a. We do not ask that question directly nor is it directly observable. Moreover, in selecting variables one must be careful not to confound inputs to sustainability with sustainability itself. This may be hard or even counterproductive: what is the difference between an ex ante sign of sustainability and an input or requirement for sustainability? Ideally, having a causal sequence would be helpful, though data we have for most variables on the following list with which to construct an indicator make it difficult to make the distinction. i. number of full-time staff (Q0.4bA); ii. uses consistent donor-approved overhead rate for budgeting (Q3.11a); iii. timesheet system (Q3.10a); iv. received non-CEPPS international funding for latest election (QB.2e>0); v. receives non-USAID funding (Q3.12); vi. has adopted CEPPS’ capacity building approaches (Q2.8); vii. dichotomous variables for answering “1” and for answering “4” to Q3.13; viii. age (Q0.5.1); ix. above-average – we need to decide on the average of what – number of local awards in CEPPS3 period (Q3.12); 110 x. number of grants/contracts from local clients increased from CEPPS2 to CEPPS3; xi. number of grants/contracts from international clients increased from CEPPS2 to CEPPS3, has worked abroad (QA.3); xii. greater than average (which?) number of responses to “main activities” (QA.1); and xiii. greater than average (which?) number of responses to “items the organization has” (QA.2a). b. Covariates for country characteristics include: i. Freedom indicators ii. Population iii. GDP per capita iv. Number of CSOs in the country v. In Africa vi. Current or former communist country vii. Arab country c. Covariates for Implementer include:5 i. Which implementer (dummies for each of the three) ii. Whether there is more than one implementer in the country d. Options to compute the sustainability indicator: i. Use expert weight and aggregate on all or on alternative subsets ii. Use principal components on all or alternative subsets of variables iii. Use alternative individual variables from the above list iv. The list could be divided into a set of “more” and “less” outcome-oriented variables, with the latter considered causal antecedent CSO factors of sustainability. e. For now, over the full sample we created 𝐼𝐼𝑖𝑖,𝑐𝑐 𝑥𝑥 , where x refers to either 𝐼𝐼𝑖𝑖,𝑐𝑐 PC (the first principal component) or 𝐼𝐼𝑖𝑖,𝑐𝑐 W ( the equi-weighted average) of the standardized following variables: i. Change in the number of full-time staff from CEPPS2 to CEPPS3 (Q0.4bA1, Q0.4bA2); ii. Change in the number of grants/contracts from local clients increased from CEPPS2 to CEPPS3; iii. Change in the number of grants/contracts from international clients increased from CEPPS2 to CEPPS3, has worked abroad (QA.3); iv. age in 2014 (Q0.5.1); 5 Note the threat of endogeneity here: Does more CEPPS action imply that greater help was required or that the chances for sustainability were higher? Either way, causation will go in both directions, compromising the validity of hypothesis tests. 111 2. In principle, the following are sources of influence: CSO, country (or region), and CEPPS activity, and implementer characteristics. Thus, for each 𝐹𝐹𝑖𝑖,𝑐𝑐 x not used in the I construction, we conducted the following two regressions (i.e., for x=PC and W): 𝐼𝐼𝑖𝑖,𝑐𝑐 𝑥𝑥 = 𝛽𝛽0 x + 𝛽𝛽1 x𝐹𝐹𝑖𝑖,𝑐𝑐 x + 𝛽𝛽2 x 𝑁𝑁𝑐𝑐𝑐𝑐 + 𝛽𝛽3 x𝐺𝐺𝑐𝑐𝑐𝑐 + 𝛽𝛽4 x 𝑌𝑌𝑐𝑐 + 𝛽𝛽5 x𝑃𝑃𝑐𝑐,9 + 𝛽𝛽3 x𝐵𝐵𝑐𝑐,9 + 𝛽𝛽6 x 𝐷𝐷𝑖𝑖,𝑐𝑐,9 Pro + 𝛽𝛽7 x𝐷𝐷𝑖𝑖,𝑐𝑐 N + 𝛽𝛽8 x 𝐷𝐷𝑖𝑖,𝑐𝑐 I + 𝛽𝛽9 x𝐷𝐷𝑖𝑖,𝑐𝑐 R +𝛽𝛽10 x 𝐷𝐷𝑐𝑐 # + {𝐷𝐷𝑟𝑟} + 𝜖𝜖𝑐𝑐,𝑡𝑡 x where 𝐹𝐹𝑖𝑖,𝑐𝑐 x is any one the standardized versions of the variables in Task A.1.1.a, above, 𝐷𝐷𝑟𝑟 are region dummies, and 𝐷𝐷𝑖𝑖,𝑐𝑐,9 Pro is equal to 1 if CSO i did electoral processes in 2009 and zero otherwise. B. Question 4.2: What are the characteristics or prerequisites of the local and regional organizations receiving CEPPS3 support which most align with potential for success? 1. The list of CSO characteristics in Task A.1.1.a can be interacted with to see whether the factors of success differ for CEPPS3 beneficiaries. 𝐼𝐼𝑖𝑖,𝑐𝑐 𝑥𝑥 = 𝛽𝛽0 x + 𝛽𝛽1 x 𝐹𝐹𝑖𝑖,𝑐𝑐 x + 𝛿𝛿𝑥𝑥 𝐹𝐹𝑖𝑖,𝑐𝑐 x 𝐹𝐹𝑖𝑖,𝑐𝑐 (3) + 𝛽𝛽2 x𝑁𝑁𝑐𝑐𝑐𝑐 + 𝛽𝛽3 x𝐺𝐺𝑐𝑐𝑐𝑐 + 𝛽𝛽4 x𝑌𝑌𝑐𝑐 + 𝛽𝛽5 x 𝑃𝑃𝑐𝑐,9 + 𝛽𝛽3 x 𝐵𝐵𝑐𝑐,9 + 𝛽𝛽6 x𝐷𝐷𝑖𝑖,𝑐𝑐,9 Pro + 𝛽𝛽7 x 𝐷𝐷𝑖𝑖,𝑐𝑐 N + 𝛽𝛽8 x𝐷𝐷𝑖𝑖,𝑐𝑐 I +𝛽𝛽9 x𝐷𝐷𝑖𝑖,𝑐𝑐 R + 𝛽𝛽10 x 𝐷𝐷𝑐𝑐 # + {𝐷𝐷𝑟𝑟} + 𝜖𝜖𝑐𝑐,𝑡𝑡 x where 𝐹𝐹𝑖𝑖,𝑐𝑐 (3) is a dummy equal to 1 if CSO received CEPPS3 funding and zero otherwise. It may be advis￾able to drop those covariates that were not statistically significant in the analysis of Question 4.1. Research Question 5 Note: for all regressions, use the covariates GDP per capita, region dummies, and implementer dummies (if we can link them to specific CSOs, otherwise leave them out). A. Pre-processing: 1. Create a dummy for Obtained_Funds = 1 if Q5.1d == 1 2. Create three dummy variables using Q5.2a: a. imp_loe_1 = 1 if Q5.2a == 4; =missing if Q5.2a == 98; =0 if Q5.2a b. imp_loe_2 = 1 if Q5.2a == 3 c. imp_loe_3 = 1 if Q5.2a == 4 | 3 3. Create a table to show the proportions for Q5.1d == 1, Q5.2a == 4, and Q5.2a == 3 by region and by implementer. B. Question 5.1: What CEPPS efforts have supported local organization opportunities for becoming direct recipients of local funding? 1. Run a series of t-tests, testing the mean of Obtained_Funds, by the three dummy variables imp_loe_*. 2. Regress 5.1d dummy (Obtained_Funds) against the Q3.2b, Q3.3b, Q3.4b, Q3.5b dummies, which equal 1 if the question == 4 (“to a great extent”) 112 3. Create separate dummies for Q5.1a, Q5.1b, and Q5.1c that =1 if the questions are “yes,” 0 if “no,” and missing otherwise. 4. Regress the Q5.1d dummy (Obtained_Funds) jointly against the three dummies Q5.1a, Q5.1b, and Q5.1c. 5. Create a table with the proportions by region and implementer from each dummy. C. Question 5.2: Have efforts been made to help local organizations’ planning in order to be prepared to apply for and receive larger and more complicated awards? 1. Compute the shares, 𝜇𝜇X, 𝜇𝜇𝑚𝑚 X , and 𝜇𝜇𝑟𝑟 X, of CSOs answering CEPPS3 helped on compleX award preparation (Q5.3) “To a great extent (4)” for the sample as a whole, by implementer m ϵ {I,N,R} (to be done later), and by region. Create a table with these shares. (Consider, if appropriate, consolidating the tables mentioned herein.) 2. These are each from a binomial population so we used a standard Normal to statistically test the hypotheses that implementer efforts were considered successful 100 percent of the time, 50 percent of the time, 25 percent of the time, or never. 3. Using only the following dummies, estimate a probit of 𝒫𝒫�𝐾𝐾𝑖𝑖 = 1|𝐷𝐷𝑐𝑐 I ,𝐷𝐷𝑐𝑐 N,𝐷𝐷𝑐𝑐 R,𝐷𝐷𝑐𝑐 #�, where Ki is equal to 1 if CSO i answers Q5.3 “To a great extent” (4) and zero otherwise. Run an F-test on whether the hypothesis that the shares are equal across the three implementers can be rejec￾ted. 4. Re-estimate the previous model after adding in a few covariates. Examine individual implementer effectiveness. 5. Run a probit of Obtained_Funds against the imp_loe_* dummies, separately. D. Question 5.3: What other efforts has CEPPS made to increase sustainability of local partners (such as efforts to build management, administrative and project management practices)? 1. Regress the alternative sustainability indicators constructed, above, against the variables {𝑆𝑆𝑖𝑖 (𝑄𝑄) } (as defined in Q4). One might also add in the covariates 𝐷𝐷𝑐𝑐 I ,𝐷𝐷𝑐𝑐 N,𝐷𝐷𝑐𝑐 R, �𝑆𝑆𝑖𝑖 (𝑄𝑄) 𝐷𝐷𝑐𝑐 I � , �𝑆𝑆𝑖𝑖 (𝑄𝑄) 𝐷𝐷𝑐𝑐 N�, and �𝑆𝑆𝑖𝑖 (𝑄𝑄) 𝐷𝐷𝑐𝑐 R�. 2. Regress each of the two sustainability indices against the Q3.2b, Q3.3b, Q3.4b, Q3.5b dummies. 113 ANNEX VIII: REGRESSIONS Regressions for Research Question 1: VARIABLES Number of CSOs in Electoral Processes Number of CSOs in Electoral Processes (Country Fixed-Effects) Number of CSOs in Electoral Processes Number of CSOs in Electoral Processes (Country Fixed-Effects) Number of CSOs in electoral/political processes Number of CSOs in electoral/political Processes (Country Fixed-effects) Year 0.624* 1.563*** 0.646 1.727*** -0.688*** 0.715*** (0.335) (0.340) (0.404) (0.414) (0.261) (0.192) Number of Electoral CSOs (CEPPS3) 1.011*** 0.312 (0.117) (0.266) Number of Political CSOs (CEPPS3) 1.252*** 1.268*** (0.145) (0.296) Number of Grantees 0.801*** 0.100* (0.0851) (0.0539) Number of CEPPS2 Grantees 0.180 -0.0459 0.141 (0.153) (0.172) (0.111) GDP per-capita -0.000141** -0.000148 -0.000149* -0.000253 -1.85e-05 -3.93e-05 (6.74e-05) (0.000188) (7.96e-05) (0.000196) (4.55e-05) (8.53e-05) Not Free -0.639 -0.904 0.199 (0.683) (0.809) (0.462) Partially Free -0.484 -0.510 0.0792 (0.648) (0.767) (0.438) Number of NGOs 0.00155** 0.000631 0.000815* (0.000667) (0.000793) (0.000440) No Election Last Year 0.429 -0.132 0.810** 0.233 -0.0284 0.114 (0.341) (0.327) (0.404) (0.345) (0.230) (0.149) IRI 0.252 0.648 -0.323 (0.626) (0.738) (0.420) 114 Number of Distinct Implementers 1.236** 1.267** 0.361 (0.484) (0.574) (0.326) IFES 0.428 0.465 -0.176 (0.654) (0.774) (0.442) NDI - - - Average CSO Age (per-country) 0.0649*** -0.196*** 0.0550*** -0.240*** 0.0355*** -0.101*** (0.0151) (0.0580) (0.0186) (0.0581) (0.00977) (0.0259) Constant -2.329*** 5.615*** -2.353** 4.251*** -0.237 3.501*** (0.816) (1.292) (0.967) (1.315) (0.557) (0.443) Observations 95 95 95 95 95 95 R-squared 0.886 0.981 0.857 0.981 0.866 0.990 Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 115 VARIABLES Average CSO Age Average CSO Age (Country Fixed-Effects) Average CSO Age Average CSO Age (Country Fixed-Effects) Average CSO Age Average CSO Age (Country Fixed-Effects) Year 1.519 3.811*** -0.316 4.879*** -1.467 5.209*** (2.434) (0.698) (2.386) (0.812) (2.925) (0.822) Number of Electoral CSOs (CEPPS3) 2.933*** -2.590*** (0.789) (0.593) Number of Political CSOs (CEPPS3) 3.602*** -2.516*** (0.762) (0.691) Number of Grantees 2.280** -1.098*** (0.923) (0.276) Number of CEPPS2 Grantees -3.189*** -3.268*** -2.777** (1.056) (0.949) (1.204) GDP per-capita 0.000281 -0.000509 0.000256 -0.000279 0.000226 -0.000355 (0.000490) (0.000499) (0.000469) (0.000524) (0.000511) (0.000512) Not Free -3.518 -3.219 -3.402 (4.968) (4.762) (5.178) Partially Free 0.780 1.403 1.140 (4.723) (4.526) (4.922) Number of NGOs 0.00113 -0.000493 0.00423 (0.00486) (0.00468) (0.00492) 116 No Election Last Year 0.126 -0.548 0.395 -1.268 -0.0685 -1.055 (2.483) (0.877) (2.381) (0.905) (2.587) (0.886) IRI 8.005* 6.976 6.590 (4.481) (4.288) (4.668) Number of Distinct Implementers -3.404 -3.727 -2.701 (3.508) (3.363) (3.646) IFES 3.461 3.083 3.041 (4.754) (4.558) (4.958) NDI - - - Constant 10.86* 18.78*** 10.66* 18.44*** 11.82* 14.62*** (5.828) (1.872) (5.583) (2.047) (6.120) (1.390) Observations 95 95 95 95 95 95 R-squared 0.238 0.983 0.300 0.981 0.172 0.982 Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 117 Regressions for Research Question 4.1 (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) VARIABLES Sustain ability Index (weight ed) (a) Sustain ability Index (weight ed) (b) Sustain ability Index (PCA) (a) Sustain ability Index (PCA) (b) Sustain ability Index (weight ed) (a) Sustain ability Index (weight ed) (b) Sustain ability Index (PCA) (a) Sustain ability Index (PCA) (b) Sustain ability Index (weight ed) (a) Sustain ability Index (weight ed) (b) Sustain ability Index (PCA) (a) Sustain ability Index (PCA) (b) Uses Consistent Donor-Approved Overhead Rate 0.141 0.145 0.257* * 0.251* * (0.109) (0.106) (0.125) (0.121) Uses a timesheet system - 0.0102 0.0016 0 - 0.0515 - 0.0300 (0.099 8) (0.099 8) (0.118) (0.117) Received non-CEPPS international funding for latest election - 0.0660 - 0.0562 - 0.0891 - 0.0650 (0.105) (0.106) (0.128) (0.128) Number of CSOs (per country) - 0.0056 8 - 0.0068 7 - 0.0033 9 - 0.0045 8 - 0.0092 5 - 0.0099 2 - 0.0064 6 - 0.0078 8 - 0.0123 - 0.0142 - 0.0108 - 0.0158 (0.009 05) (0.008 57) (0.010 4) (0.009 74) (0.008 50) (0.008 18) (0.010 1) (0.009 62) (0.010 6) (0.010 9) (0.012 9) (0.013 2) Number of NGOs (per country) 0.0001 32 0.0000 701 0.0003 75 0.0002 46 0.0000 146 - 0.0000 231 0.0001 34 0.0000 537 - 0.0002 46 - 0.0003 29 - 0.0000 835 - 0.0002 22 (0.000 435) (0.000 427) (0.000 499) (0.000 485) (0.000 420) (0.000 414) (0.000 498) (0.000 487) (0.000 416) (0.000 412) (0.000 506) (0.000 498) GDP Per-Capita 0.107* 0.124* 0.0900 0.113 0.123* * 0.140* * 0.124* 0.144* * 0.141* * 0.155* * 0.142* 0.158* * (0.063 8) (0.062 9) (0.073 2) (0.071 5) (0.060 1) (0.059 6) (0.071 3) (0.070 1) (0.062 2) (0.060 4) (0.075 6) (0.073 1) 118 Population (2009) 0.0005 24 - 0.0012 2 - 0.0010 4 - 0.0037 0 0.0008 13 - 0.0007 23 - 0.0001 99 - 0.0023 3 0.0027 1 0.0018 9 0.0016 7 0.0001 26 (0.002 96) (0.003 15) (0.003 39) (0.003 58) (0.002 85) (0.003 05) (0.003 37) (0.003 58) (0.002 89) (0.003 02) (0.003 52) (0.003 66) Number of implementers in country 0.199 0.253 0.171 0.159 0.209 0.224 (0.379) (0.435) (0.348) (0.412) (0.357) (0.434) 0.0612 -0.145 0.0435 - 0.0740 - 0.0687 -0.213 IRI (0.385) (0.441) (0.359) (0.426) (0.365) (0.444) - 0.0993 - 0.0599 -0.134 - 0.0776 - 0.0293 0.0291 NDI (0.416) (0.477) (0.384) (0.455) (0.386) (0.470) 0.371 0.520 0.598 0.844 0.214 0.350 0.341 0.521 0.110 0.143 0.271 0.297 IFES (0.491) (0.483) (0.563) (0.549) (0.459) (0.454) (0.544) (0.534) (0.493) (0.484) (0.599) (0.585) 0.140 - 0.0251 0.114 - 0.0712 - 0.0294 -0.162 -0.170 -0.330 -0.148 -0.236 -0.237 -0.386 Asian Country (0.400) (0.401) (0.460) (0.455) (0.367) (0.371) (0.435) (0.436) (0.380) (0.396) (0.462) (0.479) -0.229 -0.206 -0.285 -0.208 -0.368 -0.343 -0.612 -0.585 -0.456 -0.447 -0.687 -0.714 Eurasian Country (0.484) (0.461) (0.555) (0.524) (0.458) (0.442) (0.544) (0.520) (0.457) (0.452) (0.555) (0.547) -0.154 - 0.0946 0.0839 0.263 -0.352 -0.291 -0.219 - 0.0981 -0.633 -0.598 -0.333 -0.235 Latin America/Caribbean Country (0.541) (0.518) (0.621) (0.588) (0.518) (0.501) (0.614) (0.589) (0.591) (0.561) (0.719) (0.679) 0.268 0.471* 0.225 0.353 0.208 0.403 MENA Country (0.208) (0.236) (0.200) (0.235) (0.226) (0.274) 119 -0.428 -0.936* -0.418 - 1.447* * -0.296 -0.756 -0.309 -1.074* -0.168 -0.583 -0.211 -1.021 Constant (0.509) (0.531) (0.584) (0.603) (0.481) (0.515) (0.570) (0.605) (0.523) (0.522) (0.636) (0.631) Observations 89 89 89 89 99 99 99 99 85 85 85 85 R-squared 0.143 0.150 0.144 0.168 0.120 0.121 0.079 0.095 0.196 0.193 0.144 0.150 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 120 (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) VARIABLES Sustain ability Index (weight ed) (a) Sustain ability Index (weight ed) (b) Sustain ability Index (PCA) (a) Sustain ability Index (PCA) (b) Sustain ability Index (weight ed) (a) Sustain ability Index (weight ed) (b) Sustain ability Index (PCA) (a) Sustain ability Index (PCA) (b) Sustain ability Index (weight ed) (a) Sustain ability Index (weight ed) (b) Sustain ability Index (PCA) (a) Sustain ability Index (PCA) (b) Has adopted CEPPS's capacity￾building approaches - 0.0974 -0.101 -0.129 -0.137 (0.123) (0.123) (0.148) (0.149) Needs help with financial management 0.0202 0.0252 0.0816 0.101 (0.110) (0.108) (0.130) (0.127) Needs help with fundraising 0.108 0.0874 0.112 0.0804 (0.116) (0.113) (0.138) (0.133) Number of CSOs (per country) - 0.0138 - 0.0114 - 0.0152 - 0.0102 - 0.0091 5 - 0.0099 5 - 0.0059 6 - 0.0078 0 - 0.0092 9 - 0.0097 2 - 0.0062 0 - 0.0075 2 (0.011 3) (0.010 8) (0.013 6) (0.013 0) (0.008 47) (0.008 16) (0.010 0) (0.009 57) (0.008 43) (0.008 14) (0.010 0) (0.009 59) Number of NGOs (per country) 0.0005 75 0.0004 81 0.0004 48 0.0003 08 0.0000 230 - 0.0000 216 0.0001 74 0.0000 781 0.0000 232 - 0.0000 241 0.0001 67 0.0000 685 (0.000 550) (0.000 555) (0.000 662) (0.000 671) (0.000 416) (0.000 411) (0.000 493) (0.000 482) (0.000 414) (0.000 410) (0.000 492) (0.000 483) GDP Per-Capita 0.127 0.155* 0.142 0.171* 0.121* * 0.139* * 0.117 0.139* 0.127* * 0.144* * 0.127* 0.147* * (0.082 2) (0.079 6) (0.098 9) (0.096 2) (0.060 4) (0.059 8) (0.071 6) (0.070 1) (0.060 0) (0.059 5) (0.071 3) (0.070 2) 121 Population (2009) - 0.0051 0 - 0.0059 6 - 0.0045 4 - 0.0063 1 0.0008 65 - 0.0006 83 0.0000 0241 - 0.0022 7 0.0008 20 - 0.0006 53 - 0.0002 16 - 0.0023 5 (0.003 78) (0.004 09) (0.004 55) (0.004 94) (0.002 86) (0.003 04) (0.003 39) (0.003 56) (0.002 83) (0.003 02) (0.003 36) (0.003 56) Number of implementers in country 0.411 0.655 0.174 0.175 0.207 0.204 (0.385) (0.463) (0.347) (0.411) (0.348) (0.413) IRI 0.327 0.126 0.0461 - 0.0639 0.0373 - 0.0825 (0.383) (0.461) (0.359) (0.425) (0.357) (0.424) NDI 0 0 -0.136 - 0.0869 -0.120 - 0.0636 (.) (.) (0.384) (0.454) (0.382) (0.454) IFES 0.826 0.794 0.868 1.043 0.207 0.341 0.313 0.496 0.260 0.389 0.395 0.567 (0.684) (0.659) (0.822) (0.796) (0.461) (0.454) (0.546) (0.532) (0.459) (0.454) (0.545) (0.535) Asian Country 0.237 0.0185 - 0.0306 -0.170 - 0.0096 3 -0.146 - 0.0873 -0.262 0.0304 - 0.0992 - 0.0953 -0.267 (0.489) (0.492) (0.589) (0.595) (0.376) (0.376) (0.446) (0.441) (0.369) (0.378) (0.438) (0.445) Eurasian Country -0.215 -0.238 -0.694 -0.644 -0.346 -0.329 -0.512 -0.497 -0.376 -0.341 -0.585 -0.555 (0.649) (0.644) (0.780) (0.778) (0.456) (0.438) (0.540) (0.514) (0.448) (0.432) (0.533) (0.509) Latin America/Caribbean Country -0.485 -0.627 -0.278 -0.111 -0.326 -0.269 -0.107 0.0182 -0.269 -0.223 -0.104 - 0.0108 (1.211) (1.188) (1.456) (1.435) (0.523) (0.504) (0.619) (0.591) (0.517) (0.500) (0.614) (0.590) MENA Country 0.185 0.332 0.229 0.383 0.217 0.357 (0.285) (0.345) (0.198) (0.232) (0.196) (0.232) Constant -0.411 -0.599 -0.340 -0.950 -0.305 -0.767 -0.350 -1.162* -0.371 -0.795 -0.404 -1.147* 122 (0.443) (0.656) (0.533) (0.793) (0.479) (0.505) (0.568) (0.592) (0.482) (0.503) (0.573) (0.592) Observations 54 54 54 54 99 99 99 99 99 99 99 99 R-squared 0.198 0.181 0.182 0.158 0.121 0.122 0.082 0.101 0.129 0.127 0.085 0.098 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 123 VARIABLES Sustainabi lity Index (weighted ) (a) Sustainab ility Index (weighte d) (b) Sustaina bility Index (PCA) (a) Sustaina bility Index (PCA) (b) Sustaina bility Index (weighte d) (a) Sustaina bility Index (weight ed) (b) Sustain ability Index (PCA) (a) Sustain ability Index (PCA) (b) Sustain ability Index (weight ed) (a) Sustain ability Index (weight ed) (b) Sustainabi lity Index (PCA) (a) Sustainabil ity Index (PCA) (b) Number of local awards above Regional Average 0.316*** 0.308*** 0.195 0.177 (0.0999) (0.0990) (0.124) (0.121) Number of international awards above Regional Average 0.430*** 0.418** * 0.420** * 0.404** * (0.0990) (0.0979) (0.122) (0.119) Engaged in more activities than regional average (a) 0.0264 0.0013 1 -0.0170 -0.0392 (0.110) (0.103) (0.131) (0.121) Number of CSOs (per country) -0.00962 -0.00944 -0.00636 - 0.00743 - 0.00759 - 0.00750 - 0.0045 4 - 0.0053 6 - 0.0090 6 - 0.0099 2 -0.00616 -0.00817 (0.00802) (0.00775) (0.00992 ) (0.0094 9) (0.0076 8) (0.0074 5) (0.0094 4) (0.0090 6) (0.0084 8) (0.008 25) (0.0101) (0.00970) Number of NGOs (per country) 0.000005 02 - 0.000032 3 0.00015 5 0.00006 38 - 0.00016 2 - 0.00016 1 - 0.0000 131 - 0.0000 640 0.0000 0528 - 0.0000 247 0.000175 0.000092 9 (0.000394 ) (0.00039 0) (0.00048 7) (0.0004 78) (0.0003 79) (0.0003 76) (0.0004 66) (0.0004 57) (0.0004 21) (0.000 416) (0.000500 ) (0.000489 ) GDP Per-Capita 0.108* 0.124** 0.113 0.134* 0.0527 0.0706 0.0545 0.0759 0.120* 0.140* * 0.124* 0.147** 124 (0.0570) (0.0568) (0.0705) (0.0696) (0.0567) (0.0567) (0.0696 ) (0.0689 ) (0.0610 ) (0.060 2) (0.0724) (0.0708) Population (2009) - 0.000061 2 -0.00126 - 0.00076 5 - 0.00272 0.00141 0.00035 3 0.0003 61 - 0.0013 8 0.0009 39 - 0.0007 13 -0.000314 -0.00259 (0.00271) (0.00288) (0.00335 ) (0.0035 3) (0.0025 8) (0.0027 7) (0.0031 7) (0.0033 7) (0.0029 0) (0.003 07) (0.00344) (0.00361) Number of Implementers in￾country 0.153 0.156 0.00596 0.0058 2 0.178 0.165 (0.329) (0.407) (0.317) (0.389) (0.348) (0.413) IRI -0.0201 -0.115 -0.0326 -0.150 0.0302 -0.0683 (0.340) (0.421) (0.325) (0.400) (0.363) (0.431) NDI -0.158 -0.0930 -0.287 -0.228 -0.148 -0.0697 (0.363) (0.449) (0.349) (0.429) (0.388) (0.460) IFES 0.530 0.636 0.543 0.696 0.0228 0.0969 0.161 0.287 0.206 0.349 0.356 0.541 (0.445) (0.439) (0.551) (0.538) (0.418) (0.416) (0.513) (0.506) (0.460) (0.453) (0.547) (0.533) Asian Country -0.0669 -0.154 -0.179 -0.321 - 0.00464 -0.120 -0.133 -0.285 -0.0192 -0.162 -0.159 -0.342 (0.346) (0.351) (0.428) (0.430) (0.331) (0.337) (0.407) (0.410) (0.367) (0.373) (0.435) (0.438) Eurasian Country -0.356 -0.316 -0.565 -0.542 -0.436 -0.441 -0.642 -0.652 -0.343 -0.344 -0.578 -0.593 (0.426) (0.411) (0.527) (0.504) (0.408) (0.395) (0.501) (0.480) (0.455) (0.443) (0.540) (0.521) Latin America/Caribbean Country -0.292 -0.223 -0.151 -0.0342 -0.230 -0.188 -0.0715 0.0269 -0.336 -0.291 -0.188 -0.0981 (0.486) (0.470) (0.601) (0.575) (0.465) (0.450) (0.572) (0.548) (0.514) (0.498) (0.610) (0.586) MENA Country 0.183 0.340 0.143 0.286 0.225 0.360 (0.187) (0.230) (0.180) (0.219) (0.197) (0.232) 125 Constant 0.0267 -0.403 -0.129 -0.906 0.305 -0.111 0.260 -0.487 -0.298 -0.755 -0.333 -1.085* (0.465) (0.489) (0.575) (0.599) (0.455) (0.481) (0.559) (0.584) (0.479) (0.505) (0.568) (0.594) Observations 99 99 99 99 99 99 99 99 99 99 99 99 R-squared 0.212 0.208 0.103 0.116 0.279 0.272 0.190 0.199 0.121 0.121 0.078 0.095 v (a) If the organization reported engagement in a greater number of the following activities than the regional average: training of political party activists; conduction election observation; conducting advocacy campaigns; media training; public opinion polling and/or focus groups; parallel vote tabulation; training of election management bodies; political dialogue and/or debates; voter education campaigns; advocacy on elections/political party/CSO law. (b) If the organization had a greater number of the following items than the regional average: logo; mission statement; brochures/pamphlets; building/office/suite with functioning utilities; transportation vehicle; computer; website; NGO license/registration; budget; specialized staff; support staff. 126 (13) (14) (15) (16) (17) (18) (19) (20) VARIABLES Sustainability Index (weighted) (a) Sustainability Index (weighted) (b) Sustainabilit y Index (PCA) (a) Sustainabilit y Index (PCA) (b) Sustainability Index (weighted) (a) Sustainability Index (weighted) (b) Sustainabilit y Index (PCA) (a) Sustainabilit y Index (PCA) (b) Number of organizational items above regional average (b) 0.198* 0.196* 0.155 0.157 (0.102) (0.0995) (0.122) (0.118) CSO Has experience abroad -0.168 -0.154 -0.184 -0.175 (0.121) (0.115) (0.144) (0.136) Number of CSOs (per country) -0.00906 -0.00941 -0.00599 -0.00730 -0.0117 -0.0120 -0.00819 -0.00998 (0.00829) (0.00799) (0.00996) (0.00952) (0.00873) (0.00851) (0.0104) (0.0101) Number of NGOs (per country) 0.0000910 0.0000300 0.000220 0.000112 -0.0000930 -0.000113 0.0000433 -0.0000382 (0.000409) (0.000403) (0.000491) (0.000480) (0.000422) (0.000417) (0.000504) (0.000493) GDP Per-Capita 0.128** 0.149** 0.126* 0.150** 0.132** 0.147** 0.134* 0.153** (0.0588) (0.0585) (0.0707) (0.0696) (0.0602) (0.0599) (0.0719) (0.0708) Population (2009) 0.000987 -0.000799 -0.0000887 -0.00248 0.000235 -0.000926 -0.000751 -0.00272 (0.00279) (0.00297) (0.00335) (0.00353) (0.00285) (0.00305) (0.00341) (0.00361) Number of Implementers in￾country 0.200 0.190 0.241 0.239 (0.340) (0.409) (0.347) (0.415) IRI -0.0212 -0.127 0.161 0.0601 (0.353) (0.424) (0.371) (0.444) NDI -0.176 -0.111 -0.0614 0.00963 (0.376) (0.452) (0.387) (0.462) 127 IFES 0.110 0.272 0.266 0.468 0.165 0.293 0.292 0.471 (0.452) (0.445) (0.543) (0.529) (0.458) (0.453) (0.547) (0.536) Asian Country -0.0846 -0.222 -0.200 -0.373 -0.0297 -0.140 -0.142 -0.312 (0.359) (0.364) (0.431) (0.433) (0.368) (0.376) (0.440) (0.444) Eurasian Country -0.419 -0.380 -0.614 -0.587 -0.293 -0.285 -0.519 -0.553 (0.442) (0.424) (0.531) (0.505) (0.457) (0.449) (0.547) (0.531) Latin America/Caribbean Country -0.370 -0.278 -0.203 -0.0626 -0.434 -0.405 -0.259 -0.185 (0.502) (0.484) (0.603) (0.576) (0.518) (0.505) (0.619) (0.597) MENA Country 0.274 0.404* 0.164 0.327 (0.194) (0.231) (0.202) (0.239) Constant -0.288 -0.892* -0.321 -1.219** -0.287 -0.544 -0.349 -0.952 (0.469) (0.496) (0.563) (0.590) (0.490) (0.531) (0.585) (0.628) Observations 99 99 99 99 97 97 97 97 R-squared 0.157 0.158 0.094 0.112 0.143 0.136 0.094 0.106 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) If the organization reported engagement in a greater number of the following activities than the regional average: training of political party activists; conduction election observation; conducting advocacy campaigns; media training; public opinion polling and/or focus groups; parallel vote tabulation; training of election management bodies; political dialogue and/or debates; voter education campaigns; advocacy on elections/political party/CSO law. (b) If the organization had a greater number of the following items than the regional average: logo; mission statement; brochures/pamphlets; building/office/suite with functioning utilities; transportation vehicle; computer; website; NGO license/registration; budget; specialized staff; support staff. 128 Regression Tables for Research Question 4.2 (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) VARIABLES Sustain ability Index (weigh ted) Sustain ability Index (PCA) Sustain ability Index (weigh ted) Sustain ability Index (PCA) Sustain ability Index (weigh ted) Sustain ability Index (PCA) Sustain ability Index (weigh ted) Sustain ability Index (PCA) Sustain ability Index (weigh ted) Sustain ability Index (PCA) Sustain ability Index (weigh ted) Sustain ability Index (PCA) Uses Consistent Donor-Approved Overhead Rate 0.196 0.359 (0.161) (0.182) CEPPS3 * Uses Consistent Donor￾Approved Overhead Rate - 0.0908 -0.191 (0.212) (0.240) Uses a timesheet system 0.0049 7 -0.198 (0.141) (0.164) CEPPS3 * Uses a timesheet system - 0.0069 4 0.346 (0.205) (0.238) Received non-CEPPS international funding for latest election - 0.0911 -0.232 (0.165) (0.198) CEPPS3 * Received non-CEPPS international funding for latest election 0.0595 0.285 (0.214) (0.257) Has adopted CEPPS's capacity-building approaches -1.725 -2.359 129 (2.361) (2.848) CEPPS3 * Has adopted CEPPS's capacity-building approaches 1.633 2.235 (2.372) (2.861) Needs help with financial management -0.117 -0.102 (0.161) (0.188) CEPPS3 * Needs help with financial management 0.257 0.368 (0.216) (0.252) Needs help with fundraising 0.0404 0.155 (0.153) (0.181) CEPPS3 * Needs help with fundraising 0.101 -0.161 (0.221) (0.260) Number of CSOs (per country) - 0.0075 4 - 0.0059 8 - 0.0099 2 - 0.0080 4 - 0.0142 - 0.0161 - 0.0107 - 0.0092 5 - 0.0098 7 - 0.0076 8 - 0.0097 3 - 0.0075 0 (0.008 76) (0.009 92) (0.008 23) (0.009 56) (0.011 0) (0.013 2) (0.010 9) (0.013 1) (0.008 14) (0.009 51) (0.008 18) (0.009 62) Number of NGOs (per country) 0.0000 783 0.0002 63 - 0.0000 208 - 0.0000 610 - 0.0003 32 - 0.0002 34 0.0005 24 0.0003 67 - 0.0000 399 0.0000 518 - 0.0000 452 0.0001 02 (0.000 430) (0.000 487) (0.000 422) (0.000 491) (0.000 414) (0.000 497) (0.000 562) (0.000 678) (0.000 410) (0.000 480) (0.000 414) (0.000 488) GDP Per-Capita 0.0001 27* 0.0001 19 0.0001 40* 0.0001 51* 0.0001 56* 0.0001 63* 0.0001 62 0.0001 80 0.0001 39* 0.0001 38 0.0001 47* 0.0001 41* (0.000 0636) (0.000 0720) (0.000 0601) (0.000 0698) (0.000 0609) (0.000 0731) (0.000 0807) (0.000 0974) (0.000 0596) (0.000 0697) (0.000 0602) (0.000 0709) 130 Population (2009) - 0.0014 3 - 0.0041 3 - 0.0007 23 - 0.0023 6 0.0020 1 0.0006 98 - 0.0060 8 - 0.0064 7 - 0.0002 38 - 0.0016 3 - 0.0007 56 - 0.0021 9 (0.003 21) (0.003 63) (0.003 06) (0.003 56) (0.003 07) (0.003 69) (0.004 12) (0.004 97) (0.003 05) (0.003 57) (0.003 05) (0.003 59) Number of implementers in-country 0.283 0.503* 0.225 0.345 0.204 0.381 0.228 0.392 0.240 0.399 0.239 0.321 (0.212) (0.240) (0.201) (0.234) (0.228) (0.274) (0.294) (0.355) (0.198) (0.231) (0.203) (0.239) Asian Country 0.544 0.896 0.350 0.508 0.120 0.188 0.848 1.117 0.254 0.372 0.393 0.560 (0.489) (0.554) (0.457) (0.531) (0.493) (0.592) (0.668) (0.805) (0.459) (0.536) (0.456) (0.537) Eurasian Country - 0.0487 -0.121 -0.162 -0.326 -0.249 -0.449 0.0626 -0.110 -0.115 -0.218 -0.124 -0.228 (0.407) (0.461) (0.373) (0.433) (0.401) (0.481) (0.500) (0.603) (0.376) (0.440) (0.384) (0.452) Latin America/Caribbean Country -0.227 -0.254 -0.345 -0.507 -0.475 -0.848 -0.199 -0.590 -0.298 -0.453 -0.358 -0.527 (0.466) (0.528) (0.447) (0.520) (0.466) (0.559) (0.650) (0.784) (0.438) (0.512) (0.436) (0.513) MENA Country - 0.0823 0.289 -0.289 -0.182 -0.617 -0.328 -0.544 0.0016 0 -0.369 -0.125 -0.245 0.0234 (0.521) (0.590) (0.506) (0.588) (0.569) (0.683) (1.201) (1.449) (0.510) (0.595) (0.505) (0.594) Constant -0.959 - 1.496* -0.756 -1.065 -0.566 -0.939 -0.802 -1.227 -0.819 - 1.236* -0.849 -1.062 (0.536) (0.607) (0.518) (0.602) (0.529) (0.634) (0.723) (0.872) (0.505) (0.590) (0.518) (0.610) Observations 89 89 99 99 85 85 54 54 99 99 99 99 R-squared 0.166 0.189 0.129 0.126 0.206 0.190 0.200 0.203 0.146 0.129 0.139 0.113 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 131 (13) (14) (15) (16) (17) (18) (19) (20) (21) (22) VARIABLES Sustainabi lity Index (weighted ) Sustainab ility Index (PCA) Sustainabi lity Index (weighted ) Sustainab ility Index (PCA) Sustainabi lity Index (weighted ) Sustainab ility Index (PCA) Sustainabi lity Index (weighted ) Sustainab ility Index (PCA) Sustainabi lity Index (weighted ) Sustainab ility Index (PCA) Number of local awards above Regional Average 0.342** 0.257 (0.129) (0.157) CEPPS3 * Number of local awards above Regional Average -0.0654 -0.151 (0.156) (0.191) Number of international awards above Regional Average 0.408** 0.413* (0.131) (0.159) CEPPS3 * Number of international awards above Regional Average 0.0182 -0.0181 (0.164) (0.200) Engaged in more activities than regional average (a) 0.110 0.0144 (0.153) (0.180) CEPPS3 * Engaged in more activities than regional average (a) -0.198 -0.0975 (0.205) (0.242) Number of organizational items above regional average (b) 0.206 0.143 (0.146) (0.174) 132 CEPPS3 * Number of organizational items above regional average (b) -0.0187 0.0273 (0.206) (0.245) CSO Has experience abroad -0.0128 -0.0230 (0.169) (0.200) CEPPS3 * CSO Has experience abroad -0.241 -0.259 (0.211) (0.250) Number of CSOs (per country) -0.00934 -0.00721 -0.00748 -0.00538 -0.0105 -0.00843 -0.00930 -0.00747 -0.0111 -0.00893 (0.00779) (0.00952 ) (0.00749) (0.00911 ) (0.00827) (0.00977 ) (0.00814) (0.00969 ) (0.00854) (0.0101) Number of NGOs (per country) - 0.000017 5 0.000098 1 -0.000171 - 0.000054 5 - 0.000059 5 0.000075 7 0.000027 4 0.000116 -0.000141 - 0.000067 4 (0.000394 ) (0.00048 1) (0.000387 ) (0.00047 1) (0.000418 ) (0.00049 4) (0.000407 ) (0.00048 4) (0.000417 ) (0.00049 4) GDP Per-Capita 0.000123* 0.000131 0.000070 6 0.000075 9 0.000138* 0.000146 * 0.000149* 0.000149 * 0.000139* 0.000144 * (0.000057 1) (0.00006 98) (0.000057 0) (0.00006 93) (0.000060 3) (0.00007 12) (0.000059 0) (0.00007 03) (0.000060 3) (0.00007 13) Population (2009) -0.00140 -0.00306 0.000406 -0.00143 -0.000701 -0.00258 -0.000762 -0.00253 -0.000665 -0.00244 (0.00292) (0.00357 ) (0.00283) (0.00344 ) (0.00307) (0.00363 ) (0.00301) (0.00359 ) (0.00305) (0.00362 ) Number of implementers in￾country 0.193 0.364 0.140 0.289 0.233 0.364 0.274 0.403 0.182 0.346 (0.190) (0.232) (0.184) (0.223) (0.197) (0.233) (0.196) (0.233) (0.203) (0.240) Asian Country 0.648 0.724 0.0888 0.295 0.320 0.527 0.269 0.472 0.208 0.380 (0.442) (0.540) (0.425) (0.517) (0.455) (0.537) (0.448) (0.534) (0.459) (0.543) 133 Eurasian Country -0.138 -0.283 -0.121 -0.285 -0.155 -0.338 -0.220 -0.375 -0.181 -0.356 (0.355) (0.434) (0.339) (0.412) (0.373) (0.440) (0.366) (0.436) (0.377) (0.446) Latin America/Caribbean Country -0.285 -0.472 -0.447 -0.645 -0.314 -0.579 -0.373 -0.597 -0.257 -0.523 (0.420) (0.513) (0.401) (0.488) (0.444) (0.525) (0.433) (0.516) (0.449) (0.531) MENA Country -0.191 0.0403 -0.194 0.0329 -0.294 -0.0992 -0.273 -0.0693 -0.421 -0.203 (0.478) (0.584) (0.456) (0.555) (0.498) (0.589) (0.489) (0.582) (0.505) (0.597) Constant -0.434 -0.978 -0.102 -0.496 -0.734 -1.075 -0.898 -1.211* -0.547 -0.955 (0.497) (0.607) (0.490) (0.596) (0.506) (0.598) (0.503) (0.599) (0.530) (0.628) Observations 99 99 99 99 99 99 99 99 97 97 R-squared 0.213 0.131 0.272 0.208 0.138 0.106 0.166 0.121 0.163 0.134 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) If the organization reported engagement in a greater number of the following activities than the regional average: training of political party activists; conduction election observation; conducting advocacy campaigns; media training; public opinion polling and/or focus groups; parallel vote tabulation; training of election management bodies; political dialogue and/or debates; voter education campaigns; advocacy on elections/political party/CSO law. (b) If the organization had a greater number of the following items than the regional average: logo; mission statement; brochures/pamphlets; building/office/suite with functioning utilities; transportation vehicle; computer; website; NGO license/registration; budget; specialized staff; support staff. 134 Regression Tables for Research Question 5.1 (1) (2) (3) (4) Obtained Funds Obtained Funds Obtained Funds Obtained Funds CSO Involved in Design Process 0.305*** (0.0851) CSO Involved in Awards Management 0.378*** (0.0931) CSO Involved in M&E 0.216** (0.0906) CSO Involved in Reporting 0.129 (0.0871) GDP Per-Capita 0.00526 0.0292 0.00960 0.00562 (0.0235) (0.0238) (0.0243) (0.0243) ASIA dummy -0.00121 0.0318 -0.0778 -0.0573 (0.106) (0.112) (0.106) (0.111) EURASIA dummy 0.260** 0.194 0.257** 0.241* (0.121) (0.119) (0.123) (0.125) LAC dummy -0.136 -0.143 -0.223 -0.150 (0.202) (0.206) (0.205) (0.209) MENA dummy -0.124 -0.233 0.0355 0.00140 (0.216) (0.217) (0.213) (0.222) Constant 0.113 0.0562 0.180** 0.192** (0.0897) (0.0950) (0.0837) (0.0927) Observations 112 110 114 114 R-Squared 0.185 0.203 0.149 0.108 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 135 Regression for Research Question 5.2 Variables (a) (1) (2) (3) CEPPS3 made efforts to help CSO obtain funds… Obtained Funds Obtained Funds Obtained Funds …"to a great extent" 1.987*** (0.406) …"to some extent" 0.517* (0.283) …"to some or a great extent" 1.746*** (0.307) GDP Per-Capita 0.0226 -0.00620 -0.0644 (0.0843) (0.0744) (0.0907) ASIA dummy -0.478 -0.159 -0.155 (0.383) (0.336) (0.381) EURASIA dummy 0.546 0.608* 0.724* (0.396) (0.363) (0.423) LAC dummy -1.032 -0.532 -0.296 (0.826) (0.664) (0.829) MENA dummy -0.0348 0.421 0.499 (0.703) (0.658) (0.753) _cons -0.868*** -0.714*** -1.408*** (0.274) (0.256) (0.319) Observations 115 115 115 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 136 Regressions for Research Question 5.3 Sustainability Index (weighted) Sustainability Index (PCA) Sustainability Index (weighted) Sustainability Index (PCA) Sustainability Index (weighted) Sustainability Index (PCA) CSO Involved in design process -0.576 -0.503 -0.465 -0.541 -0.236 -0.477 (0.519) (0.541) (0.513) (0.533) (0.540) (0.561) NDI 0.0622 -0.111 0.276 0.159 (0.396) (0.412) (0.409) (0.425) IFES -0.224 -0.232 -0.242 -0.0194 (0.374) (0.390) (0.421) (0.438) IRI 0.257 0.340 0.00654 0.265 (0.370) (0.385) (0.403) (0.419) NDI * CSO Involved in design process 0.284 -0.105 0.174 -0.0671 (0.782) (0.815) (0.775) (0.806) IFES * CSO Involved in design process 0.107 -0.118 -0.237 -0.153 (0.750) (0.781) (0.771) (0.801) IRI * CSO Involved in design process -0.153 -0.0501 -0.383 -0.121 (0.737) (0.766) (0.762) (0.793) GDP Per-Capita 0.150* 0.166* 0.150* 0.165* 0.149* 0.165* (0.0884) (0.0921) (0.0882) (0.0917) (0.0883) (0.0918) ASIA dummy 0.528 0.538 0.486 0.494 0.490 0.456 (0.460) (0.479) (0.462) (0.480) (0.474) (0.493) EURASIA dummy 0.107 -0.0557 0.0650 -0.0928 0.0707 -0.131 (0.450) (0.469) (0.449) (0.467) (0.464) (0.482) 137 LAC dummy -0.433 -0.926 -0.465 -0.964 -0.427 -0.904 (0.737) (0.768) (0.738) (0.767) (0.725) (0.754) MENA dummy -0.525 -0.272 -0.554 -0.293 -0.549 -0.320 (1.325) (1.380) (1.322) (1.374) (1.325) (1.378) Constant -0.450 -0.429 -0.648* -0.679* -0.404 -0.583* (0.377) (0.393) (0.348) (0.362) (0.324) (0.337) Observations 61 61 61 61 61 61 R-squared 0.172 0.184 0.175 0.191 0.174 0.189 Standard errors in parentheses ="* p<0.1 ** p<0.05 *** p<0.01" 138 Variables Sustainability Index (weighted) Sustainability Index (PCA) Sustainability Index (weighted) Sustainability Index (PCA) Sustainability Index (weighted) Sustainability Index (PCA) CSO Involved in awards management process -0.146 -0.0171 0.192 0.189 -0.698 -0.799 (0.648) (0.678) (0.553) (0.575) (0.541) (0.566) NDI 0.126 -0.0966 0.417 0.258 (0.431) (0.451) (0.434) (0.451) IFES -0.341 -0.363 -0.396 -0.147 (0.389) (0.407) (0.441) (0.461) IRI 0.333 0.429 -0.0449 0.259 (0.388) (0.404) (0.437) (0.457) NDI * CSO Involved in awards management process -0.653 -0.924 -0.993 -1.136 (0.857) (0.896) (0.797) (0.829) IFES * CSO Involved in awards management process 0.377 0.187 0.931 0.971 (0.859) (0.898) (0.788) (0.823) IRI * CSO Involved in awards management process -0.364 -0.280 0.536 0.719 (0.857) (0.892) (0.847) (0.885) GDP Per-Capita 0.151 0.168* 0.153 0.168* 0.154 0.171* (0.0918) (0.0960) (0.0917) (0.0955) (0.0920) (0.0961) ASIA dummy 0.495 0.456 0.431 0.389 0.494 0.408 (0.489) (0.512) (0.490) (0.511) (0.504) (0.526) EURASIA dummy 0.105 -0.0632 0.0411 -0.132 0.0975 -0.121 (0.469) (0.490) (0.471) (0.490) (0.485) (0.507) 139 LAC dummy -0.707 -1.054 -0.748 -1.106 -0.644 -0.995 (0.769) (0.804) (0.771) (0.803) (0.761) (0.795) MENA dummy -0.916 -0.715 -0.968 -0.763 -0.929 -0.769 (1.388) (1.451) (1.387) (1.444) (1.394) (1.457) Constant -0.384 -0.345 -0.656* -0.671* -0.330 -0.529 (0.389) (0.407) (0.355) (0.369) (0.345) (0.361) Observations 59 59 59 59 59 59 R-squared 0.171 0.175 0.170 0.183 0.166 0.173 Standard errors in parentheses ="* p<0.1 140 Variables Sustainability Index (weighted) Sustainability Index (PCA) Sustainability Index (weighted) Sustainability Index (PCA) Sustainability Index (weighted) Sustainability Index (PCA) CSO involved in M&E process -0.780 -0.737 0.470 0.442 -0.0181 -0.178 (0.565) (0.601) (0.597) (0.632) (0.543) (0.575) NDI -0.104 -0.262 0.191 0.0492 (0.420) (0.447) (0.425) (0.450) IFES -0.321 -0.312 -0.161 0.0549 (0.379) (0.403) (0.431) (0.456) IRI 0.370 0.417 0.204 0.441 (0.380) (0.402) (0.429) (0.454) NDI * CSO involved in M&E process 0.704 0.446 -0.526 -0.710 (0.791) (0.842) (0.814) (0.862) IFES * CSO involved in M&E process 1.238 1.126 0.483 0.579 (0.830) (0.884) (0.813) (0.862) IRI * CSO involved in M&E process -1.315 -1.284 -0.828 -0.672 (0.831) (0.880) (0.791) (0.838) GDP Per-Capita 0.185** 0.211** 0.187** 0.211** 0.187** 0.213** (0.0889) (0.0946) (0.0884) (0.0936) (0.0886) (0.0939) ASIA dummy 0.227 0.180 0.146 0.0816 0.149 0.0493 (0.477) (0.508) (0.482) (0.511) (0.491) (0.520) EURASIA dummy 0.0413 -0.0931 -0.0290 -0.170 -0.0241 -0.201 (0.470) (0.500) (0.471) (0.498) (0.480) (0.508) LAC dummy -0.524 -0.942 -0.578 -1.010 -0.558 -0.992 (0.698) (0.743) (0.698) (0.738) (0.697) (0.739) MENA dummy -0.868 -0.742 -0.919 -0.758 -0.920 -0.829 141 (1.303) (1.387) (1.290) (1.366) (1.301) (1.378) Constant -0.363 -0.399 -0.636* -0.683* -0.475 -0.687* (0.413) (0.440) (0.378) (0.400) (0.334) (0.354) Observations 59 59 59 59 59 59 R-squared 0.188 0.177 0.196 0.194 0.194 0.190 Standard errors in parentheses ="* p<0.1 142 Sustainability Index (weighted) Sustainability Index (PCA) Sustainability Index (weighted) Sustainability Index (PCA) Sustainability Index (weighted) Sustainability Index (PCA) CSO involved in the reporting process 0.227 -0.189 0.694 0.850 -0.668 -1.136** (0.503) (0.516) (0.555) (0.561) (0.506) (0.519) NDI -0.0441 -0.146 0.415 0.313 (0.391) (0.401) (0.405) (0.410) IFES -0.581 -0.546 -0.469 -0.249 (0.384) (0.394) (0.425) (0.436) IRI 0.525 0.600 0.123 0.362 (0.380) (0.384) (0.407) (0.418) NDI * CSO involved in the reporting process -1.011 -1.154 -1.489* -2.216*** (0.726) (0.745) (0.793) (0.803) IFES * CSO involved in the reporting process 0.625 1.091 1.519* 2.041** (0.781) (0.800) (0.799) (0.820) IRI * CSO involved in the reporting process -0.506 -1.167 0.867 0.827 (0.765) (0.774) (0.712) (0.730) GDP Per-Capita 0.207** 0.260*** 0.208** 0.265*** 0.207** 0.261*** (0.0884) (0.0907) (0.0890) (0.0901) (0.0889) (0.0912) ASIA dummy 0.313 0.160 0.190 -0.0234 0.318 0.0910 (0.451) (0.463) (0.463) (0.468) (0.473) (0.485) EURASIA dummy -0.144 -0.367 -0.232 -0.527 -0.138 -0.428 (0.466) (0.478) (0.483) (0.489) (0.484) (0.497) 143 LAC dummy -0.847 -1.440** -0.914 -1.580** -0.797 -1.412* (0.699) (0.716) (0.712) (0.720) (0.701) (0.719) MENA dummy -0.584 -0.576 -0.815 -0.772 -0.578 -0.627 (1.298) (1.331) (1.277) (1.292) (1.307) (1.341) Constant -0.312 -0.384 -0.736** -0.791** -0.427 -0.641* (0.379) (0.388) (0.351) (0.356) (0.321) (0.329) Observations 60 60 60 60 60 60 R-squared 0.213 0.260 0.207 0.273 0.208 0.254 144 F-Tests for Question 5.3 Regressions Variable Sustainability Index Type Implementer 1 Test 𝛿𝛿 𝑚𝑚 𝑚𝑚 Test 𝛿𝛿 𝑚𝑚𝑚𝑚 Implementer 2 Test 𝛿𝛿 𝑚𝑚 𝑚𝑚 Test 𝛿𝛿 𝑚𝑚𝑚𝑚 CSO Involved in Design Weighted Average NDI 0.237 0.411 IFES 0.164 0.472 CSO Involved in Design PCA NDI 0.176 0.638 IFES 0.165 0.639 CSO Involved in Award Management Weighted Average NDI 0.461 0.797 IFES 0.96 0.729 CSO Involved in Award Management PCA NDI 0.533 0.614 IFES 0.932 0.906 CSO Involved in M&E Weighted Average NDI 0.197 0.223 IFES 0.378 0.124 CSO Involved in M&E PCA NDI 0.193 0.343 IFES 0.416 0.18 CSO Involved in the Reporting Process Weighted Average NDI 0.593 0.341 IFES 0.208 0.878 CSO Involved in the Reporting Process PCA NDI 0.084* 0.584 IFES 0.492 0.36 CSO Involved in Design Weighted Average NDI 0.308 0.524 IRI 0.161 0.668 CSO Involved in Design PCA NDI 0.157 0.587 IRI 0.151 0.574 CSO Involved in Award Management Weighted Average NDI 0.76 0.394 IRI 0.863 0.671 CSO Involved in Award Management PCA NDI 0.684 0.365 IRI 0.827 0.725 CSO Involved in M&E Weighted Average NDI 0.5 0.444 IRI 0.917 0.213 CSO Involved in M&E PCA NDI 0.65 0.419 IRI 0.951 0.256 CSO Involved in the Reporting Process Weighted Average NDI 0.87 0.106 IRI 0.22 0.293 CSO Involved in the Reporting Process PCA NDI 0.819 0.03** IRI 0.369 0.111 CSO Involved in Design Weighted Average IFES 0.386 0.915 IRI 0.304 0.989 CSO Involved in Design PCA IFES 0.181 0.687 IRI 0.189 0.671 CSO Involved in Award Management Weighted Average IFES 0.336 0.185 IRI 0.224 0.297 CSO Involved in Award Management PCA IFES 0.261 0.165 IRI 0.209 0.227 CSO Involved in M&E Weighted Average IFES 0.742 0.731 IRI 0.49 0.587 CSO Involved in M&E PCA IFES 0.985 0.593 IRI 0.368 0.793 CSO Involved in the Reporting Process Weighted Average IFES 0.95 0.086* IRI 0.423 0.171 CSO Involved in the Reporting Process PCA IFES 0.491 0.014** IRI 0.045** 0.068* 145 ANNEX IX: SUPPLEMENTAL TABLES AND ANALYSIS FOR SELECTED RESULTS EQ 1.1. Are there more CSOs active now in elections assistance and political processes than six years ago? Table 4.1.3. Regressions for number of CSOs between 2009 and 2014 Variables Number of CSOs in only Electoral Processes (a) Number of CSOs in only Electoral Processes (Country Fixed-Effects) Number of CSOs in Political Processes (a) Number of CSOs in Political Processes (Country Fixed￾effects) Number of CSOs in electoral/poli￾tical processes (a) Number of CSOs in electoral/political Processes (Country Fixed-effects) Year 0.624* 1.563*** 0.646 1.727*** -0.688*** 0.715*** (0.335) (0.340) (0.404) (0.414) (0.261) (0.192) Number of CEPPS Electoral CSOs 1.011*** 0.312 (0.117) (0.266) Number of CEPPS Political CSOs 1.252*** 1.268*** (0.145) (0.296) Number of CEPPS grantees 0.801*** 0.100* (0.0851) (0.0539) Constant -2.329*** 5.615*** -2.353** 4.251*** -0.237 3.501*** (0.816) (1.292) (0.967) (1.315) (0.557) (0.443) Observations 95 95 95 95 95 95 R-squared 0.886 0.981 0.857 0.981 0.866 0.990 Notes: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. (a) Covariates include GDP per-capita; Economist Intelligence Unit democracy index (free; not-free; partially free); number of NGOs per country, per year; whether or not IFES, NDI, or IRI implemented in-country; whether or not they had an election the previous year; average age of CSOs in-country. An important caveat here is that the total number of CSOs remains constant over time. Table 4.1.4. Regressions for the average age of CSOs between 2009 and 2014 146 CSOs Electorally focused CSOs Politically focused CSOs both electorally and politically focused VARIABLES Average CSO Age (a) Average CSO Age, (Country Fixed￾Effects) Average CSO Age (a) Average CSO Age (Country Fixed￾Effects) Average CSO Age (a) Average CSO Age (Country Fixed￾Effects) Year 1.519 3.811*** -0.316 4.879*** -1.467 5.209*** (2.434) (0.698) (2.386) (0.812) (2.925) (0.822) Number of CEPPS Electoral CSOs 2.933*** -2.590*** (0.789) (0.593) Number of CEPPS Political CSOs 3.602*** -2.516*** (0.762) (0.691) Number of CEPPS grantees 2.280** -1.098*** (0.923) (0.276) Constant 10.86* 18.78*** 10.66* 18.44*** 11.82* 14.62*** (5.828) (1.872) (5.583) (2.047) (6.120) (1.390) Observations 95 95 95 95 95 95 R-squared 0.238 0.983 0.300 0.981 0.172 0.982 Notes: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. (a) Covariates include GDP per-capita; Economist Intelligence Unit democracy index (free; not-free; partially free); number of NGOs per country, per year; whether or not IFES, NDI, or IRI implemented in-country; whether or not they had an election the previous year; average age of CSOs in-country. An important caveat here is that the total number of CSOs remains constant over time. Table 4.1.4.A Regressions for the average age of CSOs between 2009 and 2014 CSOs Electorally focused CSOs Politically focused CSOs both electorally and politically focused VARIABLES Average CSO Age (a) Average CSO Age, (Country Fixed￾Effects) Average CSO Age (a) Average CSO Age (Country Fixed￾Effects) Average CSO Age (a) Average CSO Age (Country Fixed￾Effects) Year 1.519 3.811*** -0.316 4.879*** -1.467 5.209*** (2.434) (0.698) (2.386) (0.812) (2.925) (0.822) 147 Number of CEPPS Electoral CSOs 2.933*** -2.590*** (0.789) (0.593) Number of CEPPS Political CSOs 3.602*** -2.516*** (0.762) (0.691) Number of CEPPS grantees 2.280** -1.098*** (0.923) (0.276) Constant 10.86* 18.78*** 10.66* 18.44*** 11.82* 14.62*** (5.828) (1.872) (5.583) (2.047) (6.120) (1.390) Observations 95 95 95 95 95 95 R-squared 0.238 0.983 0.300 0.981 0.172 0.982 Notes: Standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. (a) Covariates include GDP per-capita; Economist Intelligence Unit democracy index (free; not-free; partially free); number of NGOs per country, per year; whether or not IFES, NDI, or IRI implemented in-country; whether or not they had an election the previous year; average age of CSOs in-country. An important caveat here is that the total number of CSOs remains constant over time. 148 EQ. 2.2 Was CEPPS’s process for selecting local partner organizations optimal in the sense that competitive selection processes were balanced against the need for establishing long relationships? Table 4.2.10. Average Share of Contracts and Grants Average Share of Contracts Country source CEPPS2 CEPPS3 of support N Mean N Mean Change Pr > F (a) Total Sample 73 0.29 155 0.38 0.09 0.151 IRI 50 0.27 100 0.38 0.11 0.169 NDI 67 0.29 142 0.38 0.09 0.149 IFES 62 0.28 127 0.41 0.13* 0.062 Average Share of Grants CEPPS2 CEPPS3 N Mean N Mean Change Pr > F (a) Total Sample 73 0.7 155 0.62 -0.08 0.196 IRI 50 0.71 100 0.62 -0.09 0.229 NDI 67 0.7 142 0.62 -0.08 0.196 IFES 62 0.71 127 0.59 -0.12* 0.088 Note: Countries are typically being supported by more than one CEPPS implementer at a time. (a) F-test to test if the proportions of CSOs are equal between CEPPS2 and CEPPS3 149 Table 4.2.11. Average Share of Contracts and Grants, by region Average Share of Contracts CEPPS2 CEPPS3 Region N Mean N Mean Change Pr > F (a) Africa 23 0.36 23 0.33 -0.03 0.749 Asia 10 0.27 10 0.65 0.38 0.022** Eurasia 25 0.19 25 0.31 0.12 0.244 Latin America/Caribbean 9 0.31 9 0.33 0.02 0.926 MENA 6 0.44 6 0.25 -0.19 0.387 Average Share of Grants CEPPS2 CEPPS3 Region N Mean N Mean Change Pr > F (a) Africa 23 0.61 23 0.67 0.06 0.559 Asia 10 0.73 10 0.35 -0.38 0.022** Eurasia 25 0.81 25 0.69 -0.12 0.244 Latin America/Caribbean 9 0.69 9 0.67 -0.02 0.926 MENA 6 0.56 6 0.75 0.19 0.387 (a) F-test to test if the proportions of CSOs are equal between CEPPS2 and CEPPS3 Table 4.2.12. Average Share of Competitive Awards Country source CEPPS2 CEPPS3 of support N Mean N Mean Change Pr > F (a) Total Sample 66 0.29 320 0.27 -0.02 0.71 IRI 45 0.3 229 0.23 -0.07 0.15 NDI 61 0.27 305 0.27 0.0 0.86 IFES 56 0.3 268 0.30 0.0 0.91 Note: Countries are typically being supported by more than one CEPPS implementer at a time. Looking at the global trends presented in Table 4.2.12, we see negligible changes in the average shares of reported competitive awards for CSOs. There are slight downward trends for the entire sample and the IRI-supported countries, but neither changes are statistically significant. NDI- and IFES-supported countries showed no changes at all between CEPPS II and 150 CEPPS III, and even those were not statistically significant. Table 4.2.13 below breaks down the results by region. Table 4.2.13. Average Share of Competitive Awards CEPPS2 CEPPS3 Region N Mean N Mean Change Pr > F (a) Africa 20 0.3 106 0.28 -0.02 0.75 Asia 10 0.3 67 0.5 0.2** 0.04 Eurasia 23 0.26 90 0.21 -0.05 0.43 Latin America/Caribbean 8 0.25 28 0.12 -0.13 0.29 MENA 5 0.4 29 0.05 -0.35** 0.00 (a) F-test to test if the proportions of CSOs are equal between CEPPS2 and CEPPS3 Looking at the regional trends in Table 4.2.13, we see a little more variation in average competitive award shares. Africa shows a small downward trend, but it is not statistically significant. The only statistically significant trends are for Asia and MENA. Asia shows an increase in average share of competitive awards from 30% to 50% between CEPPS2 and CEPPS3. MENA shows a sharp drop, from 40% for CEPPS2 to 5% for CEPPS3. For the other regions, there are downward trends for Eurasia and Latin America/Caribbean, though they are not statistically significant. Table 4.2.14. AOR Opinions on the competitive selection process Response Option Total % of total Optimal balance 5 23.8% Fairly good balance 16 76.2% Total: 21 100% In the actual selection of CSOs to participate in the CEPPS III program, how appropriate do you believe that the balance between the need for establishing stable continuous partnerships and ensuring an objective and competitive selection process was? [AOR survey] Table 4.2.15. AOR Opinions on the importance of selection criteria Important Not important Quality of CSO technical skills 90.5% 9.5% Quality of CSO management 95.2% 4.8% Past performance 76.2% 23.8% Length of existence 61.9% 38.1% Past experience with CEPPS or USAID 61.9% 38.1% Name recognition 61.9% 38.1% Familiarity with main actors in the field 85.7% 14.3% Cost 57.1% 42.9% How important were each of the following in selecting a local CSO? [N=21] [AOR survey] 151 Table 4.2.16. Implementer Opinions on the competitive selection process Response Option Total % of total Leaned towards establishing ongoing relationships 50 44.2% Leaned towards competitive selection process 9 8.0% Optimal balance 46 40.7% Do not know 6 5.3% Skipped 2 1.8% Total: 113 100% In the actual selection of local CSOs to participate in the CEPPS program in [country], what was the balance between the two above objectives? [Implementer survey] 152 Table 4.2.17. Logit Regressions of Being a CEPPS2 Grantee and Engaging in a CEPPS3 Competitive Selection Process. Dependent Variable: CEPPS3 Competitive Process Explanatory Variables Covariates(b) Country FE Covariates(b) Country FE CEPPS2 Grantee 0.328 -0.227 18.99 16.57 (0.225) (0.299) (1.571) (1.060) IRI 0.312 0.194 (0.351) (1.600) NDI 15.67 13.12 (1,571) (1,060) IFES 2.580** -0.0857 (1.021) (2.157) NDI * CEPPS2 -16.70 -14.51 (1,571) (1,060) IFES * CEPPS2 -2.253* -1.542 (1.159) (1.451) IRI * CEPPS2 -0.116 -1.144 (0.521) (0.795) Constant -1.825*** -1.500 -20.01 -14.93 (0.152) (1.105) (1,571) (1,060) Observations 606 444 606 444 Prob > chi2 (a) 0.286 0.408 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Group test that NDI*CEPPS2; IFES*CEPPS2; and IRI*CEPPS2 are jointly equivalent to zero. (b) See footnote 4 in the main text, under regressions for evaluation question 1.1. The dependent variable is a dummy variable equal to one if funding for a CSO’s award was obtained through competitive procurement during the CEPPS3 period of 2009 - 2014, and zero otherwise. The explanatory variable of interest is a dummy for whether or not the CEPPS III awardee had received a contract or grant during CEPPS II (from 2005 to 2008). In effect, the logistic regression attempts to estimate the relationship between a CSO participating in CEPPS II, and then later undergoing a competitive selection process during CEPPS III. Table 4.2.17 presents the logit regression results for the first round of regressions. High-level summary statistics for AOR and implementer opinions on the competitive selection processes in CEPPS III countries can be found in the Annex in Tables 4.2.14 – 4.2.16. The units of the coefficients in Table 4.2.17 are presented as log-odds ratios – the likelihood of being a CEPPS2 grantee in a competitive procurement process during the CEPPS3 period. In effect, a positive sign on the CEPPS2 Grantee coefficient means CEPPS II grantees were more likely to engage in competitive procurement processes during CEPPS III, while a negative sign on the coefficient would indicate that they were less likely to engage in competitive procurement. In the first specification – run￾ning a simple logit with no covariates – we see a positive relationship between being a CEPPS2 grantee and engaging in a competitive CEPPS3 procurement process. However, when adding country fixed￾effects to the logit, the sign of the coefficient changes to negative. Neither coefficients are statistically significant. 153 The second set of regressions in Table 4.2.17 adds implementer dummy variables for whether or not a CSO was active in an implementer’s country, along with interaction terms between the implementer dummies and the CEPPS II grantee variable. As you can see in the right-hand columns, other things being equal, IFES is more likely to run competitive procurements than the other implementers. However, once a CEPPS II CSO becomes involved, that distinction disappears and, as is the case for NDI and IRI, there appears to be no relationship between being a CEPPS II grantee and engaging in a competitive procurement process during the CEPPS III period. Table 4.2.18. Logit regressions, relationship between the number of CEPPS2 awards and engaging in a CEPPS3 competitive selection process Dependent Variable: CEPPS3 Competitive Process Explanatory Variables Covariates(b) Country FE Number CEPPS2 Awards = 1 -0.000561 -0.000929 (0.00123) (0.00185) Number CEPPS2 Awards > 1 18.72 16.26 (1,386) (935.3) IRI 0.312 0.157 (0.351) (1.600) NDI 15.42 12.87 (1,386) (935.3) IFES 2.580** -0.125 (1.021) (2.157) NDI * CEPPS2 -16.44 -14.26 (1,386) (935.3) IFES * CEPPS2 -2.234* -1.477 (1.159) (1.452) IRI * CEPPS2 -0.0951 -1.078 (0.522) (0.795) Constant -19.76 -14.64 (1,386) (935.3) Observations 606 444 Prob > chi2 (a) 0.293 0.458 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Group test that NDI*CEPPS2; IFES*CEPPS2; and IRI*CEPPS2 are jointly equivalent to zero. (b) See (a) in table notes to Table 4.2.3, for Evaluation Question 1.1. Table 4.2.18 presents the results of another series of logit regressions, attempting to mea￾sure the relationship between the number of CEPPS2 awards and whether or not a CSO engaged in a CEPPS3 competitive selection process. The dependent variable is the same dummy variable from the above regressions, but now we include two explanatory dummies: a dummy that equals one if the CSO received only one CEPPS2 award, and zero otherwise; a dummy that equals one of the CSO received more than one award, and zero otherwise. Both sets of regressions include the implementer dummies and interaction terms. 154 There is no statistically significant relationship between a CSO receiving only one CEPPS2 award and engaging in a competitive selection process. The behavior of IFES described in the first set of regressions remains statistically significant here as well. EQ 2.3. To what degree did receiving a CEPPS grant in 2009 change the probability of receiving an award in 2014, all else held equal? Table 4.2.20. AOR opinions on whether CEPPS3 CSOs participated in CEPPS2 Response Option Total % of total Yes 15 71.4% No 4 19.0% Do not know 2 9.5% Total: 21 100% Notes: Question refers to “To the best of your knowledge, were any of the CSOs selected active with CEPPS programming in the past?” [AOR survey, Q2_6e] 155 Table 4.2.21. Logit regressions of being a CEPPS2 awardee and being a CEPPS3 awardee Dependent Variable: CEPPS3 Awardee Explanatory Variables Covariates(b) Country FE Covariates(b) Country FE CEPPS2 Grantee 2.526*** 2.905*** (0.363) (0.441) CEPPS2 Grantee (de-meaned) 2.426 3.456* (1.578) (2.070) IRI -0.609** 0.462 (0.296) (1.788) NDI -0.239 -0.270 (0.505) (1.567) IFES -0.185 1.190 (0.363) (1.489) NDI * CEPPS2 0.881 2.159 (1.330) (1.691) IFES * CEPPS2 -0.496 -1.046 (1.208) (1.526) IRI * CEPPS2 -0.354 -1.906 (0.912) (1.475) Constant -0.654*** -0.564 0.719 -1.341 (0.129) (1.138) (0.557) (1.753) Observations 344 313 344 313 Prob > chi2 (a) 0.907 0.434 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Group test that NDI*CEPPS2; IFES*CEPPS2; and IRI*CEPPS2 are jointly equivalent to zero. (b) See (a) in table notes to Table 4.2.3, for Evaluation Question 1 For the last two specifications, we also ran F-tests for joint significance for each interaction term. Each test tested for the following condition: 𝛽𝛽𝑖𝑖 𝑇𝑇 = 𝛽𝛽𝑖𝑖 𝑇𝑇∗𝐼𝐼 ∗ 𝛿𝛿𝑚𝑚 𝑚𝑚,𝑚𝑚𝑚𝑚 where 𝛽𝛽𝑖𝑖 𝑇𝑇 is the coefficient on the standardized treatment variable (the de-meaned standardized version“CEPPS2 Grantee”), and 𝛽𝛽𝑖𝑖 𝑇𝑇∗𝐼𝐼 is the coefficient on the interaction term between the treatment variable and the implementer dummy. Each test was run two times, with 𝛿𝛿 taking the value of either the minimum of the standardized treatment variable or the maximum. By running these tests we are able to determine if there is a statistically significant relationship between an implementer’s presence and the treatment of interest. In other words, the test using the maximum value of the treatment (𝛿𝛿 𝑚𝑚 ) would tell us if there’s a relationship when both the implementer and the treatment are present, while the test using the minimum value would tell us if there’s an effect with the implementer present, but not the treatment. 156 The results are summarized in Table 4.2.21 below. We found statistically significance for the NDI interaction terms where the treatment was not present. The NDI terms were significant at the 10% level in the logit regression with covariates, and at the 5% level in the logit with country fixed-effects. For the IRI interaction terms in the fixed-effects model, we were able to detect an effect at the 10% level in cases where the treatment was present along with IRI, but not in the covariates model. Finally, we detected statistically significant effects with the IFES terms, at the 10% level, in cases where the treatment was not present. Table 4.2.22. F-Test Results for Interaction Terms Covariates Fixed-Effects Test with 𝜹𝜹𝒎𝒎𝒎𝒎𝒎𝒎, Test with 𝜹𝜹𝒎𝒎𝒎𝒎𝒎𝒎 Test with 𝜹𝜹𝒎𝒎𝒎𝒎𝒎𝒎, Test with 𝜹𝜹𝒎𝒎𝒎𝒎𝒎𝒎 NDI 0.072* 0.447 NDI 0.045** 0.534 IRI 0.121 0.169 IRI 0.111 0.086* IFES 0.108 0.212 IFES 0.095* 0.133 The dependent variable in our analysis is a dummy variable equal to one if the CSO was a CEPPS3 grantee. The explanatory variable of interest is a dummy which equals one if the CSO was a CEPPS2 grantee. For the last two specifications, we also ran F-tests for joint significance for each interaction term. Each test tested for the following condition: 𝛽𝛽𝑖𝑖 𝑇𝑇 = 𝛽𝛽𝑖𝑖 𝑇𝑇∗𝐼𝐼 ∗ 𝛿𝛿𝑚𝑚 𝑚𝑚,𝑚𝑚𝑚𝑚 where 𝛽𝛽𝑖𝑖 𝑇𝑇 is the coefficient on the standardized treatment variable (the de-meaned standardized ver￾sion“CEPPS2 Grantee”), and 𝛽𝛽𝑖𝑖 𝑇𝑇∗𝐼𝐼 is the coefficient on the interaction term between the treatment vari￾able and the implementer dummy. Each test was run two times, with 𝛿𝛿 taking the value of either the minimum of the standardized treatment variable or the maximum. By running these tests we are able to determine if there is a statistically significant relationship between an implementer’s presence and the treatment of interest. In other words, the test using the maximum value of the treatment (𝛿𝛿 𝑚𝑚𝑚𝑚 ) would tell us if there’s a relationship when both the implementer and the treatment are present, while the test using the minimum value would tell us if there’s an effect with the implementer present, but not the treatment. The results are summarized in Table 4.2.21 below. We found statistical significance for the NDI interaction terms where the treatment was not present. The NDI terms were significant at the 10% level in the logit regression with covariates, and at the 5% level in the logit with country fixed-effects. For the IRI interaction terms in the fixed-effects model, we were able to detect an effect at the 10% level in cases where the treatment was present along with IRI, but not in the covariates model. Finally, we detected statistically significant effects with the IFES terms, at the 10% level, in cases where the treatment was not present. Table 4.2.22. F-Test Results for Interaction Terms Covariates Fixed-Effects Test with 𝜹𝜹𝒎𝒎𝒎𝒎𝒎𝒎, Test with 𝜹𝜹𝒎𝒎𝒎𝒎𝒎𝒎 Test with 𝜹𝜹𝒎𝒎𝒎𝒎𝒎𝒎, Test with 𝜹𝜹𝒎𝒎𝒎𝒎𝒎𝒎 157 NDI 0.072* 0.447 NDI 0.045** 0.534 IRI 0.121 0.169 IRI 0.111 0.086* IFES 0.108 0.212 IFES 0.095* 0.133 EQ3.1: To what degree did CEPPS involve local partners in the design, awards management, M&E processes, and reporting of programs with which they were affiliated? Table 4.3.1. Share of CSOs reporting implementer decided on programmatic activities Implementer decided which activities CSO would undertake (a) No Yes Share Implementer Decided (b) Share != 0 (c) Share != 0.5 (c) Share != 1 (c) Total Sample 138 22 0.14** 0 0 0 Africa 45 9 0.17** 0.002 0 0 Asia 33 9 0.21** 0.002 0 0 Eurasia 35 2 0.05 0.16 0 0 Latin America/Caribbean 11 1 0.08 0.339 0 0 MENA 14 1 0.07 0.334 0 0 (a) Reported as number of CSOs per region (b) Asterisks indicate that the share is statistically significant for all three hypothesis tests at the 10% (*) or 5% (**) level. (c) P-value in tests that the share of CSOs where the implementer decided on programming decisions is equal to 0, 50%, or 100%, respectively Table 4.3.2. Share of CSOs reporting implementer decided on programmatic activities, by implementer-supported countries Share of CSOs where implementer decided N Share Total Sample 141 0.15 IFES 40 0.18 IRI 61 0.15 NDI 40 0.13 Prob > F (a) 0.825 (a) F test to check if the proportions, per column, are equal across all regions. 158 Table 4.3.3. Share of CSOs reporting implementer involved them “to a great extent” in a programmatic activity. Implementer involved CSO in program activities to a high extent (a) No Yes Share CSO involved to a high extent (b) Share ≠ 0 (c) Share ≠ 0.5 (c) Share ≠ 1 (c) Design Total Sample 93 66 0.42** 0 0.032 0 AFR 30 24 0.44 0 0.419 0 ASIA 30 12 0.29** 0 0.004 0 EURASIA 23 14 0.38 0 0.141 0 LAC 6 5 0.45 0.016 0.779 0.006 MENA 4 11 0.73* 0 0.068 0.041 Awards Management Total Sample 105 52 0.33** 0 0 0 AFR 30 25 0.45 0 0.505 0 ASIA 34 5 0.13** 0.023 0 0 EURASIA 25 13 0.34** 0 0.05 0 LAC 9 1 0.1 0.343 0.003 0 MENA 7 8 0.53 0.001 0.806 0.004 M&E Processes Total Sample 102 51 0.33** 0 0 0 AFR 33 18 0.35** 0 0.034 0 ASIA 31 10 0.24** 0.001 0.001 0 EURASIA 25 11 0.31** 0 0.017 0 LAC 6 4 0.4 0.037 0.555 0.005 MENA 7 8 0.53 0.001 0.806 0.004 Reporting Total Sample 88 68 0.44 0 0.11 0 AFR 24 28 0.54 0 0.584 0 ASIA 31 12 0.28** 0 0.003 0 EURASIA 18 19 0.51 0 0.872 0 LAC 7 3 0.3 0.081 0.223 0.001 MENA 8 6 0.43 0.008 0.612 0.001 (a) Reported as number of CSOs per region 159 Table 4.3.4. Share of CSOs reporting that the implementer involved them “to a great extent” in a programmatic activity, by implementer-supported country. Share of CSOs… …where CSO involved in the design process …where CSO involved in awards management …where CSO involved in M&E processes …where CSO involved in reporting N Share N Share N Share N Share IFES 39 0.41 41 0.32 39 0.31 41 0.49 IRI 61 0.41 57 0.26 59 0.31 58 0.29 NDI 40 0.53 40 0.43 40 0.4 39 0.59 Prob > F (a) 0.617 0.327 0.523 0.175 (a) F test to check if the proportions, per column, are equal across all regions. EQ3.2: Did CEPPS incorporate CSO organizational capacity building as an integral part of collaboration or undertake it as a separate task? Table 4.3.7. Degree that program involved building capacity of local CSOs, Implementer Survey How much of this program involved building capacity of local CSOs? [Implementer survey] Response Option Total % of total A large part of program activities 25 21.0% A substantial part of program activities 54 45.4% A small part of program activities 33 27.7% No capacity building of CSOs 3 2.5% Do not know 4 3.4% Total: 119 100% Table 4.3.6 and 4.3.7 show the extent to which implementers believed the programs involved building capacity of local CSOs. According to Table 4.3.6, implementers responded that 78.2% of the programs had capacity-building as an explicit part of the program. Table 4.3.7 shows that 66.4% of implementers’ programs involved a substantial- or large-portion of capacity building activities – 45.4% reporting substantial and 21% reporting large. Only 27.7% of implementers’ programs were rated as having a small part capacity-building, and 2.5% of the programs included zero capacity building, according to implementers. 160 Table 4.3.8. Was organizational capacity-building integrated into programs, or taken as a separate parallel track, Implementer Survey Did [implementer] integrate organizational capacity-building activities into your collaborations on elections and political processes, or did the activities occur on a separate parallel track? [Implementer survey] Response Option Total % of total Integrated capacity building 88 73.9% Parallel track 18 15.1% Do not know 13 10.9% Total: 119 100% Table 4.3.8 gets directly to the question of whether capacity building was integrated into the collaborations on elections and political processes. 73.9% of the implementers reported integrating organizational capacity-building activities into their collaborations on elections and political processes, while only 15.1% of implementers reported that the activities occurred on a separate parallel track. CSOs Receiving Joint Work Below we present a series of tables summarizing the share of CSOs that received assistance in the form of joint work. We present summary statistics of the entire sample first for organizational capacity building, and again for technical capacity-building, and then disaggregated by implementer-supported countries, followed by a region-level analysis. Table 4.3.6. Share of CSOs receiving joint-work assistance, Organizational Capacity Building Share of CSOs Receiving Joint Work - Organizational CEPPS2 CEPPS3 N Mean N Mean Change Pr > F (a) Total Sample 20 0.7 47 0.57 -0.13 0.33 IRI-Supported Countries 9 0.56 30 0.5 -0.06 0.781 NDI-Supported Countries 17 0.65 41 0.56 -0.09 0.549 IFES-Supported Countries 18 0.72 39 0.59 -0.13 0.33 (a) F-test to test if the proportions of CSOs are equal between CEPPS2 and CEPPS3 Note: Countries are typically being supported by more than one CEPPS implementer at a time. Table 4.3.6 presents the share of CSOs receiving joint-work assistance for organizational capacity building between CEPPS2 and CEPPS3. We see downward trends across the board in CSOs receiving organizational joint-work assistance. However, these trends are not statistically significant for any group of countries. Table 4.3.7 below breaks down these numbers by region. 161 Table 4.3.7. Share of CSOs receiving joint-work assistance for Organizational Capacity Building, by region Share of CSOs Receiving Joint Work - Organizational CEPPS2 CEPPS3 Region N Mean N Mean Change Pr > F (a) Africa 9 0.67 18 0.61 -0.06 0.788 Asia 1 1 7 0.29 -0.71** 0.006 Eurasia 7 0.57 14 0.79 0.22 0.366 Latin America/Carribean 0 0 1 0 0 NA MENA 3 1 7 0.43 -0.57** 0.02 (a) F-test to test if the proportions of CSOs are equal between CEPPS2 and CEPPS3 Looking at trends by region in Table 4.3.7, we again see downward trends for all regions except Eurasia. The only statistically significant differences are for MENA and Asia, but given the very small sample sizes, these results can only be taken as indicative. Table 4.3.8. Share of CSOs receiving joint-work assistance, Technical Capacity Building Share of CSOs Receiving Joint Work - Technical CEPPS2 CEPPS3 N Mean N Mean Change Pr > F (a) Total Sample 29 0.52 74 0.54 0.02 0.83 IRI-Supported Countries 21 0.43 49 0.45 0.02 0.877 NDI-Supported Countries 26 0.5 69 0.52 0.02 0.853 IFES-Supported Countries 25 0.48 63 0.51 0.03 0.82 (a) F-test to test if the proportions of CSOs are equal between CEPPS2 and CEPPS3 Note: Countries are typically being supported by more than one CEPPS implementer at a time. Shares of CSOs receiving technical capacity-building in the form of joint work stayed relatively level between CEPPS2 and CEPPS3. For the total sample, the share of CSOs increased by only 2 percentage points between the two periods. The implementer-specific countries, the trends are similar. For the total sample and for each of the implementer-specific countries, the F-tests indicated that the differences in proportions of CSOs between CEPPS2 and CEPPS3 were not statistically significant. 162 Table 4.3.9. Share of CSOs receiving joint-work assistance for Technical Capacity Building, by region Share of CSOs Receiving Joint Work - Technical CEPPS2 CEPPS3 Region N Mean N Mean Change Pr > F (a) Africa 11 0.64 30 0.57 -0.07 0.697 Asia 4 0.5 19 0.58 0.08 0.802 Eurasia 9 0.56 13 0.62 0.06 0.793 Latin America/Carribean 4 0.25 4 0.75 0.5 NA MENA 1 0 8 0.13 0.13 NA (a) F-test to test if the proportions of CSOs are equal between CEPPS2 and CEPPS3 Asia and Eurasia had slight upwards trends in the shares of CSOs receiving joint work for technical capacity-building. Africa had a slight downward trend, dropping seven percentage￾points. The Latin America/Caribbean region had the largest increase, with shares of CSOs increasing by 50 percentage points between CEPPS2 and CEPPS3. MENA also had an increase of 13 percentage points. However, the results of the F-tests for equivalence of proportions between CEPPS2 and CEPPS3 indicate that none of the changes by region are statistically significant. EQ 3.3. To what extent were best practices adopted by local CSOs? The Effect of Training Table 4.3.10 presents the results of a series of probit models to determine if the capacity￾building trainings have an effect, using the CSO survey data. The outcome is a dummy variable which equals one if the CSO reporting incorporating more gender analysis into their programming after collaborating with a CEPPS3 implementing partner. The treatment variable of interest is a dummy variable that equals one if the CSO received training in gender analysis. The first probit uses no covariates and a standard dummy variable which equals one if the CSO received gender analysis training, while the second model uses a standardized version of the same variable, includes dummy variables for each implementing partners, and interacts the dummy variables with the standardized treatment variables. The regression table can be found in Annex IX under Table 4.3.10. For both model specifications, we see no statistically significant effect on the “gender trained” variable. In both cases there is a positive relationship between the variables, however none of the coefficients are statistically significant. For the second specification, we ran an F-test to test if the coefficients on the interaction terms were jointly equal to zero. This yielded a P-value of 0.9722, indicating that we fail to reject that the coefficients are jointly zero. We also conducted F-tests for joint significance for each interaction term. None of the F-tests showed statistically significant results on the interaction terms. A table detailing the results of this analysis can be found in the Appendix under Table 4.3.11. Table 4.3.10. Probit Models on the Effects of Training 163 (1) (2) Variables CSO Used Gender Analysis More during/after CEPPS3 CSO Used Gender Analysis More during/after CEPPS4 Gender Trained 0.409 (0.313) Gender Trained (de￾meaned) 4.706 (602.8) NDI -0.139 (406.4) IRI 2.280 (360.2) IFES 2.175 (360.2) NDI * Gender Trained -0.345 (680.1) IRI * Gender Trained -4.683 (602.8) IFES * Gender Trained 2.175 (360.2) Constant -1.657*** -3.654 (0.235) (360.2) Observations 148 148 Prob > chi2 * 0.9722 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 * Group test that NDI*Gender Trained; IFES*Gender Trained; and IRI*Gender Trained are jointly equivalent to zero. EQ 4.1. Which Factors contribute to the sustainable elevation of local organizations into sustainable providers of TA? Table 4.4.1a. Regression Results for Sustainability Indices on Inputs to Sustainability 164 Variables (a) Sustainabili ty Index (weighted) (a) Sustainabili ty Index (weighted) (b) Sustainabi lity Index (PCA) (a) Sustainab ility Index (PCA) (b) Sustaina bility Index (weighte d) (a) Sustaina bility Index (weight ed) (b) Sustain ability Index (PCA) (a) Sustain ability Index (PCA) (b) Uses Consistent Donor￾Approved Overhead Rate 0.141 0.145 0.257** 0.251** (0.109) (0.106) (0.125) (0.121) Number of local awards above Regional Average 0.316*** 0.308** * 0.195 0.177 (0.0999) (0.0990) (0.124) (0.121) IRI 0.199 0.253 0.153 0.156 (0.379) (0.435) (0.329) (0.407) NDI 0.0612 -0.145 -0.0201 -0.115 (0.385) (0.441) (0.340) (0.421) IFES -0.0993 -0.0599 -0.158 -0.0930 (0.416) (0.477) (0.363) (0.449) Constant -0.428 -0.936* -0.418 -1.447** 0.0267 -0.403 -0.129 -0.906 (0.509) (0.531) (0.584) (0.603) (0.465) (0.489) (0.575) (0.599) 0 0 0 0 Observations 89 89 89 89 99 99 99 99 R-squared 0.143 0.150 0.144 0.168 0.212 0.208 0.103 0.116 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: number of CSOs; number of NGOs; GDP per-capita; 2009 population; region dummies. (b) Additional covariates include: number of CSOs; number of NGOs; GDP per-capita; 2009 population; number of implementers in-country; region dummies. (c) If the organization had a greater number of the following items than the regional average: logo; mission statement; brochures/pamphlets; building/office/suite with functioning utilities; transportation vehicle; computer; website; NGO license/registration; budget; specialized staff; support staff. Table 4.4.1b. Regression Results for Sustainability Indices on Inputs to Sustainability 165 Variables (a) Sustainab ility Index (weighte d) (a) Sustaina bility Index (weighte d) (b) Sustaina bility Index (PCA) (a) Sustain ability Index (PCA) (b) Sustain ability Index (weight ed) (a) Sustain ability Index (weight ed) (b) Sustain ability Index (PCA) (a) Sustai nabilit y Index (PCA) (b) Number of international awards above Regional Average 0.430*** 0.418*** 0.420** * 0.404** * (0.0990) (0.0979) (0.122) (0.119) Has large number of organizational items (c) 0.198* 0.196* 0.155 0.157 (0.102) (0.0995 ) (0.122 ) (0.118 ) IRI 0.00596 0.00582 0.200 0.190 (0.317) (0.389) (0.340) (0.409 ) NDI -0.0326 -0.150 -0.0212 -0.127 (0.325) (0.400) (0.353) (0.424 ) IFES -0.287 -0.228 -0.176 -0.111 (0.349) (0.429) (0.376) (0.452 ) Constant 0.305 -0.111 0.260 -0.487 -0.288 -0.892* -0.321 - 1.219* * (0.455) (0.481) (0.559) (0.584) (0.469) (0.496) (0.563 ) (0.590 ) Observations 99 99 99 99 99 99 99 99 R-squared 0.279 0.272 0.190 0.199 0.157 0.158 0.094 0.112 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: number of CSOs; number of NGOs; GDP per￾capita; 2009 population; region dummies. (b) Additional covariates include: number of CSOs; number of NGOs; GDP per-capita; 2009 population; number of implementers in-country; region dummies. (c) If the organization had a greater number of the following items than the regional average: logo; mission statement; brochures/pamphlets; building/office/suite with functioning utilities; transportation vehicle; computer; website; NGO license/registration; budget; specialized staff; support staff. 166 EQ4.2: What are the characteristics or prerequisites of the local and regional organizations receiving CEPPS III support which most align with potential for success? Table 4.4.2. Implementer and CSO Opinions on Factors to CSO Success Please select up to 3 critical factors to CSO success in [country]. Implementer CSO Response Option Total % of total Total % of total Collaborations with other CSOs 53 44.9% 187 66.5% Absence of government intimidation 63 53.4% 68 24.2% Strong admin. systems 51 43.2% 81 28.8% Expertise of senior staff 32 27.1% 87 31.0% Adequate funding 45 38.1% 172 61.2% Capacity building assistance from international orgs 69 58.5% 115 40.9% Access to skilled staff 13 11.0% 70 24.9% Other(1) 9 7.6% 15 5.3% Other(2) 1 0.8% 1 0.4% Total: 118 100% 281 100% Table 4.4.3. Identifying the Effects of CEPPS3 Programs on Sustainability Variables (a) Sustaina bility Index (weighte d) Sustain ability Index (PCA) Sustaina bility Index (weighte d) Sustain ability Index (PCA) Sustaina bility Index (weight ed) Sustain ability Index (PCA) Sustai nabilit y Index (weig hted) Sustai nabilit y Index (PCA ) Uses Consistent Donor-Approved Overhead Rate 0.207 0.368* (0.164) (0.185) CEPPS3 * Uses Consistent Donor￾Approved Overhead Rate -0.124 -0.207 (0.221) (0.250) Number of local awards above Regional Average 0.363*** 0.270 (0.134) (0.164) CEPPS3 * Number of local awards -0.104 -0.171 167 above Regional Average (0.168) (0.206) Number of international awards above Regional Average 0.440** * 0.433** (0.140) (0.171) CEPPS3 * Number of international awards above Regional Average -0.0290 -0.0493 (0.178) (0.218) Has large number of organizational items (b) 0.203 0.145 (0.14 8) (0.17 7) CEPPS3 * Has large number of organizational items (b) 0.008 27 0.046 8 (0.21 0) (0.25 0) Constant - 0.821 - 1.140 (0.67 5) (0.80 6) Observations 99 99 R-squared 0.170 0.118 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: number of CSOs; number of NGOs; GDP per￾capita; 2009 population; number of implementers in-country; implementer dummies (IRI, NDI, IFES); region dummies. (b) If the organization had a greater number of the following items than the regional average: logo; mission statement; brochures/pamphlets; building/office/suite with functioning utilities; transportation vehicle; computer; website; NGO license/registration; budget; specialized staff; support staff. EQ 5.1. What CEPPS efforts have supported the opportunities of local organizations for becoming direct recipients of local funding? Table 4.5.3. Is assistance to CSOs to become direct recipients a program objective? 168 Was helping local CSOs in [country] to become direct recipients of donor funding an objective for your program? [Implementer survey] Response Option Total % of total Yes, it was an explicit objective 14 12.3% It was not explicit, but it was one of the project's objectives 28 24.6% It was not a major objective, but it was viewed as a positive parallel benefit 42 36.8% No, it was not an objective at all 29 25.4% Skipped 1 0.9% Total: 114 100% 169 EQ5.2: Have efforts been made to help local organizations’ planning in order to be prepared to apply for and receive larger and more complicated awards Table 4.5.10. Implementer and CSO Opinions on the Extent to Which Implementers Built CSO Capacity to Apply for Larger Awards Variables (a) CEPPS3 made efforts to help CSO become capable of applying for larger awards IFES -1.027 (0.718) IRI -0.541 (0.667) NDI -0.309 (0.659) Constant -0.806 (0.903) Observations 121 Prob > chi2 (b) 0.337 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: GDP per-capita; region dummies, and the number of implementers in-country. (b) Test that shares are equal across the three implementers Table 4.5.11. Variables (a) CEPPS3 made efforts to help CSOs obtain funds… Obtained Funds with CEPPS3 Assistance Obtained Funds with CEPPS3 Assistance Obtained Funds with CEPPS3 Assistance …"to a great extent" 1.987*** 170 (0.406) …"to some extent" 0.517* (0.283) …"to some or a great extent" 1.746*** (0.307) Constant -0.868** -0.714** -1.408*** (0.274) (0.256) (0.319) Observations 115 115 115 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: GDP per-capita; region dummies. Table 4.5.13. Variables (a) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) CSO Involved in design process -0.576 -0.503 -0.465 -0.541 -0.236 -0.477 (0.519) (0.541) (0.513) (0.533) (0.540) (0.561) NDI 0.0622 -0.111 0.276 0.159 (0.396) (0.412) (0.409) (0.425) IRI 0.257 0.340 0.00654 0.265 (0.370) (0.385) (0.403) (0.419) IFES -0.224 -0.232 -0.242 -0.0194 (0.374) (0.390) (0.421) (0.438) NDI * CSO Involved in design process 0.284 -0.105 0.174 -0.0671 (0.782) (0.815) (0.775) (0.806) 171 IRI * CSO Involved in design process -0.153 -0.0501 -0.383 -0.121 (0.737) (0.766) (0.762) (0.793) IFES * CSO Involved in design process 0.107 -0.118 -0.237 -0.153 (0.750) (0.781) (0.771) (0.801) Constant -0.450 -0.429 -0.648* -0.679* -0.404 -0.583* (0.377) (0.393) (0.348) (0.362) (0.324) (0.337) Observations 61 61 61 61 61 61 R-squared 0.172 0.184 0.175 0.191 0.174 0.189 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: GDP per￾capita; region dummies. Table 4.5.14. Variables (a) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) CSO Involved in awards management process -0.146 -0.0171 0.192 0.189 -0.698 -0.799 (0.648) (0.678) (0.553) (0.575) (0.541) (0.566) NDI 0.126 -0.0966 0.417 0.258 (0.431) (0.451) (0.434) (0.451) IRI 0.333 0.429 -0.0449 0.259 (0.388) (0.404) (0.437) (0.457) IFES -0.341 -0.363 -0.396 -0.147 (0.389) (0.407) (0.441) (0.461) 172 NDI * CSO Involved in awards management process -0.653 -0.924 -0.993 -1.136 (0.857) (0.896) (0.797) (0.829) IRI * CSO Involved in awards management process -0.364 -0.280 0.536 0.719 (0.857) (0.892) (0.847) (0.885) IFES * CSO Involved in awards management process 0.377 0.187 0.931 0.971 (0.859) (0.898) (0.788) (0.823) Constant -0.384 -0.345 -0.656* -0.671* -0.330 -0.529 (0.389) (0.407) (0.355) (0.369) (0.345) (0.361) Observations 59 59 59 59 59 59 R-squared 0.171 0.175 0.170 0.183 0.166 0.173 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: GDP per￾capita; region dummies. Table 4.5.15. Variables (a) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) CSO involved in M&E process -0.780 -0.737 0.470 0.442 -0.0181 -0.178 (0.565) (0.601) (0.597) (0.632) (0.543) (0.575) NDI -0.104 -0.262 0.191 0.0492 (0.420) (0.447) (0.425) (0.450) IRI 0.370 0.417 0.204 0.441 173 (0.380) (0.402) (0.429) (0.454) IFES -0.321 -0.312 -0.161 0.0549 (0.379) (0.403) (0.431) (0.456) NDI * CSO involved in M&E process 0.704 0.446 -0.526 -0.710 (0.791) (0.842) (0.814) (0.862) IRI * CSO involved in M&E process -1.315 -1.284 -0.828 -0.672 (0.831) (0.880) (0.791) (0.838) IFES * CSO involved in M&E process 1.238 1.126 0.483 0.579 (0.830) (0.884) (0.813) (0.862) Constant -0.363 -0.399 -0.636* -0.683* -0.475 -0.687* (0.413) (0.440) (0.378) (0.400) (0.334) (0.354) Observations 59 59 59 59 59 59 R-squared 0.188 0.177 0.196 0.194 0.194 0.190 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: GDP per￾capita; region dummies. Table 4.5.16. Variables (a) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) Sustaina bility Index (weighte d) Sustaina bility Index (PCA) CSO involved in the reporting process 0.227 -0.189 0.694 0.850 -0.668 -1.136** (0.503) (0.516) (0.555) (0.561) (0.506) (0.519) NDI -0.0441 -0.146 0.415 0.313 (0.391) (0.401) (0.405) (0.410) 174 IRI 0.525 0.600 0.123 0.362 (0.380) (0.384) (0.407) (0.418) IFES -0.581 -0.546 -0.469 -0.249 (0.384) (0.394) (0.425) (0.436) NDI * CSO involved in the reporting process -1.011 -1.154 -1.489* - 2.216*** (0.726) (0.745) (0.793) (0.803) IRI * CSO involved in the reporting process -0.506 -1.167 0.867 0.827 (0.765) (0.774) (0.712) (0.730) IFES * CSO involved in the reporting process 0.625 1.091 1.519* 2.041** (0.781) (0.800) (0.799) (0.820) Constant -0.312 -0.384 -0.736** -0.791** -0.427 -0.641* (0.379) (0.388) (0.351) (0.356) (0.321) (0.329) Observations 60 60 60 60 60 60 R-squared 0.213 0.260 0.207 0.273 0.208 0.254 Standard errors in parentheses: *** p<0.01, ** p<0.05, * p<0.1 (a) Additional covariates include: GDP per￾capita; region dummies. Table 4.5.17. F-Tests for Sustainability Outcomes Variable Sustainability Index Type Implemen ter 1 Test 𝛿𝛿 𝑚𝑚 𝑚𝑚 Test 𝛿𝛿 𝑚𝑚𝑚𝑚 Imple ment er 2 Test 𝛿𝛿 𝑚𝑚 𝑚𝑚 Test 𝛿𝛿 𝑚𝑚𝑚𝑚 CSO Involved in the Reporting Process PCA NDI 0.084* 0.584 IFES 0.492 0.36 CSO Involved in the Reporting Process PCA NDI 0.819 0.03** IRI 0.369 0.111 CSO Involved in the Reporting Process Weighted Average IFES 0.95 0.086* IRI 0.423 0.171 CSO Involved in the Reporting Process PCA IFES 0.491 0.014** IRI 0.045* 0.068* 175 176 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 2052