CONSORTIUM FOR ELECTIONS AND POLITICAL PROCESSES STRENGTHENING (CEPPS) PROGRAM EVALUATION REPORT AUGUST 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Lawrence Robinson, Lincoln Mitchell, Fares Braizat, Rawan Abadneh, Rajan Kapoor, Maggie Lada, and Diana Al Qutub for Management Systems International (MSI), A Tetra Tech Company. (THIS PAGE INTENTIONALLY LEFT BLANK) CONSORTIUM FOR ELECTIONS AND POLITCAL PROCESSES STRENGTHENING (CEPPS) Program Evaluation Report Contracted under AID-278-C-13-00009 Jordan Monitoring and Evaluation Support Project (MESP) DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. (THIS PAGE INTENTIONALLY LEFT BLANK) USAID.GOV CEPPS PROGRAM EVALUATION REPORT | i CONTENTS ACRONYMS...........................................................................................................IV EXECUTIVE SUMMARY ........................................................................................1 INTRODUCTION................................................................................................................................................... 1 BACKGROUND ...................................................................................................................................................... 1 EVALUATION METHODOLOGY...................................................................................................................... 1 FINDINGS AND CONCLUSIONS ..................................................................................................................... 2 RECOMMENDATIONS.......................................................................................................................................... 9 I. INTRODUCTION ...........................................................................................12 ACTIVITY SUMMARY ..........................................................................................................................................12 BACKGROUND AND PROJECT DESCRIPTION........................................................................................12 EVALUATION PURPOSE, AUDIENCE, AND INTENDED USES.............................................................13 EVALUATION QUESTIONS...............................................................................................................................13 II. BACKGROUND ........................................................................................14 COUNTRY CONTEXT .......................................................................................................................................14 CEPPS OBJECTIVES...............................................................................................................................................15 THEORY OF CHANGE .......................................................................................................................................15 III. EVALUATION METHODOLOGY .........................................................16 TIMELINE AND PROCESS OVERVIEW ..........................................................................................................16 DATA COLLECTION METHODS....................................................................................................................16 DATA ANALYSIS PROCEDURES .....................................................................................................................18 SURVEY DATA.......................................................................................................................................................18 DATA STORAGE AND TRANSFER.................................................................................................................18 STUDY LIMITATIONS..........................................................................................................................................19 IV. FINDINGS AND CONCLUSIONS .........................................................20 EQ 1: WHAT IS THE OVERALL EFFECTIVENESS OF CEPPS FOR ACHIEVING ITS GOALS AND OBJECTIVES?...........................................................................................................................................................20 SPECIAL SECTION: CEPPS PHONE SURVEY THEMATIC BRIEF..............37 EQ2: HOW DID THE STRATEGY AND IMPLEMENTATION APPROACH ENHANCE OR WEAKEN ACHIEVEMENT OF THE INTENDED OUTCOMES? ARE THERE CERTAIN AREAS/ACTIVITIES AND APPROACHES THAT HAVE BEEN MORE EFFECTIVE? WHY? .............56 EQ3: HOW HAS THE PROGRAM ADAPTED TO CHANGES AND HOW HAS COLLABORATION WITHIN CEPPS AND OTHER DRG PARTNERS/DONORS INFLUENCED ACTIVITIES AND THE ACHIEVEMENT OF RESULTS?..............................................................................61 EQ4: WHICH INTERVENTIONS ARE MOST LIKELY TO SUSTAIN OVER TIME (AND WHICH WILL BE DIFFICULT TO SUSTAIN)? WHY AND HOW? WHAT SHOULD BE CONSIDERED TO ENHANCE SUSTAINABILITY AND LOCAL OWNERSHIP?............................................................64 EQ5: WHAT ARE SOME KEY LESSONS LEARNED THAT CAN INFORM THE ACTIVITY AND THE MISSION GOING FORWARD? ...............................................................................................................68 V. RECOMMENDATIONS............................................................................70 A. FOR SUPPORTING THE IEC AND ELECTION MONITORING .......................................................71 B. FOR IMPROVING COORDINATION........................................................................................................72 C. GENDER, YOUTH AND PWD....................................................................................................................72 D. POLITICAL PARTIES AND PUBLIC OPINION.......................................................................................73 ii | CEPPS PROGRAM EVALUATION REPORT USAID.GOV E. LOCAL GOVERNANCE .................................................................................................................................74 G. PARLIAMENTARY STRENGTHENING.....................................................................................................75 H. SUSTAINABILITY AND NATIONAL OWNERSHIP..............................................................................75 I. SCHOOLS PROGRAMMING..........................................................................................................................75 J. ENGAGEMENT WITH UNIVERSITY YOUTH...........................................................................................76 K. WOMEN CANDIDATES AND PROSPECTIVE CANDIDATES TRAINING PROGRAMS ..........76 L. TRANSPARENCY AND OPENNESS...........................................................................................................77 M. MONITORING, EVALUATION, AND LEARNING ...............................................................................77 ANNEX A. EVALUATION STATEMENT OF WORK.....................................78 INTRODUCTION.................................................................................................................................................78 BACKGROUND AND ACTIVITY DESCRIPTION ......................................................................................78 PURPOSE OF THE EVALUATION....................................................................................................................79 OBJECTIVES AND THEORY OF CHANGE...................................................................................................79 EVALUATION QUESTIONS...............................................................................................................................81 EVALUATION DESIGN AND METHODOLOGY........................................................................................81 PERFORMANCE PERIOD....................................................................................................................................83 DELIVERABLES AND TIMELINE........................................................................................................................83 ANNEX B. EVALUATION DESIGN...................................................................84 INTRODUCTION.................................................................................................................................................84 BACKGROUND AND PROJECT DESCRIPTION........................................................................................84 EVALUATION QUESTIONS...............................................................................................................................85 DATA COLLECTION METHODS....................................................................................................................92 DATA ANALYSIS METHODS ............................................................................................................................93 LIMITATIONS .........................................................................................................................................................94 GETTING TO ANSWERS MATRIX..................................................................................................................97 WORK PLAN........................................................................................................................................................101 DETAILED PROPOSED CONSULTATION LIST........................................................................................102 DATA COLLECTION INSTRUMENTS .........................................................................................................104 CEPPS OBJECTIVE MAP.....................................................................................................................................125 ANNEX C. SURVEY QUESTIONNAIRE.........................................................127 SURVEY MODULES.............................................................................................................................................127 A. RESPONDENT BACKGROUND (PRE-CODED)..................................................................................127 B. TRAINING EXPOSURE AND PERCEPTIONS........................................................................................128 C. GENERAL OUTLOOK..................................................................................................................................130 D. ENGAGEMENT AND AGENCY................................................................................................................131 E. ATTITUDES AND PERCEPTIONS .............................................................................................................138 F. DEMOGRAPHICS............................................................................................................................................142 ANNEX D. SURVEY METHODOLOGY TECHINICAL REPORT................144 STUDY OBJECTIVES...........................................................................................................................................144 STUDY OVERVIEW............................................................................................................................................144 RESPONDENTS....................................................................................................................................................144 SAMPLE DESIGN AND SELECTION..............................................................................................................144 RESPONSE RATES...............................................................................................................................................145 QUESTIONNAIRE REVIEW..............................................................................................................................150 QUALITY CONTROL MEASURES .................................................................................................................151 CODING AND DATA ENTRY........................................................................................................................151 PROJECT SCHEDULE.........................................................................................................................................152 STUDY CHALLENGES .......................................................................................................................................153 USAID.GOV CEPPS PROGRAM EVALUATION REPORT | iii SURVEY METHODOLOGY APPENDIX: POST-RESEARCH DATA PROCESSING AND QUALITY CONTROL PROCEDURE CHECKLIST ........................................................................................................153 ANNEX E. SURVEY KEY FINDINGS REPORT ..............................................155 CEPPS PHONE SURVEY FINDINGS...............................................................................................................155 SURVEY CONCLUSIONS .................................................................................................................................165 iv | CEPPS PROGRAM EVALUATION REPORT USAID.GOV ACRONYMS AMEP Activity Monitoring and Evaluation Plan AOR Agreement Officer’s Representative CEPPS Consortium for Elections and Political Process Strengthening CIS Civic Initiative Support COP Chief of Party COR Contracting Officer’s Representative CBO Community-Based Organization CSO Civil Society Organization CSS Center for Strategic Studies (University of Jordan) DRG Democracy, Human Rights and Governance EMB Electoral Management Body EOM Election Observation Mission EQ Evaluation Question ET Evaluation Team G2G Government to Government GCHR Governmental Coordinator for Human Rights GESI Gender Equity and Social Inclusiveness GI Group Interview GOJ Government of Jordan IEC Independent Election Commission IFES International Foundation for Electoral Systems IP Implementing Partner IR Intermediate Result IRI International Republican Institute JNCW Jordanian National Commission for Women KII Key Informant Interview LSG Local Self-Governance M&E Monitoring and Evaluation MEMRC Middle East Marketing and Research Consultants MESP USAID/Jordan Monitoring and Evaluation Support Project USAID.GOV CEPPS PROGRAM EVALUATION REPORT | v MoE Ministry of Education MoI Ministry of Interior MoMA Ministry of Municipal Affairs MoPIC Ministry of Planning and International Cooperation MoPPA Ministry of Political and Parliamentary Affairs MoTA Ministry of Tourism and Antiquities MOU Memorandum of Understanding MSI Management Systems International NAMA Strategic Intelligence Solutions NCHR National Center for Human Rights NDI National Democratic Institute for International Affairs NGO Non-Governmental Organization PIRS Performance Indicator Reference Sheets PWD Persons with Disabilities SME Small and Medium-sized Enterprise SOW Statement of Work UNDP United Nations Development Programme US United States USAID United States Agency for International Development USG United States Government USD United States Dollar YLA Youth Leadership Academy USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 1 EXECUTIVE SUMMARY INTRODUCTION USAID/Jordan requested an external performance evaluation of the Consortium for Elections and Political Processes Strengthening (CEPPS) program. Management Systems International (MSI), through the Monitoring and Evaluation Support Project (MESP), brought together an experienced team of staff and consultants for the evaluation which was conducted with fieldwork in Jordan between March and July 2018. The CEPPS award under evaluation covered a period of performance of the CEPPS partners between January 2010 and June 2017. The three CEPPS partners were the National Democratic Institute for International Affairs (NDI), the International Republican Institute (IRI), and the International Foundation for Electoral Systems (IFES). The total value of the award over this period was $53.3 million. CEPPS was re-awarded in 2017 for the performance period July 2017 to December 2020, with a total expected value of $19.2 million. The activity is now implemented by NDI and IRI. At USAID’s request, the evaluation included the first quarter of this new award and the support for the 2017 decentralization elections through CEPPS. A separate Government-to-Government (G2G) award of $1.5 million was provided in May 2017 to the Independent Election Commission (IEC) for the administration of the Decentralization Elections which is not examined here. BACKGROUND Key goals of US Government (USG) foreign policy in Jordan include to help ensure that Jordan becomes increasingly responsive to citizens and that it becomes more supportive of civil and political rights. These goals require supporting the development and consolidation of pluralistic, fair, broad-based, and representative elected institutions. The CEPPS program aims to strengthen the development of more democratic and open political processes in the Kingdom and, specifically, to support the participation of candidates, activists, monitors, and voters in elections. To this end, the program worked to achieve specific objectives including: • Improving the ability of civil society organizations (CSOs) to build grassroots demand and effectively advocate for a new legal framework for elections; • Developing avenues to engage youth in the election process and civic engagement; • Strengthening political alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels; • Encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls; • Bolstering public demand for candidates’ commitment to open and more accountable performance as future members of parliament; and • Building the capacity of the Independent Election Commission (IEC) to conduct transparent and credible election processes. EVALUATION METHODOLOGY USAID sought an evaluation to inform its decisions on the future course of work under the existing CEPP award, to better integrate current programming with other parts of the democracy and governance portfolio, to inform future Mission strategy, and to better support the Jordanian-led reform 2 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV agenda. The audience for the evaluation is USAID, the CEPPS partners, Jordanian stakeholders, and the development community engaged in supporting democracy, human rights and governance programming in the Middle East. USAID asked the evaluation team to answer the following evaluation questions: EFFECTIVENESS AND RELEVANCE 1. What is the overall effectiveness of CEPPS for achieving its goals and objectives? 2. How did the strategy and implementation approach enhance or weaken achievement of the intended outcomes? a. Are there certain areas/activities and approaches that have been more effective? b. Why? 3. How has the program adapted to changes and how has collaboration within CEPPS and other DRG partners/donors influenced activities and the achievement of results? SUSTAINABILITY 4. Which interventions are most likely to sustain over time (and which will be difficult to sustain)? a. Why and how? b. What should be considered to enhance sustainability and local ownership? LEARNING 5. What are some key lessons learned that can inform the activity and the Mission going forward? The MESP evaluation team collected, reviewed, and analyzed both secondary and primary data to answer all the evaluation questions. Methods used were document review of CEPPS strategy, reporting, and learning documents; site visits; key informant and small group interviews; and a survey of 2017 beneficiaries of key CEPPS partner activities. FINDINGS AND CONCLUSIONS The findings and conclusions drawn from the main findings are organized by the questions asked by USAID. This summary briefly presents some of the key main findings and conclusions from the evaluation team’s (ET) work that are used in Chapter V to generate recommendations. EVALUATION QUESTION (EQ) I: OVERALL EFFECTIVENESS OF CEPPS FOR ACHIEVING ITS GOALS AND OBJECTIVES The evaluation examined the six objectives of the program sequentially to address this EQ. CEPPS OBJECTIVE 1: IMPROVING THE ABILITY OF GROUPS OF POLITICALLY ACTIVE CITIZENS AND CIVIL SOCIETY ORGANIZATIONS (CSOS) TO PARTICIPATE IN THE ELECTORAL PROCESS, BUILD GRASSROOTS DEMAND AND EFFECTIVELY ADVOCATE FOR A NEW LEGAL FRAMEWORK FOR ELECTIONS The ET’s findings noted that while CSOs valued the support received from NDI, the scope of NDI’s engagement with CSOs became focused on only a few of the organizations that would like to partner with NDI. This work opened space for civil society, however, in particular with government. As a U.S organization, NDI encountered challenges working with Parliament. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 3 NDI’s choices to work around rather than through parliamentary structures led to institutional resistance that limited NDI’s reach and scope of direct work in Parliament. NDI’s strategy to work around parliamentary structures was informed by USAID and the 2012 SUNY parliamentary assessment report, which saw parliamentary structures, specifically leadership structures, as a main challenge to implementing parliamentary reform. Despite these limitations, NDI’s work was viewed as valuable by the Members of Parliament with whom NDI worked directly. CEPPS OBJECTIVE 2: DEVELOPING AVENUES TO ENGAGE WOMEN, YOUTH AND THE DISABLED IN THE ELECTION PROCESS AND CIVIC ENGAGEMENT NDI did what was seen as path-breaking work in Jordan, opening universities to work on civic engagement and advocacy through the Ana Usharek (“I participate”) family of programs that NDI developed under CEPPS. These activities reached more than 25,000 students at universities. NDI built on this model to reach younger students in Ministry of Education schools as well. NDI also provided training for women candidates and potential candidates for office that was valued by participants, and developed an internship program that placed women as staff for Members of Parliament. NDI needs continual engagement with university deans to keep them fully informed and on board with activities at Jordanian universities. Many deans seek to continue Ana Usharek programs, and are interested in controlling them for sustainability. The Parliamentary Fellows program benefitted participants. NDI identified women who were interested in community activism and potentially running for elected office, but NDI’s candidate training had too diverse of an audience for the single training module delivered. IRI’s empowerment program developed advocacy and communications skills for people with disabilities (PWD), the Youth Leadership Academy (YLA) developed young people’s engagement in politics, and the Empower program targeted hard-to-reach women in poor areas of Jordan. Beneficiaries and stakeholders of these programs reported that their political skills and behaviors were changed by the programs, and that they now knew their rights, how to exercise them, and how to advocate to the authorities. IRI’s approach focused on intensive work with a smaller number of beneficiaries, identified using referrals and through community organizations. The evaluation found that working with a smaller number of beneficiaries, previously identified through trusted channels, was an effective approach. The PWD empowerment program was an effective way to expose PWD to politics, in many cases for the first time. The PWD empowerment program helped members begin to get a sense of their own political agency. The tiered structures of YLA helped participants to remain involved with the program and build upon the skills they developed. Discussions and workshops, as opposed to lectures, are by far the better way to reach people, particularly young people, in Jordan. IFES improved access to polling places for PWD and supported a technical committee to mainstream gender in the IEC. Addressing questions of accessibility for PWD to the electoral process is the kind of technical work that IFES was well positioned to do and that helped strengthen the IEC. The IEC could have done more work around gender. Gender mainstreaming at the IEC has not yet been institutionalized. To understand how the beneficiaries from NDI’s Ana Usharek and Usharek+ programs and IRI’s Youth Leadership Academy program interact with and understand different avenues for civic engagement, the evaluation team conducted a phone survey (n=463) of beneficiaries from all years of each program. The survey was designed to be used as a contextual tool to understand the attitudes, perceptions and reported behaviors of this cohort of beneficiaries around civic life in Jordan. As such the survey asked a 4 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV series of questions aimed at gauging respondents’ reported attitudes towards key political/civic themes covered in NDI’s and IRI’s programs, perceptions of the effectiveness of different civic/political institutions in Jordan, and participants’ perceptions of their own agency in engaging in key avenues for civic participation (e.g., voting, campaigning, community activism) that have been focused on heavily by NDI and IRIs program. Many of the survey questions were taken from the 2017 and 2016 IRI national surveys to allow for comparisons of survey respondent characteristics and national-level averages. The survey found that while respondents showed higher than average knowledge of key civic concepts, and high levels of agency attitudes, there were still low levels of reported formal or informal civic engagement relative to knowledge and attitudes. Data from the survey suggests that this disconnect could in part be due to the fact that while, overall, the CEPPS survey respondents show increased levels of trust and faith in Jordanian political/civic institutions, the levels of trust remain very low. The survey also showed a troubling and persistent bias against women’s participation in civics and politics. While this gender bias was seen in both men and women, it was more strongly associated with male beneficiaries. A full report of key findings from the survey may be found in Annex E. CEPPS OBJECTIVE 3: STRENGTHENING POLITICAL PARTIES AND ALLIANCES AND THE ABILITY OF CANDIDATES TO ARTICULATE, ORGANIZE AND IMPLEMENT CLEAR POLITICAL ALTERNATIVES AT THE NATIONAL AND SUB-NATIONAL LEVELS IRI provided technical assistance to 43 partner political parties on long-term development, such as party structure; outreach to youth and local communities; and policy and issue-based message development as well as short-term support towards running effective campaigns. IRI’s work with party leadership was challenging because of the nature of political parties in Jordan and their perception in society. IRI encountered challenges in their efforts to engage youth with political parties, and the approach featured little gender emphasis. IRI activities targeted the political parties together, which disregarded the differences and the intellectual gaps among the parties. Overall, IRI’s work on capacity building was helpful, but it did not help parties confront the larger issues they face in Jordan. CEPPS OBJECTIVE 4: JORDANIAN CITIZENS, POLITICAL PARTIES AND ORGANIZATIONS HAVE GREATER ACCESS TO QUALITATIVE AND QUANTITATIVE PUBLIC OPINION RESEARCH IRI conducted three national polls and provided access to this polling data to citizens, political parties, and CSOs through workshops, briefings, public presentations, and media outreach, as well as workshops to increase the understanding of polling. The three omnibus polls had many, varied goals and audiences. The knowledge and literacy around polls and public opinion research is still low in Jordan. IRI polling presentations effectively explained key ideas and principles of polling. Political parties did not, however, see polling as relevant to their own activities; this reflects the dysfunction of political parties. CEPPS OBJECTIVE 5: BOLSTERING PUBLIC DEMAND FOR LOCAL SELF-GOVERNMENT AND CANDIDATES’ COMMITMENT TO OPEN AND MORE ACCOUNTABLE PERFORMANCE AS FUTURE MEMBERS OF PARLIAMENT NDI supported the work of several CSOs in disseminating principles of good conduct in electoral politics among candidates and supporting development of local self-governance in Jordan. These efforts were appreciated by civil society. NDI noted that this early work helped to open up of political space to civil society to discuss a draft law that had no precedent and was an important step forward for Jordan. In the wake of the large new refugee flows to Jordan from the Syrian civil war, NDI took up USAID’s USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 5 challenge to incorporate refugees into USAID/Jordan programming by adapting Ana Usharek programming to create the Mujtam3i Democracy and Civic Education Program to bring refugees and host communities together through education and advocacy. The program was implemented in 12 communities with 165 neighborhoods or villages in Mafraq and Irbid. The development of the program and recruitment and training of coordinators took substantial effort. NDI was proud to have developed the program and felt that project implementation processes were sensitive to the difficult context of the time. Pre-post data on beneficiaries demonstrate that participants asserted greater knowledge and capacity in the wake of the training. NDI found communities in the North to be quite conservative and apprehensive about politics. Beneficiaries sampled for the evaluation did not note increased social cohesion in their communities. IRI assisted civic-minded volunteers called Citizens Committees to channel citizen priorities to municipal councilor and mayoral candidates, and then to those elected. Citizens’ Committees also implement local governance initiatives with IRI support that brought municipal officials and their constituents together to work towards common goals. To improve mayors’ and municipal councilors’ responsiveness to citizen concerns, IRI met with the local officials regularly to ensure their cooperation and engagement. Mayors had a mixed response to IRI’s work at the municipal level. Most Citizens Committee members believed the activities that IRI helped organize contributed to their community. IRI sources told the ET that some government officials and municipal officials have begun to use research tools like the IRI-developed Baldytak (“Your Municipality”) Application to help guide their decisions. IRI’s approach of working with both the demand and supply sides of municipal government was helpful. IRI’s work at the municipal level was strengthened because IRI engaged in several different tactics. Mayors who were not happy with the municipal governance program seemed driven by a more general disdain for IRI or western-supported NGOs. CEPPS OBJECTIVE 6: PROMOTING THE TRANSPARENCY AND INTEGRITY OF ELECTION PROCESSES AND BUILDING THE CAPACITY OF THE IEC TO CONDUCT TRANSPARENT AND CREDIBLE ELECTION PROCESSES. NDI and IRI deployed increasingly in-depth International Election Observation Missions in 2010, 2013, and 2016 (a joint mission). NDI interviews noted the important precedent that domestic and international election monitoring set in Jordan through the project. International election monitoring was not seen by stakeholders as a large component of NDI’s work. NDI’s support for domestic elections monitoring through the Rased coalition, which had monitored all Jordan-wide elections over this period, is appreciated by Jordanian stakeholders. Some dissatisfaction with election monitoring was expressed to the ET, with assertions that monitoring groups (both domestic and international) have not adequately recognized, in public forums, the flaws in the practice of elections in Jordan. International election observation missions by NDI and IRI helped the IEC demonstrate that they were competent to administer elections. These missions increased their ability to monitor elections over time. NDI support contributed to institutionalizing an enduring domestic election monitoring coalition able to routinely mobilize and monitor elections. IFES supported efforts to build the IEC’s long-term institutional capacity and sustainability, strengthen the legal framework for electoral administration, strengthen the IEC’s electoral management capacity, and build public confidence in the IEC. IFES contributed substantially to the evolution of the IEC and, therefore, to the improved integrity and transparency of the overall electoral process. The IEC found many foreign consultants that IFES brought in to Jordan were not helpful. There were challenges to effectiveness in technical support in the later years of the program. IFES representatives as well as 6 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV current and former IEC officials noted that the IEC would still benefit from additional technical support in areas such as voter education, how voters are processed at polling stations, vote counting, and other areas. Being responsible for both procurement and technical support sometimes hampered IFES. At an important moment in the development of the IEC, IFES was able to play a very crucial role. That role became more complex over time as the nature of the technical support, and the relationship between IFES and the IEC, became more complex. The IEC has made substantial progress since it was created six years ago but restricting technical support now could slow down that progress. Foreign consultants can be valuable, but it is imperative to choose them wisely and carefully. Once initial technical support had been delivered, the IEC became increasingly interested in procurement, as they needed resources to do their work. PROGRAM GOAL: STRENGTHENING THE DEVELOPMENT OF MORE DEMOCRATIC AND OPEN POLITICAL PROCESSES AND THE PARTICIPATION OF CANDIDATES, ACTIVISTS, MONITORS, AND VOTERS IN ELECTIONS Most ET findings and conclusions were on the objectives rather than goals of the program, as most activities of the CEPPS partners targeted objectives rather than the goal directly. However, some findings and conclusions directly focus on the goal. NDI’s youth programming reached large numbers of beneficiaries, opening up political processes at universities and schools. Sustained NDI engagement has also built a sustainable domestic election monitoring coalition. Women have benefited from the national and international experts provided by CEPPS partners. Women found trainings on conducting door-to-door visits and public communication and community outreach particularly useful. Although both NDI and IRI also had programs targeting women specifically, they also considered gender questions in other areas of their work. Helping women develop a battery of community outreach-related skills allows them to become more involved in their community and in political life. IRI achievements were particularly notable in providing information, resources and transfer of skills to women, young people and PWD; all of this makes it more possible for these groups to engage in political processes. IRI (and NDI) were able to develop effective international election monitoring in Jordan, thus contributing to the transparency of elections. Through its work with the IEC, IFES improved the electoral process while strengthening confidence in it and made it easier for people to vote. The IEC changed Jordanian electoral policies and became much more welcoming to election monitors. EQ2: HOW DID THE STRATEGY AND IMPLEMENTATION APPROACH ENHANCE OR WEAKEN ACHIEVEMENT OF THE INTENDED OUTCOMES? ARE THERE CERTAIN AREAS/ACTIVITIES AND APPROACHES THAT HAVE BEEN MORE EFFECTIVE? WHY? NDI sources noted the benefits of working through a strategy of working with the most interested, willing participants. NDI management and staff emphasized the value of connecting across programs. Working with the most interested Parliament members and most active committees helps increase the effectiveness of program implementation, but it has effects on the scope and scale of assistance. NDI’s engagement with the Ministry of Education has the advantage of working with state institutions. NDI has blended the approach of working through their own staff for the Ana Usharek family of programs and Parliament with an approach of working through the leadership of partner civil society organizations. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 7 NDI’s strategy of working with most interested parties produces expandable programs. NDI’s approach of working through their own staff presents challenges to sustainability and Jordanian ownership. There is a desire for greater Jordanian ownership in the areas where NDI works. IRI used internal evaluations and research, including public opinion research, to determine the strategic directions for their programs and activities. IRI found that one-on-one mentorship over a longer-than￾average program time frame was instrumental in the effectiveness of the Empower program. IRI found that improving local governance and civic engagement was best addressed by working on improving supply (mayors and staff) and demand (Citizens Committees) together. For IRI, strategy was very important, but this sometimes limited the scope of their work. IRI’s decision to eschew, for the most part, larger, more frequent trainings in favor of smaller, more intense ones was central to their work. IRI needs to address the current landscape of opportunities for assistance to political parties. IFES’s strategy of being housed within the IEC helped them establish close working relations with the Commission. Combining the technical support and procurement assistance to the IEC in one organization raised some challenges. Proximity made it easier for IFES to establish a strong working relationship with the IEC in the early years. Too frequently, efforts to provide technical support were sidetracked by concerns and questions about procurement. EQ3: HOW HAS THE PROGRAM ADAPTED TO CHANGES AND HOW HAS COLLABORATION WITHIN CEPPS AND OTHER DRG PARTNERS/DONORS INFLUENCED ACTIVITIES AND THE ACHIEVEMENT OF RESULTS? NDI has had tremendous program growth through various program modifications from 2010 through 2017. NDI demonstrated a willingness to develop new programs in response to new opportunities that developed. NDI adapted a worldwide NDI model for advocacy and discussion, the civic forum model, to the Jordanian context and then expanded this model. NDI staff and Ana Usharek/Usharek+ coordinators/managers noted ways that the feedback from participants led to the development of new modules. NDI’s initially larger portfolio of CSO partners became more focused over time on Al Hayat, Al Quds and a set of core activities (election observation and parliamentary monitoring). NDI only modestly changed its management structure as the small, short-term program expanded. NDI continued to rely heavily on its own staff for program implementation throughout the awards. IRI deemphasized its political party program as it grew increasingly apparent that placing a lot of effort into party work in Jordan was not fruitful. Given the slow progress, IRI has indicated that USAID asked that they stop working with parties to conduct research to identify more effective approaches for engaging political parties in Jordan. IRI was able to adapt in some respects, but sometimes this led them to limit the scope or intensity of their programs. Many at the IEC reported that coordination between IFES and the UNDP was not smooth, especially as IFES struggled to adapt to change and growth in the IEC. Coordination between IFES and UNDP was an ongoing problem. Several IEC staff reported that they were never clear as to which organization was responsible for what, and that the two organizations often appeared to be in constant competition to host activities and to enhance visibility. Most of the coordination within CEPPS and between the CEPPS partners with democracy, human rights and governance (DRG) partners happens informally at the country director or chief of party level. The three CEPPS partners worked well together and have had little to no issues of coordination. CEPPS partners and other USAID DRG implementing partners (IPs) generally reported few conflicts with 8 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV coordination and collaboration. There is a desire from USAID and other DRG partners to increase collaboration and coordination with CEPPS partners. EQ4: WHICH INTERVENTIONS ARE MOST LIKELY TO SUSTAIN OVER TIME (AND WHICH WILL BE DIFFICULT TO SUSTAIN)? WHY AND HOW? WHAT SHOULD BE CONSIDERED TO ENHANCE SUSTAINABILITY AND LOCAL OWNERSHIP? NDI’s approach has used NDI staff for project implementation with youth. This use of NDI staff and management of these staff has come at a cost to national ownership and sustainability. NDI partners recognize that NDI’s implementation of projects itself does not support CSOs’ sustainability. Areas of NDI technical support for CSO partners has, however, enabled sustainability, particularly in election monitoring and parliamentary monitoring. CSO partners continued to depend on donor funding – including through CEPPS – for engagement in elections and policy dialogue. University deans noted that they could sustain Ana Usharek if funded. Interest and institutional capacity may make the Ministry of Education a potential owner of Ana Usharek Schools. IRI notes that the Citizens Committee program may be able to sustain itself somewhat, as committees now frequently identify and recruit new members. The skills and capacities that participants in various IRI youth-oriented programs have gained will go with them if they continue to be involved in political life. IRI shifted to working with mayoral staff as well as elected mayors to support sustainability. Citizens Committees, with some more support, could become an important and enduring part of municipal governance. Although IRI programs will need continued support from USAID, the longer-term sustainability will likely be real, but hard to measure. Working with staff is valuable but getting buy-in from more mayors would also be helpful. The IEC has remained a functioning and competent electoral management body (EMB) after IFES's work with the IEC has concluded. However, many interviews emphasized that the IEC would still benefit from some technical support. While the IEC is still functioning and well-positioned to administer Jordan’s next elections, it still needs technical support in some areas. EQ5: WHAT ARE SOME KEY LESSONS LEARNED THAT CAN INFORM THE ACTIVITY AND THE MISSION GOING FORWARD? NDI’s monitoring and evaluation (M&E), analysis, and reporting has had little to say about lessons learned. NDI’s reporting mechanisms have not been used to develop or disseminate explicit lessons learned. NDI identified plans for analysis and learning as part of its 2017 proposal and award for expanding Ana Usharek, Usharek +, and Ana Usharek Schools. NDI reporting and interviews emphasized that sustained efforts and a longer time period were needed to build support for change within the context of Jordan. NDI’s long-term engagement helps support organizations and practices that are likely to persist. NDI does appear to recognize the need for stronger M&E, analysis, and stakeholder engagement in programming. NDI, USAID, and Jordanian partners and stakeholders would benefit from stronger analysis, and learning products based on this analysis. Making it possible for IRI to be a true partner so that IRI could develop its own strategy and ideas has led to better programming. The structure of IRI’s agreement and its relationship with USAID made it possible for the program to evolve, adapt and take advantage of opportunities as the context in Jordan changed. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 9 IFES was able to make critical early interventions to support the new IEC and rapidly develop a sense of the ongoing and evolving technical needs of the institution. The potential to build on the early work of supporting the IEC was not fully realized in part because of the completion of the IFES contract in Jordan. A strategy of incrementally phasing out assistance over a more extended period likely would have been more effective. RECOMMENDATIONS The ET offers the following recommendations for USAID and the CEPPS partners to consider: A. FOR SUPPORTING THE IEC AND ELECTION MONITORING A1. USAID should renew technical support to the IEC, but not with the same structure and intensity of the IFES program. A2. USAID should fund an independent needs assessment of the IEC. A3. The organization offering technical support to the IEC should not be responsible for procurement. A4. USAID should support a gender mainstreaming strategy for the IEC, as well as a full-time gender specialist at the IEC to help incorporate gender-related issues into the work of the IEC. A5. A process should be created so that the IEC is involved with selecting any foreign consultants with which they will work. A6. USAID should continue to support both domestic and foreign election monitors. A7. Rather than seek ways to quickly grow the IEC into a regional training body, USAID should focus on shoring up the gains the IEC has already made. A8. USAID should create formal coordination structures between any organization it supports to assist the IEC and existing international groups doing the same. B. FOR IMPROVING COORDINATION B1. USAID should create opportunities for CEPPS partners, as well as other DRG partners, to coordinate at levels below that of chiefs of party. C. GENDER, YOUTH, AND PWD C1. USAID should support programming that seeks to build the capacity of women in government ministries and agencies. C2. USAID should support efforts to develop a code of conduct that endorses women, youth, refugees and PWD-specific needs. C3. IRI should track YLA, Empower, and PWD Empowerment participants for years after they finish. C4. IRI should develop phase II programs for youth, women and PWD programming. 10 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV D. POLITICAL PARTIES AND PUBLIC OPINION D1. IRI should restart the party assistance program, but with more modest, attainable goals and more less intensive activities. D2. IRI should work with some political parties and relevant government offices to reform the political party law. D3. New IRI political party programming should have a clear gender component. D4. IRI should determine goals for public opinion research program. E. LOCAL GOVERNANCE E1. IRI should craft a strategy for working in municipalities with less cooperative mayors. F. CIVIL SOCIETY ENGAGEMENT IN MORE DEMOCRATIC AND OPEN PUBLIC POLICY PROCESSES F1. USAID should work through NDI to support the engagement of more civil society organizations in a variety of areas of public engagement in policymaking, to strengthen the development of more democratic and open political processes. F2. NDI should consider developing and holding networking and information-sharing events for civil society organizations engaged in public policy or civic engagement. F3. USAID should encourage NDI to continue to engage with Jordanian CSOs to maintain and support a culture of parliamentary and election monitoring. G. PARLIAMENTARY STRENGTHENING G1. USAID should work with NDI to assess the opportunities for working with Parliament and Members of Parliament in a comprehensive way and develop activities that both strengthen Parliament as an institution as well as the activity of diverse blocs and individual Members of Parliament. H. SUSTAINABILITY AND NATIONAL OWNERSHIP H1. NDI should consider a broader range of approaches to support sustainability and national ownership for all its programming. I. SCHOOLS PROGRAMMING I1. NDI should work towards and prepare to hand over Usharek Schools to the Ministry of Education. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 11 J. ENGAGEMENT WITH UNIVERSITY YOUTH J1. USAID should work with NDI to increase the sustainability of NDI’s work with university youth by encouraging national ownership. J2. NDI should consider how to expand the number of Ana Usharek and Usharek+ participants beyond those already interested in civic participation. J3. NDI should increase efforts to network Ana Usharek alumni and strengthen program implementation to encourage current alumni networking. J4. NDI should conduct a thorough review of its Ana Usharek and Usharek+ experience, with focus on deepening student engagement in the program. K. WOMEN CANDIDATES AND PROSPECTIVE CANDIDATES TRAINING PROGRAMS K1. USAID should work with NDI to develop training programs for prospective and declared candidates for public office that target the varying experience and capacity levels of prospective candidates. L. TRANSPARENCY AND OPENNESS L1. USAID should work with IRI and NDI to broaden public understanding of USAID’s support for strengthening the development of more democratic and open political processes in Jordan. M. MONITORING, EVALUATION, AND LEARNING M1. NDI should consider whether its management structure can be strengthened to better support learning and adaptation from program implementation. M2. NDI should conduct research on the longer-term effects of Ana Usharek and Usharek+ and incorporate the results of this research into program implementation to seek to have more enduring effects on alumni behavior. 12 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV I. INTRODUCTION ACTIVITY SUMMARY With a total value of $53.3 million, covering the performance period of January 2010 – June 2017, the Consortium for Elections and Political Processes Strengthening program (CEPPS) was implemented by a group composed, of the National Democratic Institute for International Affairs (NDI), the International Republican Institute (IRI), and the International Foundation for Electoral Systems (IFES). CEPPS was re-awarded during the last year, covering the performance period of July 2017 – December 2020, with a total value of $19.2 million. The activity is now implemented by NDI and IRI. USAID/Jordan also requested that the evaluation consider the support for the decentralization elections held in August 2017, under the follow-on CEPPS activity. A separate, Government-to-Government (G2G) award of $1.5 million was provided in May 2017 to the Independent Election Commission (IEC) for the administration of the Decentralization Elections. This award was increased to $2.05 million in August 2017. The evaluation thus includes the first quarter of the current CEPPS grant through September 2017 but does not examine the current grant to the IEC. BACKGROUND AND PROJECT DESCRIPTION Jordan has been on a sluggish but steady course towards political reform. Over the last 10 years, Jordan has implemented various structural reforms that aim at engaging a greater proportion of citizens in the political process. The country’s unique experience during the Arab Spring has produced the first amendments to the constitution in decades and a new legal framework for elections and political parties. Most recently, parliamentary elections were carried out in 2016, and local elections were held in August 2017 under new municipalities and decentralization laws, ushering in new councils that can potentially bring change through more effective community engagement. However, regional developments, the influx of Syrian refugees, and a struggling economy continue to cast long shadows over the social, economic and political landscape in the country, leaving it vulnerable to conflicts originating from across its borders. Key goals of USG foreign policy in Jordan include to help ensure that Jordan becomes increasingly responsive to citizens and that it becomes more supportive of civil and political rights. These goals require supporting the development and consolidation of increasing pluralistic, fair, broad-based, and representative elected institutions. CEPPS contributes to USAID/Jordan’s current Country Development Cooperation Strategy (2013-2019), as amended and extended, under Development Objective #2, “Democratic Accountability Strengthened” and Special Development Objective #4: “Gender Equality and Female Empowerment Enhanced.” In particular, it contributes to IR 2.1, “Accountability of, and Equitable Participation In, Political Processes Enhanced,” and IR 2.3 “Civil Society Engagement and Effectiveness Increased.” The CEPPS activity is intended to support domestic election monitoring, increase participation in election processes, train candidates and political parties in effective campaigning and polling, and provide technical and in-kind assistance to the Independent Election Commission (IEC). The CEPPS program aims to strengthen the development of more democratic and open political processes in the Kingdom and, specifically, to support the participation of candidates, activists, monitors, and voters in elections. To this end, the program worked to achieve specific objectives including: USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 13 • Improving the ability of civil society organizations (CSOs) to build grassroots demand and effectively advocate for a new legal framework for elections; • Developing avenues to engage youth in the election process and civic engagement; • Strengthening political alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels; • Encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls; • Bolstering public demand for candidates’ commitment to open and more accountable performance as future members of parliament; and • Building the capacity of the IEC to conduct transparent and credible election processes. EVALUATION PURPOSE, AUDIENCE, AND INTENDED USES This report responds to the USAID SOW, provided as annex A, requesting the USAID/Jordan Monitoring and Evaluation Support Project (MESP) to conduct an evaluation of CEPPS performance. The objective of this evaluation is to provide USAID with findings and strategic recommendations related to the effectiveness of the CEPPS’ interventions and implementation approach. The evaluation seeks to inform USAID’s decisions on the future course of work under existing activities to better integrate current programming with other parts of the democracy, human rights and governance portfolio, inform future Mission strategy, and to better support the Jordanian-led reform agenda. The audience for this report is expected to be: 1. USAID, specifically those stakeholders involved with DRG programming; 2. Implementing organizations National Democratic Institute (NDI), International Republican Institute (IRI), International Foundation for Electoral Services (IFES); 3. Jordanian stakeholders (e.g., civil society organizations, governmental bodies and research institutions); and 4. The wider development community engaged in supporting democracy, human rights and governance programming in the Middle East. EVALUATION QUESTIONS USAID asked the evaluation team (ET) to answer the following evaluation questions: EFFECTIVENESS AND RELEVANCE 1. What is the overall effectiveness of CEPPS for achieving its goals and objectives? 2. How did the strategy and implementation approach enhance or weaken achievement of the intended outcomes? a. Are there certain areas/activities and approaches that have been more effective? b. Why? 3. How has the program adapted to changes and how has collaboration within CEPPS and other DRG partners/donors influenced activities and the achievement of results? SUSTAINABILITY 4. Which interventions are most likely to sustain over time (and which will be difficult to sustain)? 14 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV a. Why and how? b. What should be considered to enhance sustainability and local ownership? LEARNING 5. What are some key lessons learned that can inform the activity and the Mission going forward? II. BACKGROUND COUNTRY CONTEXT During the period of the CEPPS program (2010-217), Jordan conducted three rounds of parliamentary elections, two rounds of municipal elections and one round of decentralization elections (jointly with 2017 municipal elections). Despite an increase in the number of registered voters and people turning out at polling stations, political efficacy and perceptions of personal agency remained rather limited, according to consecutive polling by the Center for Strategic Studies (CSS) at the University of Jordan.1 In recent years, public confidence and satisfaction with representative institutions such as Parliament and political parties fell off significantly when compared to other public institutions such as those concerned with security, education, health, and formal justice. According to IRI polling data from April 2016 to May 2017, public confidence in political parties dropped 6 percentage points, from 25% of respondents reporting having some to a lot of confidence to 19%. Public confidence in Parliament dropped 27 percentage points, from 44% respondents reporting having some to a lot of confidence to 17% of respondents. In a wider context, despite political and monetary investment in political reform processes, Jordanians’ sense of justice and equality in their society has decreased significantly since 1999. In 2018, only 10% of adult Jordanians reported “justice exists in Jordan to a great extent,” down from 30% in 1999. Those reporting “justice doesn’t exist in Jordan” increased from 8% in 1999 to 23% in June 2018, according to CSS (Center for Strategic Studies) and NAMA (Strategic Intelligence Solutions) polls.2 Perceptions of inequality follow similar yet sharper turns. The percentage of those reporting “equality exists to a great extent” decreased from 20% in 1999 to 7% in 2018. And the percentage reporting “equality doesn’t exist in Jordan” increased from 13% in 1999 to 30% in 2018. Although people may associate various qualities to the ideas of justice and equality, clearly, they are putting a governance problem on the table for policy makers to reckon with. Interestingly, feelings of injustice and inequality are highest in areas with the highest voter turnout. For example, in the 2017 municipal/ decentralization elections, governorates registering high above the national average level of turnout (31.7%), reported the highest levels of public perceptions of injustice and inequality. The same trend applies to Parliamentary elections, too.3 Survey evidence demonstrates that there is a need for further work on governance issues. 1 Braizat, Fares, “Razzaz Restorative Justice and National Revival.” Jordan Times, July 21, 2018. 2 Ibid. 3 Ibid. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 15 CEPPS OBJECTIVES The CEPPS program aims to strengthen the development of more democratic and open political processes in the kingdom and, specifically, to support the participation of candidates, activists, monitors, and voters in elections. To this end, the program worked to achieve specific objectives including: • Improving the ability of civil society organizations (CSOs) to build grassroots demand and effectively advocate for a new legal framework for elections; • Developing avenues to engage youth in the election process and civic engagement; • Strengthening political alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels; • Encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls; • Bolstering public demand for candidates’ commitment to open and more accountable performance as future members of parliament; and • Building the capacity of the IEC to conduct transparent and credible election processes. THEORY OF CHANGE Success in electoral management, and the provision of assistance to electoral managers, depends not only on preparation for electoral events. Electoral assistance is more effective if delivered during the complete electoral cycle, as it enables election management bodies (EMBs) to identify and apply lessons learned from previous electoral processes in the country itself. Through long-term programs, electoral assistance providers and election managers are better able to improve key elements that help ensure credible elections. Institutional management structures and strategic plans can be reviewed and improved. Professional development programs can be developed and implemented to advance the knowledge and skills of its permanent and temporary staff. Legislation, regulations and executive instructions that govern elections can be reviewed and improved. Better quality operational plans and operational training programs can be developed for specific elections. And, perhaps most importantly, effective public outreach and awareness programs can be developed to involve all electoral stakeholders in the electoral process and boost the trust of all stakeholders in the electoral process. An empowered and effective parliament will build trust among constituents and promote accountability with the government. Public opinion of political parties may begin to change if parties can focus on issues of importance and talk about solving them, rather than about personal attributes. If youth are more aware of opportunities within the political sphere and have an increased understanding of political engagement, then they are more likely to participate in political activity. Local elected officials will be better equipped to deal with the pressures of managing limited resources and growing demands on municipal services, and better positioned to handle decentralization when that opportunity arises, if they engage in constructive dialogue with citizens about priorities and needs. Women will be more likely to vote their conscience if they have access to information about the voting process and know their vote is anonymous and will be more likely to eventually run for office, if they are encouraged by and learn from other independent women in elected office and other leadership positions with similar backgrounds. 16 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Faith in Jordanian public institutions will increase when citizens have a deeper understanding of democratic processes and decision-making structures; when processes for legislative drafting and appropriation of public funds are more transparent and reflective of input from civil society and the citizenry; when Parliament and other government institutions are held accountable through a combination of mutual oversight and robust civil society monitoring; and when barriers to the political participation of marginalized populations, such as women and youth and persons with disabilities, are reduced. Furthermore, electoral processes will gain legitimacy with the adoption of a new, fairer legal framework. Grassroots, civil society-led advocacy campaigns will help ensure that the new election-related legislation considers input from a broad range of stakeholders. Tensions between Syrian refugees and host Jordanian communities will decrease if both groups have a platform for dialogue and joint civic engagement through which they can address issues of mutual concern. III. EVALUATION METHODOLOGY This section describes the overall methodology that was used to answer the evaluation questions (EQs). The evaluation’s design is detailed in Annex B, which includes final qualitative and quantitative data collection instruments. TIMELINE AND PROCESS OVERVIEW The evaluation team’s initial review of CEPPS activity documentation began in March 2018. Over the course of March and April, the evaluation team developed the study’s design and work plan, including the generation of open-ended topic guides for key informant interviews (KIIs) and group interviews (GIs). The evaluation team also developed a survey questionnaire for select CEPPS beneficiaries including the NDI’s Usharek family of programs and the IRI Youth Leadership Academy. Primary data was collected through July and focused on in-depth interviews with key informants from USAID, IFES, NDI, IRI, Government of Jordan (GOJ) officials, and local partners. Additionally, the ET conducted group interviews with select NDI and IRI program beneficiaries. In-depth interview respondents were selected to provide the ET with a range of perspectives. These included viewpoints from those who received financial and technical support or participated in trainings, as well as those representing local partners and government institutions. The ET also conducted a survey of select program beneficiaries. Final presentation of findings to USAID and co-generation of recommendations with USAID occurred in August 2018. Further consultation with the CEPPS implementing partners (IPs) to discuss findings also took place in the same month. After finalization, the draft Evaluation Report was presented to USAID in October 2018. DATA COLLECTION METHODS The evaluation team collected, reviewed, and analyzed both secondary and primary data for this evaluation. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 17 SECONDARY DATA SOURCES Major secondary sources consulted for this evaluation included the following activity and context documents. These documents were the source of the activity monitoring data used in this report. For a full list of documents used to ground the evaluation in the overarching regional and Jordanian context, see Annex C. Documents consulted include: 1. The CEPPS-USAID Agreement; 2. USAID/Jordan’s Country Development Coordination Strategy; 3. The CEPPS Performance Management Plan; 4. CEPPS Work Plan; 5. CEPPS Quarterly Reports; 6. CEPPS internal assessments; 7. CEPPS Performance Monitoring Data; 8. CEPPS polls and surveys; and 9. Ministry of Tourism and Antiquities (MOTA) statistical data relating to tourism. PRIMARY DATA COLLECTION The team developed instruments to collect both qualitative and quantitative data. Tables 1 and 2 show how each of these instruments was utilized for each respondent type. The instruments are provided in Annex B. TABLE 1: SUMMARY OF KEY INFORMANT INTERVIEWS RESPONDENT TYPE INTERVIEWS USAID 6 CEPPS staff (IRI, IFES, NDI) 28 Local partners, institutions, experts 25 Government officials 7 Members of Parliament and candidates 17 Political parties 7 International organizations 4 TOTAL 94 TABLE 2: SUMMARY OF GROUP INTERVIEWS RESPONDENT TYPE INTERVIEWS GROUP INTERVIEW PARTICIPANTS Usharek Programs 2 13 Empower 2 13 18 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV RESPONDENT TYPE INTERVIEWS GROUP INTERVIEW PARTICIPANTS Citizen Committees 2 23 PWD 1 9 YLA 1 3 TOTAL 8 61 DATA ANALYSIS PROCEDURES Qualitative Data: The evaluation team analyzed all qualitative data collected through three different methods: descriptive analysis, content analysis, and trend analysis. • Descriptive Analysis: The team used descriptive analysis to define and describe CEPPS programs and to frame interview responses within the evaluation questions. • Content Analysis: The team reviewed the contents of KIIs, site visits, GIs, and relevant program documents to identify data and information directly relevant to answering the evaluation questions. Content analysis was also used to identify critical program activities that were of interest to the evaluation as well as to provide illustrative examples that help explain the findings. • Trend Analysis: The team reviewed data provided by the IPs and USAID that provided information about program implementation over time. This included measures such as quantitative indicators about program participants as well as descriptions of how programs and activities have evolved over the course of the program. KIIs with respondents outside of the direct programming, but who had substantial institutional memory were used to provide supporting data for trend analysis. Additionally, the team examined IRI’s polling data for use in comparisons over time. SURVEY DATA Frequencies: The team has drawn on frequency data from the survey of beneficiaries to help answer questions regarding the effectiveness of project activities. These data have been compared through content and descriptive analysis to identify patterns or inconsistencies in the findings. Crosstabulations: By using crosstabs from the public opinion survey, the team shows how specific groups such as women or youths viewed programs activities. This approach has made it possible for the team to learn more about the IPs’ performance with these key groups. DATA STORAGE AND TRANSFER Data storage procedures for this evaluation were governed under the provisions set out in the MESP contract signed between USAID and MSI. Survey data collected for this evaluation will be cleaned for submission to the Development Data Library in a machine-readable format. Personal identifying information has been redacted in accordance with MSI and MESP ethical guidelines. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 19 STUDY LIMITATIONS All evaluation designs and methodologies face some limitations. These limitations may affect the quality and quantity of data the ET can collect and analyze. The ET has identified the following limitations in developing the design and methods and implementing the evaluation: 1. Limited information on some program staff and beneficiaries: Key staff from earlier periods of program implementation were no longer engaged with the activity, the IP, or in the country. 2. Potential for limited independence of NDI and IRI staff: NDI and IRI requested that the ET not conduct individual KIIs with key program staff other than the COPs. USAID approved group interviews with key program staff which were held with senior NDI and IRI staff present. Generally, a group interview with the presence of management has the potential to inhibit individuals from speaking freely with an ET. Therefore, individual KIIs are standard best practice in performance evaluations. The ET notes, however, that it did not find there was any hesitancy or other related response bias from individuals interviewed with management present. The team does not feel that this adjustment to the data collection approach negatively impacted or biased the assessment or findings. 3. Recall bias: Key informant interviewees, group interviewees, and survey respondents may have had difficulties remembering details about the past relative to the present. This presented potential issues of bias in their recall of information. 4. Selection bias: Some key informants were not available to be interviewed or surveyed. This presented the possibility of selection bias within our purposive sampling. Respondents who chose to be interviewed or surveyed might differ from those who did not, in terms of their attitudes and perceptions or other areas. The telephone survey may have featured additional biases related to differences in trunk line or mobile phone access among different groups. 5. Halo bias: KII, GI, and survey respondents may have under- or over-reported socially undesirable answers or altered their responses in accordance with what they perceive as prevailing social norms. The extent to which respondents were prepared to reveal their true opinions may have also varied for questions that call upon the respondents to assess the performance of partners that provided them with benefits. 6. Absence of baselines: The ET did not have access to, nor was it able generate, baseline data to understand the situation prior to the CEPPS program. The ET was able to learn about the achievements of the program but was unable to systematically compare them to what the situation was pre-intervention. 7. Inability to assess attribution: Given the lack of a control group or baseline data, the ET could not authoritatively assess causality, i.e., whether any changes in individuals or organizations can be attributed to interventions or the work of the project. 20 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV IV. FINDINGS AND CONCLUSIONS EQ 1: WHAT IS THE OVERALL EFFECTIVENESS OF CEPPS FOR ACHIEVING ITS GOALS AND OBJECTIVES? The evaluation report answers this question sequentially by discussing the objectives of the CEPPS program and outlining key findings by implementing partner. The ET has organized the 23 discrete objectives of the three CEPPS partners into six larger CEPPS objectives by linking common themes and approaches pursued as individual objectives under the program by NDI, IRI, and IFES. Findings from the ET’s research on each objective are listed under each CEPPS partner; conclusions based on analysis of these findings are also listed by CEPPS partner and come at the end of each of these six objective sections. A brief section on overall effectiveness towards achieving the goal that covers NDI’s work concludes this relatively long section on the most complex of the evaluation questions. CEPPS OBJECTIVE 1: IMPROVING THE ABILITY OF GROUPS OF POLITICALLY ACTIVE CITIZENS AND CIVIL SOCIETY ORGANIZATIONS (CSOS) TO PARTICIPATE IN THE ELECTORAL PROCESS, BUILD GRASSROOTS DEMAND AND EFFECTIVELY ADVOCATE FOR A NEW LEGAL FRAMEWORK FOR ELECTIONS NDI Civil Society Capacity and Engagement on Decentralization Four NDI objectives fall under this single CEPPS objective. Under NDI’s Objective 1, NDI worked to strengthen civil society’s capacity to effectively advocate for a new legal framework on decentralization, elections and political parties through the provision of technical and financial support to four Jordanian CSOs—the National Center for Human Rights (NCHR), Identity Center, Al-Hayat Center for Civil Society Development (Al-Hayat), and Al-Quds Center for Political Studies (Al-Quds) to advocate amendments to reform legislation through nationwide campaigns. This work began in 2010 and continued through 2012 with NCHR and through 2017 with Al-Hayat. Support to Al-Quds was provided in 2015 and 2016, in partnership with the Jordanian National Commission for Women (JNCW). A partner stated: “We consider ourselves a strategic partner with NDI. We have been accommodative and responsive to embrace new, challenging projects related to Parliament elections introduced by NDI, and they provided us with the proper capacity building. We joined their global NDI network, and benefited from their knowledge transfer. Now we are considered as regional resources to monitor elections.” NDI’s civil society partners valued the capacity strengthening, exposure and advocacy they were able to achieve via campaigning for decentralization. Interviews with leaders from these CSOs revealed appreciation for NDI support and value for the capacity and experience they had had with NDI. A respondent said: “We consider our entity is a success story for the NDI, where they managed to build our capacity to do our work efficiently and effectively and act as a catalyst to build relationship between the Parliament and the local community. The financial funding from NDI is not more than 15% of our annual budget, though the size of work we do for the monitoring of the Parliament and elections is greater than that. Being the main national partner who manages the monitoring of the elections gave us so much credibility and great reputation, not only in Jordan but in the MENA region.” USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 21 NDI partnered consistently with two CSOs, Al Hayat and Al Quds, which built a strong, lengthy relationship between these two organizations and NDI. The initial work under this objective was part of a longer, larger relationship NDI developed with these two organizations in supporting the engagement of CSOs in democratic processes. However, organizations no longer receiving NDI support lamented that they were no longer partnered with NDI. CSO partners that had had challenges in program implementation in these awards or in awards from other partners - which some other experts and CSOs asserted was corrupt behavior – nevertheless felt qualified to continue to contribute to the goals of building civic activism, public engagement in elections, and advocacy and sought NDI support for this work. These potential partners did not seem to understand why NDI had chosen not to continue to work with them or in these areas. One respondent mentioned that even though they had not recently received a grant from NDI, their NDI award was a turning point that distinguished their CSO by developing a proper research system with proper resources. The CSO continues to produce quality reports using the techniques learned with NDI assistance. Domestic Election Observation Capacity NDI worked towards its Objective 4 to build civil society’s capacity to strategically and effectively monitor parliament and national and sub‐national elections by providing technical and financial assistance to CSO partners. Each parliamentary election in recent years in Jordan (2010, 2013, and 2016) has been administered differently under a different, new electoral law and system. Continued changes to electoral processes meant that CSOs needed support to understand these changes and additional training to monitor electoral processes correctly for each new system. The continued changes in electoral systems also meant that the public was substantially uninformed; NDI thus supported outreach campaigns to the public through CSOs to increase public information about the 2013 and 2016 parliamentary elections. NDI provided assistance to monitor the 2010 elections to one partner, the Al Hayat Center. For 2013, NDI supported three domestic election observation coalitions for parliamentary elections: a coalition led by NCHR; the Rased Coalition, led by Al-Hayat; and the Nizaha Coalition, led by the Identity Center. NDI also funded a national voter awareness campaign through NCHR. For 2016, NDI again supported the Rased coalition through Al-Hayat for domestic election monitoring. In 2016, NDI also funded Rased for a comprehensive public outreach campaign on the elections and in 2017 for monitoring of the local and decentralization elections. NDI’s civil society partners valued the capacity strengthening and sustained support for domestic election monitoring which they saw as important to the success of the Rased coalition. KIIs with beneficiaries from these organizations revealed pride in their experience and capacity to monitor elections, which had been routinized over the last three national elections and the 2017 local decentralization elections. NDI reporting noted broader benefits from the activity for raising the capacity of these organizations, and building the ability of CSOs to convey the views of citizens to decision-makers. The second component of this objective developed CSO capacity to monitor parliamentary performance. NDI worked with Al-Quds and Al-Hayat to develop parliamentary monitoring methods as well as to hold meetings across governorates to increase dialogue between parliament and Jordanian society. NDI’s CSO partners valued capacity support plus funding for parliamentary monitoring. These partners suggested in KIIs that NDI support had been instrumental in starting this monitoring work and developing their initial capacity to monitor. NDI reported that the strength of these organizations’ methods increased with continued assistance over time from NDI. NDI however noted that some MPs are critical of the rating for individual MPs which they felt were not fair to them. MPs and former MPs interviewed were 22 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV more supportive of the monitoring, however, including the ranking system. It was their view that the population needs to be aware of what their members do in Parliament to hold them accountable for representation, oversight, and lawmaking. MPs interviewed saw this knowledge as in their interest, as well as important for their constituents, but sought to have CSOs use a more nuanced system to rate performance than the one in use. In particular, MPs wanted a monitoring system that more comprehensively captured what MPs do rather than rewarding any participation equally (For example, “I confirm what this gentleman said” in order to get recognized was valued equally in the ratings with a substantive interjection about the substance of policy). Parliament monitoring was also seen to be highly incomplete, as most votes in Parliament (other than a vote of confidence and votes on the budget) are not taken through electronic voting that is recorded but instead simply through a show of hands which cannot accurately and completely be monitored. One former parliamentarian called parliamentary monitoring very effective, noting that changing the evaluation indicators for the parliamentarians exposed the travel of parliamentarians and showed people which MPs were not regularly attending sessions. NDI support was also needed at the time when the speaker challenged the accuracy of parliamentary monitoring and did not let the monitors enter Parliament. Assistance to Parliament, Members of Parliament, and CSOs on Parliamentary Processes NDI worked to also strengthen the relationship between parliament and civil society, and between parliamentarians and their constituents (NDI Objective 8). This work has at least three strands. First, to address the weak relationships and low trust that characterize many relationships between MPs and their constituents, NDI worked with MPs, including Parliamentary leaders, on their outreach to constituents. NDI staff consulted with MPs and blocs on organization and constituent outreach, as well as helped build the capacity of MPs to analyze draft laws. MPs and Parliamentary Committee leaders valued NDI’s support for orienting them to their responsibilities. There is high turnover in the ranks of MPs in each election in Jordan. New MPs come to office with limited knowledge and experience in Parliamentary practice. Interviews with former and current MPs noted the value of orientation presentations supported by NDI for their learning on their roles and responsibilities as MPs. Bylaw reform was recognized as important by MPs and former MPs as well as by independent expert interviews for the evaluation. The bylaws established the structure to define and regulate how blocs operate within the parliamentary system, ways to determine legislative priorities, and ways for MPs to represent constituents. Independent experts and MPs interviewed were not satisfied, however, with the extent of bylaw reform. MPs, former MPs, and independent experts noted subsequent efforts to reform bylaws further have not been successful and are seen as still needed. NDI reported that support for outreach through public consultations and the provision of staff resulted in greater information being conveyed to constituencies. A former parliamentarian noted that still there is a need to work on Parliament reform to have an improved internal decision-making structure. Reforming the Parliament bylaws and procedures is needed again to enhance the performance of parliamentary processes. The second strand of this work was to support the organization, staffing, and procedures of Parliament. NDI supported induction workshops on the roles and responsibilities of MPs and two rounds of revision of Parliamentary Bylaws. NDI also provided a new type of staff to MPs through Parliamentary Fellows. NDI fielded 128 Parliamentary Fellows over the period of the award. Women MPs, in particular, were each provided with a Parliamentary Fellow who served as staff for the member. NDI also provided fellows to work with key committees. NDI facilitated four rounds of fellows who were trained by NDI and served as staff, each for a 9-month period. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 23 This section focuses on findings on how NDI effectively engaged with Parliament and MPs; the effectiveness of engagement with women and youth - the fellows themselves – is addressed under Objective 2 and support for women and youth below. The Parliamentary Fellows program was hampered by institutional resistance from Parliament towards staff coming from channels outside of Parliament. The Speaker and General Secretary of Parliament felt that they were not adequately included as key stakeholders in the Parliamentary Fellows program and reacted by closing access to Parliament to NDI fellows in 2017. As a consequence, the fourth class of fellows was not able to physically work in the Parliament. However, fellows were able to support MPs on their outreach to constituencies, if not in their work in Parliament itself. Parliamentarians’ views of the assistance from NDI fellows varied, with some supporting and some lamenting the quality extent of training and support from NDI and NDI’s communications with them. MPs found that the utility and capacity of fellows varied. Some MPs found them useful while others, including ones that we not able to use the fellow in Parliament due to the ban on their entry to Parliament, asserted that they did not have much use for fellows. MPs noted some challenges working with Parliamentary Fellows from NDI and some reported issues in communications about fellow availability with NDI. NDI emphasized the limits on staffing in Parliament’s system, under which MPs were provided with one staffer of variable quality and knowledge without consultation. This situation needed to be – and was – augmented by the program. One female MP mentioned that she had to replace three fellows that worked for her in the 17th parliament, as their skills and experience were not up to the tasks for which she needed support. That a fellow could type the Human Rights report for the Committee on Human rights or help establish a website was not sufficient for her. NDI support to found and develop the Women’s caucus was recognized by MPs of both genders. While the Westminster Foundation activities to support a women’s caucus were also noted, NDI’s work in establishing the caucus was clear to current and former women MPs interviewed. Women MPs interviewed recognized the value of coming together as a group, although they noted that not all women MPs participate in the caucus. NDI facilitated the development of bylaws for the caucus and helped members set legislative priorities. NDI also supported meetings for women MPs with the public, which a group of women MPs noted “helped us to improve our outreach with the community and set actual priorities.” NDI addressed ad hoc requests and provided information on international standards and best practices to blocs, coalitions, and MPs to support their ability to consider draft legislation. These consultations were based on requests from MPs, including from committee chairs. NDI reported working with the Speaker to develop public outreach for the Parliament, and with committee chairs to develop their outreach. Some Parliamentarians were candid about that challenges of working with NDI in the context of Jordanian politics, which limited the appeal of NDI to MPs. NDI is recognized to be an American organization working in Jordan with the support of the USG. This connection to the USG is problematic in the context of Jordanian politics for some MPs. MPs of a more Islamist orientation were not interested in NDI assistance, as part of demonstrating their personal and party opposition to US policies in the Middle East. MPs and Parliamentary Committee leaders appreciated NDI’s provision of technical support for selected hearings and key issues where they lacked staff and knowledge. NDI management, committee chairs, and MPs interviewed particularly noted the importance of providing specialized consultants to support committee hearings on the budget and taxation; all noted that they lacked the specialized expertise to engage with the government ministries that develop and present budget and tax legislation for discussion 24 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV and approval by Parliament. NDI set up meetings for MPs with independent experts to analyze the draft budget and support the engagement of committees and their members in substantive discussion of these key measures. One female MP noted that “the budget trainings were very good, especially the gender responsive budgeting training.” Policy Research The third strand of Objective 8 addressed the civil society side of policy development. NDI provided technical assistance and sub-grants to seven CSOs on conduct policy research and development. NDI staff trained personnel from five organizations in 2013-14 (including Al-Quds) and from two CSOs in 2015-17, and supported their development of research and research-based policy papers with recommendations on specifics topics (including outreach to policymakers). Each subgrant awardee produced three policy papers. In addition, NDI funded Al-Hayat after the 2016 Parliamentary elections to hold focus groups for youth and disseminate their views on parliament and the government with respect to addressing the priorities of young people. CSO partners valued the opportunity to do funded policy research. CSO leaders interviewed from organizations that won competitive awards from NDI felt that the competition was valuable. The implementation was viewed as helping them to build their experience and gain visibility. CSOs with grants noted that NDI support opened the door for CSOs to find other grants based on their experience in monitoring elections. CSOs from competitive rounds wondered why there was not more competition for funded policy research from NDI in subsequent years. Some independent experts and CSOs felt providing sub-grants without competitive processes was unfair. These CSO leaders did not feel consulted in NDI’s processes of concluding and evaluating grant work done by their partners or included in any discussion about next steps. Some concerns were expressed by CSO leaders on the limited reach of NDI’s work with CSOs, which was criticized as using “preset partners.” These civil society interviews felt that NDI’s CSO engagement did not have strategy to break “clientelism” in the sector. Some CSO leaders felt that NDI should do more to reach out to underserved “Eastern” Jordanian CSOs. NDI reported that the Institute reduced the extent of its civil society work because the USAID DRG team had developed other projects designed to support civil society development in Jordan, such as the USAID Civic Initiatives Support (CIS) project. Engagement with Government Institutions NDI worked towards its Objective 10 to encourage a more participatory political process working with government institutions. NDI reported working with officials from the Ministry of the Interior (MoI), Ministry of Municipal Affairs (MoMA), Prime Minister’s Office of the Governmental Coordinator for Human Rights (GCHR), and Ministry of Parliamentary and Political Affairs (MoPPA). This work was reportedly focused on strengthening the public outreach done by these institutions. The apparent idea behind this engagement was to develop the supply-side of government institutions interested in working with the demand-side for good governance that NDI and others were stimulating through work with Parliament, youth, women, and civil society. NDI interviews asserted instead that the engagement was more focused and really about supporting decentralization. NDI also reported engaging consultants to work with key ministries on decentralization because each key ministry had “had a completely different understanding of what decentralization was and means.” In addition, there was not an inter-ministerial process in place for MoMA, MoPPA, the MoI, and the Ministry of Planning and International Cooperation (MoPIC) to come to an agreement on decentralization USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 25 NDI KIIs and reporting asserted that this engagement with government institutions was critical to the program to build support for NDI’s work. NDI leaders reported substantial contact with GOJ counterparts and emphasized the importance of building relationships with these key stakeholders who could otherwise hinder or halt NDI programming. Government agencies are clearly critical stakeholders with the potential to impede activities conducted under the program. Some leading CSO partners also noted the value of NDI’s engagement with government to “open doors” for their own engagement with these government partners. One partner appreciated the ability of the NDI Country Director, “who has extensive experience in Jordan and understands the local political context well.” NDI targeting of local governments, ministries (MOPPA, MOMA, and MOI) and Senators from the upper house while involving partners in their meetings opened the door for them and started a new kind of relationship that facilitated their work afterward with the government. Ministerial KIIs in institutions where NDI reported providing assistance to encourage a more participatory political process lacked institutional memory of this assistance in their ministries. There are several possible reasons for this. Government ministries may not have the systems and practices in place to build institutional capacity and instead remain mainly based on personal relationships. ET interviews may have missed the counterparts that NDI had engaged with; the key counterparts reached in this work stream may have moved on since engaging with NDI. Or, alternatively, they may not recognize this assistance and engagement as significant. NDI reported successful engagement through consultants and a study tour to Kosovo to build a common understanding of decentralization across key ministries. This engagement contributed to stakeholder agreement and buy-in and the draft legislation on decentralization. Kosovo was seen as an example of a country with devolved authority, based in that case on devolution as an avenue to manage minority populations. NDI reported that its own recommendations on decentralization were not taken in the draft Jordanian legislation. NDI emphasized that the promise of decentralization has not yet been fully realized. Key attributes of the decentralization legislation are for fiscal decentralization; 8% of Jordan’s oil revenues are supposed to go to the municipalities which would give local governments resources to respond to the priorities and demands of the population. This right does not appear to have been exercised to date. NDI emphasized the important precedent-setting nature of the accomplishments made through engagement with government institutions. NDI asserted that the engagement of CSOs and the public in debate and discussion of the decentralization law in 2012 and 2013 was the first time that public comment had been brought into the discussion of legislative changes of this kind. NDI interviews also noted the importance of support for the GCHR’s National Plan on Human Rights. This was said to be the first time a public entity in Jordan had developed a comprehensive plan for human rights. NDI also emphasized the importance of how the plan was developed, in consultation with international organizations as well as civil society and Jordanian citizens. CONCLUSIONS: IMPROVING THE ABILITY OF GROUPS OF POLITICALLY ACTIVE CITIZENS AND CIVIL SOCIETY ORGANIZATIONS (CSOS) TO PARTICIPATE NDI’s support to CSO partners was seen as useful by these partners. CSO leaders valued the financial and technical support from NDI that they had used for activities towards their own objectives, which were also objectives of the CEPPs program. 26 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV NDI support became more focused on fewer CSOs over the course of implementation. NDI reported providing TA to 17 CSOs which held a total of 129 events to gather information on citizen priorities or engage with Jordanian stakeholders. By 2017, however, NDI’s CSO partners were limited to Al Hayat and Al Quds for parliamentary monitoring and Al Hayat for work on election observation through the Rased coalition (discussed further as part of CEPPS Objective 6 below regarding suppor to the transparency of electoral processes). Other CSOs would like opportunities to work with NDI. Former partner institutions felt that the NDI program had been limited as a partner as NDI left them without support beyond a single, time-limited grant. These CSOs sought competitive procurements from NDI as a potential avenue for them to resume collaboration with NDI on areas of mutual interest. NDI demonstrated capacity to work with government institutions to open space for civil society and NDI’s work. Engaging government institutions appears to have more of an effect on enabling NDI and key partners to work in the sensitive political space of encouraging advocacy and citizen participation than in changing government openness to society or communications with the public more generally. Evaluation interviews did not find that government partners remembered NDI support for the outreach of government institutions. While there may have been effects on government institutions from this engagement that directly supported the outreach of these government institutions, interviews with current government officials found that the managers and leaders of these institutions did not recall NDI support for the government’s outreach to citizens. Working with parliament is challenging for a US organization. NDI is understood as an American organization working in conjunction with the USG. Where the organization is from and who funds it is important in the context of Jordanian politics. Political movements and parties that have many issues with US policies and actions in the Middle East are resistant to working with a US organization like NDI, to protest US engagement in the region. NDI’s choices to work around rather than through Parliamentary structures led to institutional resistance that limited NDI’s reach and scope of direct work in Parliament. NDI’s strategy to work around parliamentary structures was informed by USAID and the 2012 SUNY parliamentary assessment report, which saw parliamentary structures, specifically leadership structures, as a main challenge to implementing parliamentary reform. Despite these limitations, NDI’s work was seen as valuable by the members of parliament with whom NDI worked directly. CEPPS OBJECTIVE 2: DEVELOPING AVENUES TO ENGAGE WOMEN, YOUTH AND THE DISABLED IN THE ELECTION PROCESS AND CIVIC ENGAGEMENT NDI NDI reported on several main activities under the award for its Objective 2, “Develop civil society’s capacity to engage youth in electoral and political processes.” Jordanian universities had been spaces where political engagement was not supported or generally allowed; international assistance in this area had not been permitted prior to the Arab Spring. NDI developed a portfolio of youth work through engagement at the university level after an announcement by King Abdullah II that opened Jordanian universities to some outside political and civic engagement for the first time. From this opening, NDI developed a robust USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 27 set of youth programs to: (a)reach University students, (b) develop the practice of advocacy further for more advanced alumni of the regular university program, and (c) reach school-age children through Ministry of Education schools. NDI reported training a total of 204 young Jordanians to be the facilitators for Ana Usharek (“I participate”). Over the course of the CEPPS award – starting from the development of the program in 2012 through July 2017, NDI reports that facilitators led a total of 1,729 Ana Usharek groups with 25,600 university student participants from 28 universities across Jordan. Of these participants, NDI reports that 3,493 subsequently attended the Usharek + program and further developed advocacy skills. Ana Usharek and Usharek+ NDI’s work in universities on civic engagement and advocacy was recognized by many as path breaking. Interviews with NDI, independent experts, and university deans noted that NDI has successfully been able to identify and harness opportunities to work in universities, which earlier had been prohibited, once there was an opening. The key feature in this development was the invitation from King Abdullah II in the wake of the Arab Spring to universities that some civic engagement from university youth would be desirable. NDI was able to adapt the Institute’s civic forum methodology that has been used to stimulate youth engagement in civic and political life in other countries. NDI also developed relationships with university deans of students, who play a gatekeeper role for student activities, to obtain their endorsement for NDI to work with students. This approval was essential to be able to work in each university. NDI was able to develop and implement a whole new program under the CEPPS mechanism. NDI obtained USAID approval to pilot the training, developed contacts with Jordanian universities to test the methods, adapted a civic education curriculum used by the institute in other countries to Jordan, identified and trained facilitators to conduct this work, and tested these methods within the existing framework of the award. This demonstrated the flexibility of the CEPPS mechanism and the capacity of NDI/Jordan to develop new initiatives. At Jordanian universities Ana Usharek was a new avenue for youth to engage in the election process and civic engagement. NDI was able to attract large numbers of students to the Ana Usharek program. The fact that NDI was able to reach more than 25,000 students demonstrates that students are interested in more activism in civic life. NDI achieved this reach by mobilizing the students that were most interested in civic life. This method of working with the most interested students does not, however, reach all students. NDI was effectively able to build upon positive results and momentum from Ana Usharek. NDI expanded this activity and worked more extensively with the most interested university students who were alumni of Ana Usharek through Usharek+. NDI was able to build upon the experience of civic education at the university level to develop programming and gain access to primary and secondary schools and implement civics engagement for students through the Ana Usharek Schools program. Success in implementation varied across universities. Ana Usharek coordinators reported different levels of interest and support from different universities. This negatively affected program implementation when university support was limited. Without as much support from deans and universities, Ana Usharek was less successful; the program was begun and stopped, for example, at Aqaba University based on limited support, engagement and success for coordinators working with this university. 28 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Deans varied in the extent to which they saw NDI consultation and communication with them as sufficient. Some deans reported high levels of NDI collaboration and support while others saw the support as insufficient. Deans of students are reportedly frequently rotated at most Jordanian universities. This frequent rotation made consulting with the deans demanding for NDI. Deans, it should be noted, also play different roles at Jordanian universities than they do in many other university systems around the world. In Jordan, deans of students play a gatekeeper role in determining what is allowed on campus. Political activism, or the extent of civic activities, has traditionally been a controversial area for universities that deans have sought to stay away from entirely. Ana Usharek broke through this barrier at first in some universities and then more broadly across Jordan. In general the deans interviewed by the evaluation team were pleased that they had a peaceful civics program on campus. Deans with longer time in position (a small number) appreciated the development of Ana Usharek and its contribution in encouraging dialogue rather than violence as a political approach for students. Ana Usharek coordinators reported instances where participants were able to successfully engage with and persuade deans to support activities they initially saw as too “sensitive”. According to coordinators participants capitalized on new skills learned in the program to hold discussions with deans on program expansion. The coordinators interviewed stated that through these discussions Ana Usharek participants were able to get approval for expanding the issues covered by the program and on expanding the scope of participation on campus. Deans interviewed supported these discussions with students; however, as one dean noted, their support was contingent on students not going “past the red line.” Deans and coordinators sought greater outreach and advocacy from NDI for Ana Usharek to expand program size and reach within their universities. NDI staff and engaged deans expressed views that the Ana Usharek program should reach out to larger numbers of students and have a larger effect. The model of using coordinators to reach small groups of students was seen to limit how many students Ana Usharek could reach at a university. This was contrasted by some Deans to a training- of-trainers model in which University staff or students could lead groups themselves and reach larger numbers of students. Deans asserted that Ana Usharek and Usharek+ graduates were more engaged in political and civic life than other students. Deans interviewed noted that either it was their impression or that they had monitored the students and noted that alumni of the two NDI programs were more active than the rest of the student body in civic life. Deans saw this as a positive; student engagement in constructive civic activism was an attribute they sought to encourage. Deans and universities do not systematically evaluate Ana Usharek’s work; they do, however, monitor what students are doing in and around the program. One university dean reported surveying participants and faculty, which found Ana Usharek participants were more active. This attribute may have been preselected by the students themselves as well as by NDI, as the most civically active are drawn to Ana Usharek participation and the most active alumni are invited to Usharek+ by NDI. Left implicit was that less managed participation in civic life by students had the risk of being seen as non-constructive or even dangerous. The alumni were seen as important examples of advocacy and engagement at universities in a political context otherwise dominated by “tribalism.” NDI’s evaluation methods for Ana Usharek and Usharek+ focused on pre-post comparisons between program participants; the reported data showed modest growth in knowledge and modest change in attitudes immediately after participating in Ana Usharek in the directions expected by NDI in the initial cohort of teachers and students. Student attitudes have not been tracked over a longer period after participation. These evaluation methods also do not allow for attribution of observed change to program interventions. NDI’s evaluation methods also do not compare the cohort of students that participated in USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 29 Ana Usharek to those that did not directly benefit from the program. Deans and universities do not systematically evaluate Ana Usharek’s work. Ana Usharek alumni valued the active learning methods used by facilitators and skills development in public speaking and debate from the courses. Many courses at Jordanian universities are reportedly lectures that are delivered by faculty with limited participation by students. Coordinators and participants noted that Ana Usharek was different; the participatory methods and encouragement of discussion and debate were seen as important positive features of the program. Ana Usharek alumni and coordinators interviewed suggested that using even more interactive methods would be beneficial to the effectiveness of the program. Alumni recommended building in feedback to program managers from participants during program implementation itself. The evaluation’s beneficiary survey asked Ana Usharek and Usharek+ alumni about their preferences for further training. NDI youth program alumni who are interested in more training seek training in elections and politics. Of alumni that indicated a preference for further training (a third of Ana Usharek alumni and half of Usharek+ alumni), half preferred further training on elections. The only other topic with substantial support for more training was politics (with more than 20% of Ana Usharek alumni and 13% of Usharek+ alumni indicating interest). Ana Usharek and Usharek+ served as mechanisms for some of the most interested students to become coordinators in the program and even join NDI staff. The program not only educated participants but also served to identify quality staff to continue Ana Usharek. Some alumni went from participating in Ana Usharek to serving as coordinators to then working as staff for other activities of NDI in Jordan. Ana Usharek + involved and mobilized MPs, ministers, CSOs leaders and other experts during their trainings and debates to open a space for students to learn from leaders. This was seen as very useful and informative for the youth as well as the subject matter experts. One MP mentioned that “NDI used to organize debates and a review panel to select the best in these events. I was a judge in many of the competitions and it reflected the youth’s awareness of their role. Ana Usharek+ is very effective in engaging youth in political life, letting them conduct initiatives to solve problems. It made me understand how this generation thinks”. A former minister interviewed stated that Ana Usharek was very effective for youth as it “gave them a chance to work together, initiate activities and be part of working in the field. They could practice what they learned themselves.” A CSO partner of NDI noted that as NDI’s partner, “We always have a role in the Ana Usharek+ program, in universities which get youth to be engaged in political life. As a partner with NDI we provide training free of charge to Ana Usharek members and keep learning ourselves from those active members.” Ana Usharek Schools In another example of adaptation and change, NDI reported that it adapted the Ana Usharek university civic education curriculum to create civic education manuals and workbooks for students in grades 5-7 and 8-10. NDI began developing this program in 2015 through coordination with the Ministry of Education (MoE). NDI reached agreement that the MoE would support participation by social science teachers from boys’ and girls’ schools, taught by men and women respectively, across Jordan. NDI trained selected teachers using the adapted manuals and workbooks; trained teachers then taught one of their social studies classes of their choosing differently, adding the Ana Usharek Schools materials on top of the regular curriculum. NDI coordinators report assisting teachers to implement the training and assisting students 30 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV and teachers to develop and conduct community improvement initiatives, such as upgrading school facilities and improving accessibility to schools and community centers for people with disabilities. NDI expanded the size and reach of the program in three years of implementation starting from February 2016 to reportedly reach a total of 330 schools. NDI successfully implemented the program and continued to keep teachers, principals, and the MoE on board in program implementation. Training by NDI on interactive teaching methods was appreciated by the Ministry of Education. MoE interviews noted that teacher methods in Jordanian schools are overwhelmingly didactic; NDI training was appreciated for its interactive nature and use of exercises, including practical initiatives to encourage advocacy by students. The MoE asserted interest in owning the Ana Usharek Schools program. MoE management interviewed stated that the Ministry was interested in managing and owning the program. This interest appeared to be in a general sense, rather than an immediate interest in replacing NDI with ministry staff for implementing the program. The MoE did not demonstrate capacity to manage this program, however. Whether the Ministry could manage a program of this nature going forward is not an evaluation question. NDI Candidate and Potential Candidate Training for Women NDI reports on its main activities to expand and strengthen women’s participation under NDI’s Objective 3, “Expand and strengthen women’s participation in electoral and political processes by increasing the number of potential women candidates, and by building women parliamentarians’ and civil society actors’ capacities to address legislative barriers to women’s political participation.” Women face many challenges in political participation in Jordan. This objective focused on getting more women to run for Parliament, getting more women elected to Parliament, and making elected Women parliamentarians more effective in their work in Parliament and with their constituents. To support more women as potential candidates, NDI’s Women’s Participation Program provided three rounds of skills-building workshops focused on effective electoral campaign techniques to potential women candidates for the 2010, 2013 and 2016 parliamentary elections and provided individual consultations to assist individual candidates. Many, but understandably not all, of the women trained elected to run for Parliament. NDI conducted longer, more intensive training for additional potential women candidates in 2014, a period when elections were not expected. In addition, NDI worked with women’s organizations that year in districts where women’s electoral participation had been low to hold roundtables to identify additional potential candidates for the future. NDI also supported training and skills development for potential women candidates for the 2017 municipal and decentralization elections. NDI identified, invited, and trained diverse sets of women to increase the knowledge and skills of potential candidates for parliamentary elections and for the municipal and decentralization elections. NDI’s mechanisms led to recruitment of diverse groups of women with different skills, experience, and aspirations for training. Some of the best-qualified women participants were critical of the selection process; in their view, some participants in the trainings were not qualified to run (or not well qualified to have realistic prospects of prevailing in elections). Participants felt that the training needed to reach women earlier in the electoral process to provide them with adequate time to build their skills and then run and win; this was especially important for women with less experience in politics. In a parliamentary system, elections can be called before the end of Parliament’s mandate. Thus, trainings cannot always be reliably planned to precede elections by a fixed USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 31 amount of time. The training in 2014 would not have faced these timing challenges for potential candidates as it was held long before elections were anticipated, or any concrete date was set for elections. A former parliamentarian noted that a 2012 training was during the campaign period, so she could not effectively implement what she had learned - although the information was very useful. She appreciated that NDI brought in Jordanian ex-parliamentarians as experts and former MPs from other countries who showed the importance of knocking at “closed doors.” Women that attended NDI’s training for prospective women candidates most valued the development of their communications skills, which they reported using in their households (particularly with husbands), with their peers, for community activism, and to run for election. Other skills from training were also appreciated. Face-to-face methods were seen by many interviewees as most important for Jordan; some interviewees credited NDI for recognizing the importance of this method and their emphasis on face-to￾face outreach for women candidates as helping them learn and adopt these methods, which they saw as the most important avenues for winning office through election. Some women who did not run remain interested in a future campaign. One woman who could not run for elections as her uncle chose to run for office noted that she is “working through the approaches and strategies proposed by the NDI trainer, such as networking; building sustained relationships; identifying people to work with me at community level; working with media influencers, including the youth in our plans; and visiting people door to door in order to build support for me for the future.” Women participants interviewed appreciated the interactive methods NDI used in training but noted differential quality in the trainers and the trainings for prospective women candidates. Two of the five key informants that had participated in these trainings were highly critical. More experienced women were in general less enthusiastic about the training, which they saw as targeting a group of women that were less experienced and less well-qualified than they were. These more experienced women candidates sought more differentiated training for candidates that targeted their need for more advanced skills. These interviewees emphasized that one training did not fit all, based on different experience and knowledge prior to the training. Former MPs sought advanced candidate workshops to further develop their skills. Candidate training was viewed by some participants as not adequately adapted to Jordanian realities. The training was seen by these women interviewed to focus on western-style elections, which ignored some of the key factors in Jordanian elections that determines who wins (such as tribal norms). These women felt that social media modes of communication were overemphasized in NDI’s training. They instead felt that training should emphasize the need to maintain consistent in-person contact in districts and reach out face-to-face to constituents in the Jordanian context. NDI’s candidate training was viewed as the key to their winning office by some former MPs. One former woman MP interviewed noted that her successful strategy of focusing on firming up most-likely support among her constituents in order to win parliamentary elections had come from NDI; she credited what she viewed as the key strategy and her political “way of thinking” to NDI. NDI, she said, also provided her with her strategy for winning: to “look at those who are almost guaranteed to vote for you and work on them to make them guaranteed votes for you.” NDI’s guidance on how to broaden her support was seen as critical, and this is how she learned to “ignore those who are definite no votes” in her strategy to win election. This was a new idea to her, as she had earlier sought to win over voters in general without prioritizing the ones that were more likely to support her candidacy. 32 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Parliamentary Fellows NDI reported identifying and training 128 young Jordanians as Parliamentary Fellows through the program. This component was a four-year initiative to support Parliament and MPs (see Objective 1 above) but also develop the skills and experience of youth, particularly female youth, in politics and governance. Parliamentary fellows trained once a week with NDI as well as supported MPs or Parliamentary Committees. Parliamentary fellows interviewed reported that the training and experience was beneficial to them personally. This was the case even for fellows who were not able to enter Parliament. Former fellows interviewed noted how the skills development and learning from weekly sessions with NDI had helped make them more employable, including by NDI, after their fellowship concluded. A former fellow indicated that she learned how to write professional reports, which is considered her strength at her new work where not many staff know how to write business-like documents. IRI IRI approached this objective through three main streams of programming, the people with disabilities (PWD) empowerment program targeting people with disabilities, the Youth Leadership Academy targeting youth, and the Empower program, targeting hard to reach women in poverty pockets throughout Jordan. While these programs featured beneficiaries with overlapping gender equity and social inclusion (GESI) identities (such as female youth with disabilities), the evaluation examines each specific program based on their overarching goal. Therefore, PWD Empowerment is examined with a lens of empowering PWD, not a gender or youth lens. People with Disabilities IRI’s work with PWD was primarily through its PWD empowerment program in 2016 and 2017. IRI’s Persons with Disabilities Empowerment Program worked to provide PWD activists with advocacy and communication skills they could utilize to lobby for better protection of PWD rights and to fight discrimination. The participants in this program included citizens with physical limitations, hearing related disabilities, visual impairment and other disabilities. The program offered trainings for PWD in subjects including decentralization, elections, PWD human rights, employment laws, education rights, PWD needs in the community and international conventions. After an initial needs assessment meeting in 2015, IRI launched the program in 2016 and held multiple trainings in the governorates of Ajloun, Zarqa, and Karak to reach PWDs in each of Jordan’s three broad geographic regions (north, central, and south). IRI also held a series of events in late 2016 in Karak in observance of the International Day for PWDs. IRI marked the end of the 2016-2017 PWD Empowerment Program with a graduation ceremony where program participants gave speeches reflecting on their achievements in the program. The PWD empowerment program helped many participants learn about their rights and opportunities for the first time. Participants reported that after spending a year in the program they became aware of their rights, and more importantly, began to make demands based on that awareness. One participant described a university class that was scheduled on the second floor of a building that was not accessible for PWD. The participant mobilized the class to demand, successfully, that the class be moved to the first floor. Others spoke of feeling empowered to seek and gain employment or even demand better treatment and rights within their families. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 33 The PWD empowerment program also helped PWD begin to become more involved in political life. Some participants said the program made them more likely to vote or helped them think of the power of their vote differently. The participants emphasized that the PWD empowerment program created structures for the participants to have an active role in electoral process in general and political parties, not only through voting. Overall, the PWD appreciated IRI’s program, though it was reported that they needed more time during the trainings share their problems, challenges and thoughts. The PWD empowerment program helped many participants learn about their rights and opportunities for the first time. Participants reported that after spending a year in the program they became aware of their rights, and more importantly, began to make demands based on that awareness. For example, the participants mentioned that some of them went to different ministries and requested for a language translator to be present, in order for PWD to be able to communicate with ministry staff. Others spoke of feeling empowered to seek and gain employment. One participant added that prior to joining the program, her parents used to collect her salary. However, after participating in this program, she learnt that she has the right to collect her own salary and started to go by herself without her parents to collect her salary. Furthermore, the PWD empowerment program helped raise awareness about PWD rights in the elections law that they were previously not aware of, including the fact that they could enter the voting box without aid and having the right not to show their decision to the polling supervisors. The PWD empowerment program also helped PWD begin to become more involved in political life. Some participants said the IRI program made them more likely to vote or helped them think of the power of their vote differently. However, others felt that they were less likely to vote because they were more aware of their rights and the lack of engagement on behalf of the politicians and political parties on issues related to PWD. The participants emphasized that the PWD empowerment program created structures for the participants to have an active role in the electoral process in general and political parties, not only through voting, but also through advocacy and participating in civil society. Overall, the PWD appreciated IRI’s program, though it was reported that they needed more time during the trainings to share their problems, challenges and thoughts. Youth Leadership Academy The primary way in which IRI worked with youth with regards to this objective was through their Youth Leadership Academy (YLA) program. Youth were also represented in other IRI projects such as citizen committees and Empower. Additionally, part of the national opinion polls focused on youth to provide data for the political parties to increase their civic and political engagement of young people. IRI conducted YLA programs from 2015 to 2017. YLA brought youth participants from around the country together to attend trainings aimed at helping them become more politically active. With high youth unemployment rates and political bodies dominated by older people, young Jordanians often feel that their voices are ignored and become disillusioned with political participation. The YLA taught youth the fundamentals of democracy, the decentralization law, their rights, strategies to run for office, and how youth can make a difference in their country by engaging key decision makers. The YLA consists of “classes,” each spanning three multi-day sessions and culminating in a graduation ceremony. The YLA has a tier system that allows graduates to continue their political education with IRI. The four tiers of the YLA program are: 34 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV • Basic YLA, consisting of 234 youth, including 99 women, to participate in trainings aimed at making them more engaged in the political sphere in Jordan. • Advanced YLA, with179 youth, including 61 women. This program brought back the most enthusiastic graduates from the basic program to build upon their skills by teaching electoral campaign skills so that young people could run a successful campaign for parliamentary candidacy. • Policy Making YLA consisted of 175 advanced YLA graduates, including 75 women, who wanted to learn the planning and critical thinking strategies necessary to write policy papers for issues affecting Jordan. Sixty-three youth out of 175 graduated from the program. The exposure of youth to this information empowered them and helped them envision their future action in the political sphere, as was shown through IRI’s internal evaluation of the YLA program. However, only 36 percent completed the program and graduated, due to the rigor of the program’s curricula. • Political Party YLA consisted of 97 youth, including 36 women. Thirty-six youth, including 13 women, graduated from the program. YLA provided ongoing trainings and capacity building workshops during which IRI remained involved with YLA participants. This provided an opportunity for the youth participants to learn in depth about a wide range of leadership skills. Respondents also reported that they have used these skills to remain politically engaged in their community. However, some participants raised concerns that YLA trainers were arrogant and did not always share material that would have benefited the participants. In general, the youth participants’ reported experience with YLA were mixed. YLA trainings allowed ample room for discussion and were not simply lectures. Participants expressed different views. Some valued the discussions and said that it made the material seem more relevant. On the other hand, some participants expressed that YLA trainers were not neutral and took sides in discussions among participants. YLA participants reported that their behavior toward political engagement has changed completely. Participating in the IRI program raised youth curiosity and interest in politics which led them to read about the candidates’ programs as well as rely on research rather than on word of mouth to make their voting decision. Furthermore, the knowledge and skills learned from this program assisted youth to identify community needs, communicate those needs to the mayors and government officials within their governorates, as well as plan and implement advocacy programs. For example, according to the ET’s survey, when asked “to what degree do you have the ability to work in your community to address a common problem through advocacy?” 68% of YLA participants responded, “to a great extent.” The same proportion responded that they had joined a CSO. Empower On October 31, 2013, IRI launched the Empower Initiative, a series of trainings designed to increase the political and civic engagement of women from low-income communities. The launched event followed an intensive assessment phase during which IRI staff traveled to government-designated poverty pockets to determine the areas that are best suited to implement the program. The program ultimately grew to span 16 of the 32 government-designated poverty pockets. To develop the capabilities that allow women in poverty pockets to be active members of their communities, IRI trained these women on communication skills, negotiation techniques, political terminology and public speaking. After attending these workshops and a graduation ceremony, participants trained other young women the skills they learned from this program, causing a ripple effect. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 35 The Empower program set out to target women that were historically disenfranchised and disengaged and used a network and a referral-based approach to recruit a small group of women. IRI relied on these women they knew from previous work to network with other women living in these communities. According to the interviews with IRI staff and empower beneficiaries this approach to recruitment helped the effectiveness of the empower program by ensuring a basic level of buy in and a common level of knowledge amongst participants. The interviewees also stated that this approach allowed IRI to reach women that they felt would not have been reached with more traditional advertising. What makes the Empower program distinct from many other community-based organization (CBO) activities, is that this program worked with a small number of participants, its IRI trainers provided one￾on-one coaching and the activities were time- and labor-intensive. Furthermore, one of the main reasons why the Empower program was successful was that they had passionate trainers who were meeting with women, developing their skills and increasing their knowledge and exposure to politics during their personal time. Participants in one region described the IRI trainers with whom they worked as not just good at delivering the material, but as “committed” and capable of delivering the material in ways that demonstrated their sincere support for Empower participants. Furthermore, the respondents added that the IRI staff helped identify job and volunteer opportunities for their participants. Participants in another region reflected that through their interaction with Empower, friendships were formed between Empower participants and IRI staff. Respondents also expressed how participating in the Empower program allowed them to connect with other women-focused CBOs and community leaders such as the municipality staff and mayor. This linkage resulted in a sustainable avenue for the women to meet, conduct research, and carry out awareness sessions for other women and youth who plan to do other community-based initiatives. The majority of the respondents mentioned that these connections and partnered activities enabled these women to be considered by many in the community as community leaders. The Empower program helped many participants learn about their rights and how to get engaged in the political life for the first time. The women who participated in these programs had been on the periphery of Jordanian political life with little access to resources or information about political campaigns and civil society organizations. Through the Empower program, the participants learnt their rights, political terminologies, negotiation skills, public speaking, and strategies to vote for the person who reflects and plans to address their issues. This led them to gain self-confidence and feel valued in the community. As a result, the total number of women who voted increased and more women ran in the parliamentary and decentralization elections. IFES IFES pursued this objective through two primary means: improving access to polling places for people with disabilities (PWD) and forming a technical committee to mainstream gender in the Independent Election Commission (IEC). In late 2012, IFES assisted the IEC with preparations for the evaluation of polling centers identified as candidate sites to be made accessible for persons with disabilities. The IFES evaluation team included participants from the USAID-funded Takamol campaign and from the Higher Council for the Affairs of Persons with Disabilities. Following this evaluation, IFES assisted the IEC with the preparation of instructions which were sent to the Heads of the 45 Electoral District Committees, with specific 36 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV messages to address the remaining issues regarding the polling centers identified for PWD in their districts. Following on the IEC’s interest in enhancing the accessibility of polling stations, in mid-2015 IFES supported the IEC in conducting a technical assessment of polling and counting center accessibility and general barriers to participation for persons with disabilities. In late 2015, IFES conducted one training in each of the three regions (north, central, and south), convening the heads of the Education Directorate and the Training Section in each directorate (113 people total), to introduce participants to the assessment methodology and tools they would implement during the nationwide assessment. In early 2016, IFES submitted the final report of the Nationwide Polling and Counting Assessment. Overall, 3,900 facilities were initially targeted for assessment. Due to challenges encountered, 3,414 facilities were ultimately examined, in addition to the 57 facilities examined during the pilot assessment in Irbid’s 2nd Electoral District, bringing the total of the facilities included in the assessment to 3,471. In mid-2015, the IEC formed a Technical Committee for Gender, with the purpose of discussing future activities related to gender and political participation and crafting a gender strategy. The committee was based on IFES’ recommendations drawn from continuous discussions with the IEC Secretariat on the importance of placing a larger emphasis on gender-related issues. The commission was started after receiving approval from the IEC Board of Commissioners. In contrast to the PWD work, the Gender plan/strategy devised by the committee was not enforced by IEC management. The work IFES did with the IEC around PWD accessibility was generally lauded by current and former IEC staff. One interviewee noted that for IFES, the question of finding polling places for PWD was “taken seriously.” This project drew on the technical expertise and experience of IFES that was necessary for a task of this scope while at the same time raising awareness at the IEC and beyond about PWD accessibility for different parts of the election process and building the capacity of the IEC to implement these types of projects in an ongoing way. IFES supported awareness efforts aimed at female empowerment and gender equality through introducing the IEC to models and approaches from other countries. IFES has noted to the assessment team that this work included activities such as including increased numbers of female heads of polling/counting centers, serving as a key member of the IEC gender technical committee, gathering gender-based M&E data, developing gender-specific voter information modules and materials, and ensuring significant female representation in trainings and outreach. The assessment team did not verify the extent to which these efforts were implemented. Multiple interviews with current and former staff suggest, however, that in spite of these efforts the priority of gender issues within the IEC remains low. In these KIIs, responses to questions about gender issues were brief and almost dismissive. One respondent with background in gender policy indicated that the IEC simply never focused on challenges around gender and the electoral process. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 37 I. CEPPS RESPONDENTS’ ATTITUDES AND BELIEVES ON AGENCY zz Half of CEPPS survey respondents (52%) stated that they felt they had some degree of say in government decision making vs. 35% of IRI 2017 national poll respondents. SURVEY HIGHLIGHTS To understand how the beneficiaries from NDI’s Ana Usharek and Usharek+ programs and IRI’s Youth Leadership Academy program interact with and understand different avenues for civic engagement, the evaluation team conducted a 463n phone survey of beneficiaries from all years of each program. As noted in the methodology section of this report the survey was not meant to examine causality between attitudinal and behavioral outcomes and the CEPPS activities. The survey asked a series of questions aimed at gauging respondents attitudes towards key political/civic themes covered in NDI’s and IRI’s programs, perceptions of the effectiveness of different civic/political institutions in Jordan and a series of questions aimed at gauging participants’ perceptions of their own agency in engaging in key avenues for civic participation (voting, campaigning, community activism). Many of the survey questions were taken from the 2017 and 2016 IRI national surveys to allow for comparisons between survey respondents and national averages. CEPPS survey respondents reported a significantly higher interest in political reform (93%) than the national average as reported in the 2017 IRI national poll (59%). The avenue in which CEPPS survey respondents indicated the MOST confidence in their ability to engage was advocacy. Running for Office was the avenue survey respondents indicated the least confidence in their ability to engage. SPECIAL SECTION: CEPPS PHONE SURVEY THEMATIC BRIEF 38 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV The survey data suggest persistent biases around the role of women in politics may be key barriers keeping women from participating at the same level as their male peers. CEPPS survey respondents reported higher levels of civic engagement than the national average as reported in the IRI 2017 poll. However, in some key avenues engagement is still low. III. CEPPS RESPONDENTS’ ATTITUTES TOWARDS INSTITUTIONS IV. GENDER BIAS AMONGST CEPPS RESPONDENTS II. CEPPS RESPONDENT REPORTED ENGAGEMENT SURVEY HIGHLIGHTS (CONTINUED) The survey universe comprised of a population that was younger, more highly educated and more economically stable than the national average. Therefore, the respondents have more privileged backgrounds then average Jordanian youth. Additionally, most of the respondents self￾selected to participate in the programs targeted by the survey and two out of the three programs had screening and application processes which recruited only student with history of civic engagement and activism. Thus, this data should not be taken as representative of the Jordan population or Jordanian youth population. In some findings, particularly around reported engagement, there was a variance among the three beneficiary groups. However, most of that variance can be explained by the differences in participant background and participant recruitment. YLA recruited already engaged activists and political actors, Usharek+ recruited the best Ana Usharek graduates and Ana Usharek had no recruitment criteria and had beneficiaries with varying levels of exposure and interest in politics A full report on survey key findings can be found in Annex E. CEPPS survey respondents had similar if not worse perceptions of key governmental institutions compared the respondents from the 2017 IRI national poll Significantly fewer women reported participating in civic life (including voting, attending public meetings, meeting with officials, etc.) than men (23% gap in attending public meetings; 14% gap in engaging with public officials; 14% gap voting; 2016/2017). USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 39 CONCLUSIONS – DEVELOPING AVENUES TO ENGAGE WOMEN, YOUTH AND THE DISABLED NDI Activities NDI accomplished impressive expansion in the size, reach, and composition of the Ana Usharek family of programs. NDI developed an entirely new range of programs for youth that educated university youth in civic engagement in new active-learning methods that were new to Jordanian university students. NDI also created and supported the practical application of civics through advocacy projects of these youth, and then developed a successor program, Usharek+, to take these developments further with the most interested Ana Usharek students. NDI also created programming to reach younger students prior to universities through the Ministry of Education and teachers. The most engaged Ana Usharek stakeholders (NDI coordinators, deans, more active students) have sought ways to adapt and further differentiate within the program to increase program effectiveness. The most enthusiastic participants, either as coordinators, participants, or Deans, sought ways to improve Ana Usharek programs and build on the successes of the program. Engaged youth and deans were interested in taking what they had learned and using it themselves to support and expand civic activism. NDI needs continual engagement with deans to keep them fully informed and on board with activities at Jordanian Universities. Deans seek systematic communication and consultation with NDI to avoid surprises and build on their existing support for student civic engagement and dialogue. This is challenging, with the frequent rotation of deans in Jordanian universities. Therefore, it requires substantial continued efforts on top of the workload of managing facilitators and monitoring the work of Ana Usharek and Usharek+ groups. Deans need continued assurance that civic participation is in their interest and that it will continue to remain within acceptable boundaries in the Jordanian context. Many deans seek to continue Ana Usharek programs and are interested in controlling them for sustainability. Deans felt that their offices were capable, with support, of managing and running Ana Usharek. The evaluation’s methods did not assess the feasibility, advantages, and disadvantages of a different mode of implementation of Ana Usharek with a greater role for deans. The Parliamentary Fellows program benefitted participants. Fellows appreciated the skills and experience that they had had with NDI, which they had used for their own professional development. This was the case even when they had not been able to work inside the Parliament as expected, due to opposition from the leadership of Parliament. NDI identified women who were interested in community activism and potentially running for elected office. NDI was able to identify and bring in a diverse group of potential women candidates for training. NDI’s candidate training was valued by participants, although NDI’s candidate training had too diverse of an audience for a single training module. NDI identified a wide variety of women that were interested in community activism and potentially running for elected office. The range of experience, knowledge, and abilities of the diverse women made it hard to effectively present one standard training when faced with this variety of participant backgrounds. IRI Activities The PWD empowerment program was an effective way to expose PWD to politics, in many cases for the first time. Many, but not all, of the PWD participants had little previous exposure to knowledge about politics or civil society. So, this program was very valuable to them. The PWD empowerment 40 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV program only engaged a relatively small number of beneficiaries, but the program provided those beneficiaries meaningful first step and introduction to political life. The PWD empowerment program also helped members begin to get a sense of their own political agency. It would be an overstatement to claim that PWD in Jordan now have a genuine sense of political agency, but a few have more than they did before this program. That is a result of PWD empowerment because IRI sought out people who are frequently overlooked by other programs with similar goals. The tiered structure of YLA helped participants remain involved with the program and build upon the skills they developed. Several people who had participated in YLA group discussions reported that the long-term engagement with IRI through YLA helped them develop more skills. This approach also made it possible for IRI to build stronger ties with many of the individuals involved with IRI and, therefore, to help keep them engaged with civil and political life in Jordan. Discussions and workshops, as opposed to lectures, are by far the better way to reach people, particularly young people, in Jordan. The majority of the participants preferred the interactive approach through workshop discussions rather than the lecture approach. However, participants expressed different views about the quality of the performance of the trainers depending on their own respective experiences. Furthermore, there is a positive correlation between the respondent experiences and the perceived quality of performance of the trainer. For example, respondents who had positive experiences with YLA generally attributed that to the quality of the trainers, while those who had negative experiences had similarly negative views of the trainer. Conclusions for IFES Activities Addressing questions of accessibility for PWD to the electoral process is the kind of technical work that IFES was well positioned to do and that helped strengthen the IEC. This work helped many Jordanians with disabilities participate in elections more easily. This work also helped demonstrate the value that IFES brought to the IEC and the value that similar technical assistance can continue to bring in many areas that are not currently foci of the IEC. The IEC could have done more work around gender. Gender mainstreaming at the IEC has not yet been institutionalized. Policies of the electoral process at a country level are general and not gender-sensitive. Although they do not explicitly discriminate against women on the basis of gender, it is not possible to address issues of inclusion of women without being explicitly gender-sensitive. IFES may have missed an opportunity to harness the work of other CEPPS partners on issues of gender mainstreaming to help build up the IEC’s gender mainstreaming capabilities. CEPPS OBJECTIVE 3: STRENGTHENING POLITICAL PARTIES AND ALLIANCES AND THE ABILITY OF CANDIDATES TO ARTICULATE, ORGANIZE AND IMPLEMENT CLEAR POLITICAL ALTERNATIVES AT THE NATIONAL AND SUB-NATIONAL LEVELS IRI IRI’s assistance to its 43 partner political parties was geared both to long-term development (such as party structure, outreach to youth and local communities and policy and issue-based message development), as well as short-term but iterated support geared toward running effective campaigns during the many elections Jordan held over the course of the program. IRI worked with political parties’ leadership, branch USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 41 offices, and youth to develop issue-based platforms and communication campaigns. IRI focused on training participants to use data from IRI’s polling, as well as information participants collected directly from their interactions with their communities, to craft political platforms and outreach campaigns. Some of the major party-related activities undertaken by IRI included: • IRI periodically held planning sessions/assessments with political parties to assess their current capacity and opportunities for growth; • IRI conducted regular consultations with party leadership during the early part of the program. These meetings and consultations sought to build relationships between IRI and the political parties and to provide opportunities for IRI to advise the parties; • IRI conducted party trainings for more than 410 members, including at least 111 women and 110 youth, of five political parties on voter targeting, community outreach, elections strategic or action planning, campaigning, political communication and public speaking; • IRI provided eight trainings in strategic planning to 97 party members, including 10 women and 16 youth, from seven political parties; • IRI also conducted trainings on social media, worked with young members of political parties, and offered numerous campaign and candidate workshops and consultations; and • IRI conducted 10 workshops (four in partnership with IFES) to encourage political parties to agree upon and jointly advocate for recommendations to the government on political party law and electoral system reform. Working with political parties proved difficult for IRI. Jordanian parties in general are weak and neither trusted nor well-liked by the Jordanian population. In describing the role of political parties in Jordanian political life many interviewees, and several documents, used the word “stigma.” In addition, political parties in Jordan, throughout the period of the CEPPS program were largely leadership-driven. Even party leaders stressed this point in KIIs. Those leaders were often powerful politicians who had confidence in their political abilities. According to interviewees and IRI reports, IRI’s work with party leadership was challenging. IRI staff expressed the view that working with political parties in Jordan was extremely difficult because of the extent to which parties are heavily leadership driven, not integrated into the political life of the country. Parties often are either unknown or viewed with suspicion by most the Jordanian people. IRI partially sought to adjust to this by working with other party activists, but this was not effective because of the top-down style in the parties. Interviewees from political parties evinced sentiments that were consistent with the views of IRI. One party respondent said that the trainings IRI provided for party members were useful in terms of building up skills and technical capacities, but that ultimately, the work was not very important because of the failure to get party leadership engaged. He explained that Jordanian parties are very much defined by their leaders, so if party leadership does not get consistent, valuable and ongoing support from IRI, the party program cannot work. A former party leader suggested that one reason for the lack of buy-in by party leadership was that party leaders felt that they did not need a lot of technical support. Moreover, those that felt they might benefit from technical support were wary of any support given by a US-based and USG-funded institution. Representatives of another party indicated that they had a difficult working relationship with IRI and did not place a great deal of value on their work with IRI. For example, some political party respondents indicated that IRI used to invite people for trainings who IRI staff knew on a personal basis from political parties to their polls presentations and trainings, rather than individuals the party believed would most 42 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV benefit from these activities. One party respondent stated that they would rather have sent their research and program development teams to attend and learn from the data presented by IRI instead of the person selected by IRI. It was also mentioned by some party respondents that IRI, as an American entity conducting polling, created sensitivities and a lack of buy-in from political parties. Subject matter experts and independent political observers noted that they believed that IRI’s connection to the US particularly made political parties trust them less on matters of polling. The ET, on the other hand, only came across a few party respondents who expressed the same concern. These respondents suggested that IRI or USAID support a national CSO to provide institutional capacity building programs to political parties. Much of IRI’s work with political parties has focused directly on capacity building for parties. Respondents indicated that this engagement with political parties helped some party members develop their skills and capacities IRI engaged in a broad range of activities, including myriad trainings and workshops for candidates, party leadership, young party members and local party branches. IRI imparted useful information about parties and campaigns in these fora, but parties tended to downplay the value of this. A party’s representative mentioned that Jordanian parties needed stronger support in learning how to establish coalitions to lobby for common causes, even if the parties were not all from the same background. IRI program documents and interviews indicated that they explored engaging youth in political parties beyond the initial YLA program but determined that was not a fruitful path to follow. IRI encountered difficulty in working with young people, a demographic group with which they worked on several other dimensions, when it came to political parties. Young people who were interested in acquiring leadership skills, relationships and experience eschewed that opportunity when it was in the context of political party work. IRI tried to engage women in political parties; however, they encountered challenges due to the historically weak representation of women in political parties. Most political parties in Jordan are heavily male￾dominated, particularly at the leadership level. There are few women in any party who have meaningful decision-making power. IRI’s political party program was unable to change this. One of the party KII subjects mentioned that he agreed with the previous IRI leadership’s decision to conduct a national study on the reasons why people do not join parties in Jordan, but that study never happened. From its analysis of the status of political parties in Jordan at the start of the program, IRI decided to focus on strengthening parties into viable organizations before tackling female inclusion. This may have been a prudent strategy, but in terms of women’s political roles it leaves gaps in progress. CONCLUSIONS: CEPPS OBJECTIVE 3 IRI ACTIVITIES IRI struggled with its political party work throughout the years of the program. As was frequently noted in IRI documents and interviews with IRI staff, political parties are not popular, well run or particularly relevant in Jordanian politics. Moreover, IRI pursued several different approaches in its effort to work effectively with parties to little avail. This presents a dilemma because while there are many challenges associated with working with parties in Jordan, parties remain critical to Jordan’s democratic development. The findings clearly reflect the difficulty of working with political parties who may be suspicious of IRI specifically and US institutions generally, and USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 43 whose leaders view themselves as seasoned politicians who have little to learn. Nevertheless, finding ways to get beyond this is important, both for IRI and for Jordan’s democratic development. IRI encountered challenges in their efforts to engage youth with political parties. The YLA party program was less valuable than other YLA programs and it did not increase the role of young people in political parties. This is in part a reflection of the extent to which parties remain dominated by older, often individually focused, leadership. These problems within political parties are frustrating, but they are also the precise reason why it is so important to work with parties. IRI’s work on capacity building was helpful but did not help parties confront the larger part of the problems they confronted in Jordan. While there is little doubt that parties in Jordan can benefit from the kinds of workshops and training that were an important part of IRI’s programming, these are not the biggest problems they face. IRI’s political party work had insufficient emphasis on gender. Despite the strong gender imbalance in political parties, IRI did not adequately address this problem. Changing the gender balance in political parties is a critical issue. As parties grow in relevance, something that is central to Jordan’s democratic future, the absence of women in decision making positions in political parties will become a considerably bigger problem. CEPPS OBJECTIVE 4: JORDANIAN CITIZENS, POLITICAL PARTIES AND ORGANIZATIONS HAVE GREATER ACCESS TO QUALITATIVE AND QUANTITATIVE PUBLIC OPINION RESEARCH IRI ACTIVITIES In partnership with Middle East Marketing and Research Consultants (MEMRC), IRI conducted municipal and national public opinion polls. The three national polls were fielded in July 2012, November - December 2013, and February - March 2014. IRI polls aimed at capturing national attitudes and perceptions around key democratic, social and economic measures including perceptions on the direction of the country, economic optimism, and faith and trust in government institutions. The polls also covered special topics of interest that would arise during the polling cycles such as the influx of Syrian refugees. IRI provided access to the poll data to Jordanian citizens, political parties, and CSOs through workshops, briefings, public presentations, and media outreach. Beyond sharing poll data, IRI also worked with political parties, CSOs and other institutions to increase poll literacy and understanding of basic polling principals. As part of this work, in 2013 IRI launched an initiative aimed at increasing the capacity of local research institutions to effectively design, conduct and speak to the public about representative polling. The program ran into challenges and was stopped in 2016. According to interviews with IRI staff the challenges stemmed from a lack of appreciation of how low polling literacy was among Jordanian research institutions. Knowledge and literacy around polls and public opinion research are still low in Jordan. IRI still sometimes receives feedback that standard polling methodology is not reliable. For example, according to IRI, they have received comments from Jordanian party and other activists like: “I wasn’t polled so therefore this survey didn’t talk to everyone in Jordan.” This obviously reflects a very poor understanding of how public opinion research works and suggests that IRI’s efforts to create a larger constituency for, and understanding of, public opinion research has had mixed success. 44 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV IRI polling presentations effectively explained key ideas and principles of polling. Representatives of political parties who regularly attended IRI polling briefings indicated that the material was very clearly presented and explained, and that all questions were answered. They described IRI as “cooperative,” “good presenters” and the presentations as “clear and interesting.” Participants also said that they learned a lot from these presentations. However, further questioning during the KIIs revealed that parties had yet to substantially incorporate public opinion research into their work. IRI polls were omnibus in nature. The polls were well executed and are valuable tools, but they also appear to be part of an effort to provide information to many different stakeholders. Some stakeholders, for example USAID and other DRG partners, valued these polls and referred to them frequently in KIIS. Political parties did not see polling as relevant to their work because of Jordanian voting behavior. One party leader who indicated that IRI’s presentations were professional and accessible remarked that the information was not relevant to Jordanian political parties because of the tribal nature of Jordanian parties. Another who expressed positive views about the presentations themselves, stated in a matter-of-fact way that “we cannot apply it (polling knowledge) in Jordan.” KIIs with IRI also suggested that parties were not receptive to polling information. Two respondents from the polls’ presentation audience noted that they gained valuable information regarding women and youth perceptions, which helped them in reaching out to these groups on behalf of their parties. Based on the information they got from the polling briefings, they were better able to develop practical programs oriented to the needs and concerns of women and youth. Additionally, one of the participants drew special attention to the value of the informal networking that takes place during the polling presentations. CONCLUSIONS: CEPPS OBJECTIVE 4 Political parties’ beliefs that public opinion research is not useful for them because of the peculiarities of Jordanian politics reflects the dysfunction of political parties. Jordanian politics and voting may indeed be strongly influenced by tribe and clan, as party leaders have claimed and asserted as the reason why public opinion research is not useful for them, but elections in many countries are dominated by identities of one kind or another. This does not preclude the relevance and value of public opinion research, or of political parties, but is just another variable that the research must consider. Parties in many countries hold similar beliefs, not because the politics of their country are uniquely inaccessible to tools like polling, but because the leadership of parties are either unwilling to try new things, or the party does not have the capacity to use public opinion polling. This indicates that despite their work in this area, IRI has struggled to fundamentally change Jordanian views towards political parties or public opinion research. IRI’s presentations on polling results were strong and competent but have not led to any significant change in the behavior of political parties. This suggests that the unwillingness of political parties to use public opinion research, or believe it is relevant to them, is because of political, rather than technical, reasons. In other words, IRI has done the work of explaining polling, but has not been able to make it clear to political parties why it is important to them. Changing this will require a different approach to working with parties at national and sub-national levels. IRI’s polling has a lot of different goals. IRI’s polling work was mentioned in association with several different USAID, USG and IRI goals and projects. Over the course of our research, the ET was told that IRI polls are used so that American decision makers and donors can understand Jordanian politics better, USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 45 so that IRI itself can design better programs, to improve accountability and more communication between the Jordanian government at various levels and the Jordanian people, and in the hope of changing the behavior of political parties. This is a lot to expect from a polling activity, and it may be too much. The same poll that is useful to IRI and that helps them craft their programming may be much less relevant to a political party. Similarly, the same poll that helps the USG understand Jordan better may not be helpful for Jordanian citizens at the local level. CEPPS OBJECTIVE 5: BOLSTERING PUBLIC DEMAND FOR LOCAL SELF-GOVERNMENT AND CANDIDATES’ COMMITMENT TO OPEN AND MORE ACCOUNTABLE PERFORMANCE AS FUTURE MEMBERS OF PARLIAMENT NDI ACTIVITIES NDI’s Objective 5, “Bolster public demand for ensuring candidates' commitment to a more open and accountable performance as future members of parliament,” was used to support the 2010 effort of Al Quds to get candidates for Parliament to commit to a set of principles for parliamentary and political life, the “Contract with Jordan.” NDI reported that regional roundtables and consultations by Al-Quds led to many candidates committing themselves to the Contract if elected; a majority of MPs elected had signed the contract. This activity was long ago and distant from the Jordan of 2018; only one evaluation interviewee spoke about the pledge activity that was conducted before the 17th Parliament elections. This one interviewee noted that NDI worked with many candidates to commit to the priorities that came out of the national dialogues that we held at governorate level, Under NDI’s Objective 9, “Work to improve citizens’ understanding of local self-governance,” NDI supported Al Hayat to work with a national CSO coalition to analyze the 2013 draft law and implement a national advocacy campaign on reform through local round-tables. NDI reported that this campaign was effective and led directly to the withdrawal of this draft. Al Hayat then analyzed the next legislation and its review by the public as well as advocated for changes to this draft legislation. These changes were reportedly not taken in the final legislation, however. NDI’s engagement in this area was appreciated by civil society. The NDI grant for Al Hayat on local governance reform in 2013 was valued by its partner. Both NDI and Al Hayat felt that, particularly at that time, it was impressive for CSO engagement to be able to get the weak draft local governance law withdrawn, rather than simply passed despite its inadequacies. NDI noted that the expansion of political space to civil society to discuss a draft law had no precedent and was an important step forward for Jordan. NDI asserted that the Institute had been key in encouraging the government to hold public events to solicit information on the draft bill, which was a new and important idea for Jordan. The fact that the ideas suggested by the partner were not incorporated in the subsequent draft was seen as unfortunate but not surprising. Ana Usharek Mujtam3i The civil war in Syria led to large new refugee flows to Jordan. The international community, including the US, provided support to refugees through new funds via the USG and international organizations that focus on refugees (largely the US Department of State’s Bureau for Population, Refugees and Migration 46 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV and the United Nations High Commissioner for Refugees). Communities in the North were seen to be under stress by 2015 after four years of hosting large numbers Syrians. The continued influx of refugees to communities in the North was reportedly seen as more and more problematic by residents of host communities. USAID/Jordan decided to help Jordan respond to the challenges of hosting refugees by considering ways to supporting host community members and refugees through the existing portfolio of USAID/Jordan programs. USAID/Jordan reportedly asked that all ongoing programs consider what could be done with USAID/Jordan funds to address the refugee crisis in the country. NDI interviews reported that staff had recognized the challenges Jordanian communities faced in hosting Syrian refugees and that they seized this opportunity to engage. NDI reported that the decision to adapt and extend proven civic forum methodologies to work with Jordanians and refugees at the community level was developed by their staff, and USAID approval was reached. NDI thus developed a new Objective 11, “Work with communities hosting refugees on common concerns,” and applied methods under the CEPPS award. USAID/Jordan approved the objective and activities and provided additional resources to NDI to develop and implement a new two-year activity, the Ana Usharek Mujtam3i Democracy and Civic Education Program. Ana Usharek Mujtam3i was implemented in what NDI described variously as 12 communities or 165 neighborhoods or villages in Mafraq and Irbid; NDI adapted the Ana Usharek program, hiring and training 32 new coordinators/facilitators for the program. NDI formed a total of 225 mixed-gender groups of Jordanians and Syrians for a series of community forums to discuss and learn about nine topics: Democracy, Human Rights, Role of Media in Democracy, Citizenship, Electoral System, Local Governance, Political Parties, Gender, and Conflict Mitigation. NDI encouraged community forums for participants to apply what they had learned about these topics through practical initiatives to address local issues and reduce tensions in their communities. NDI conducted two generations of the program, the first in 2015 and the second in 2016. NDI figures report that the first generation of Mujtam3i reached 1,500 participants and the second had 2,370 graduates. NDI found participants through more than 60 local CBOs, particularly ones that were part of the Rased Coalition for election observation. This selection encouraged the most active people in the communities to engage further in civic life. Participants were almost three-quarters female; most had a secondary or bachelor’s level education. NDI reported designing the program to be mostly Jordanian to make it visible that Syrians were not taking over; facilitators also first met with Jordanian participants by group before including the refugees in the groups. The development of the program and recruitment and training of coordinators took substantial effort. NDI noted that it took six months to adapt the Ana Usharek program to the different set of circumstances and issues and to identify, hire, and train a team of coordinators and managers. This left 18 months for program implementation. NDI was proud to have developed the program and felt that project implementation processes were sensitive to the difficult context of the time. NDI interviews emphasized that NDI had adapted at USAID request in an area that they had identified as crucial on their own: contributed to addressing critical need to support community cohesion in North with growing strains in hosting Syrian refugees. NDI identified, trained, and mobilized a staff of 32 people from these communities with the background and experience to be able to deliver this program. NDI monitoring and evaluation followed conventional metrics. NDI did a program assessment in 2016 that asked participants in the first round of implementation whether the program had encouraged dialogue between Jordanians and refugees and supported activism; almost all beneficiaries agreed. Pre-post USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 47 measurements of beneficiaries demonstrate that participants asserted greater knowledge and capacity in the wake of the training. Longer-term monitoring was not conducted as the project ended. NDI found communities in the North to be quite conservative. Politics in particular was thought by many Jordanians to be a taboo subject; coordinators felt that the program helped participants “break their silence” on this topic. NDI coordinators and some Ana Usharek Mujtam3i participants reported gains in communications skills, interest in advocacy and representation, and demand for more training, especially among women participants. Jordanian and Syrian alumni appreciated increased knowledge and awareness, especially on decentralization and political system of the host country (for Syrians). Alumni said in group interviews that they had learned about civics and politics through engagement with Ana Usharek Mujtam3i. A couple of Syrians stated that by learning about the elections they felt empowered because they were able to participate in these discussions within their host community. On the other hand, in the ET’s group interviews, Jordanian alumni did not note ways that program built social cohesion or positive ties (beyond personal relationships) with Syrian refugees. Jordanian men and women in group interviews during the ET’s fieldwork were quick to speak to issues of competition and stress between host Jordanians and Syrians. Jordanians did not speak of the entire community or shared interests, but rather ways that the two communities continued to have issues. Syrian alumni noted better relationships but did not provide detail to support this assertion. NDI staff and coordinators sought to continue the Ana Usharek Mujtam3i project. Staff implementing the project felt that their work should continue. This implied a continued appreciation for the need to build cohesion and understanding of cohesion as a long-term challenge. NDI noted that building cohesion is a long-term task not amenable to a short-term project. NDI coordinators were especially disappointed to end their work with NDI. IRI Citizens Committees IRI assisted civic-minded volunteers called Citizens Committees to channel citizen priorities to municipal councilor and mayoral candidates with the aim of encouraging these candidates to incorporate citizen issues into their platforms ahead of the August 2013 municipal elections. To provide information on citizen priorities for candidates to incorporate into their election platforms, IRI helped the Citizens Committees identify top issues of concern in their municipalities and then draft platforms that addressed these issues. Combined, the Committees asked thousands of citizens what they wanted to see improved in their communities, proposed solutions to these particular issues, and then presented these as citizen platforms or manifestos to mayoral candidates for their approval and signature. By signing the platforms, the candidates understood that they were pledging to address these issues should they be elected. In total, 35 candidates running for mayor in nine municipalities and three candidates for councilor in Greater Amman Municipality signed the citizen platforms developed by CEPPS/IRI-supported ECCs. IRI worked with its volunteer Citizens’ Committees to implement local governance initiatives that brought municipal officials and their constituents together to work towards a common goal. Committee members planned activities in response to citizen concerns expressed in Baldytak surveys. Most commonly, Committee members organized clean-up campaigns, as Baldytak survey results often indicated a desire for greater cleanliness. During these activities, participants picked up trash, painted curbs, planted trees, and 48 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV created murals that encouraged civic pride and promoted cleanliness. Notably, IRI intensified these efforts in April 2015 and 2016 to make Earth Day into “Earth Month,” and held special preparatory meetings to plan this marathon of clean-up campaigns. A smaller number of initiatives centered around improving street lighting - likewise in response to top citizen priorities identified with Baldytak surveys - and during these activities, IRI purchased hundreds of street light bulbs to install at low cost. These campaigns not only allowed citizens to take pride in the appearance of their municipalities, but also provided an opportunity for citizens to directly engage with local officials, as mayors, municipal councilors, and even a Member of Parliament participated in these initiatives. To improve mayors’ and municipal councilors’ responsiveness to citizens’ concerns, IRI met with them regularly to ensure their cooperation and engagement in IRI programming, supported a variety of activities connecting mayors to their constituents and developed their ability to respond and reach out to citizens through trainings on governance skills and decentralization and exposure to best practices, including from Colombia. IRI also trained mayors and municipal councilors on running issues-based campaigns, as campaigning on issues citizens care about establishes a starting point for local government officials to follow through on their commitments made in the campaign period and be responsive to citizen concerns in their governance. Mayors had a mixed response to IRI’s work at the municipal level. One mayor asserted that he “had not seen any tangible change taking place” in his city due to IRI’s intervention. His view was that IRI gave abstract advice, was present a lot, but did not make any meaningful contribution to local governance. This respondent also noted that he had no connection with the Committee members without the IRI representative, and never received any documented presentation or a report from the Committee. Nonetheless, even this mayor said that some of IRI’s workshops were valuable. Another mayor was much more positive about the work he did with IRI, describing the programming as “well organized and successful.” This mayor also stressed that he valued IRI’s role in helping the Citizens Committee become a link between the mayor’s office and the people of his city. IRI’s work with the Citizens Committees focused on the demand side of municipal governance. This work included both workshops and trainings for members of Citizens Committees as well as activities and actions by members. Most Citizens Committee members found the workshops valuable. They described the trainings as thorough and covering a useful range of topics. Members also reported that they frequently shared their activities with the Citizen’s Committee with the rest of their community. Most Citizens Committee members believed the activities that IRI helped organize contributed to their community. These respondents said that cleaning up neighborhoods, repainting areas of the city and the like improved their communities. Members of one Citizens Committee stressed that IRI helped them develop a strategic plan that was particularly valuable. One group discussion participant said that IRI should work more on strengthening the relationship between the community and the municipality and other powerful entities as part of their exit strategy. A mayor suggested the same approach in a KII. IRI told the ET that some government officials and municipal officials have begun to use research tools like Baldytak to help guide their decisions. Baldytak is better understood as an application for managing and structuring citizen interactions with their government than as a true tool of public opinion research. Nonetheless, this indicates that among some government officials there is growing desire for citizen input to strengthen knowledge of what the priorities, goals and concerns of citizens are. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 49 Conclusions: Local Demand for Self-Governance NDI ACTIVITIES Consistent with a division of labor in CEPPS that assigned most local governance activities to CEPPS partner IRI, NDI did limited work in local governance before starting with Ana Usharek Mujtam3i. NDI assistance in local governance focused on supporting reform of the laws on local self-governance through support to Al Hayat in the early years of the award. Beneficiaries sampled for the evaluation did not note increased community cohesion. Beneficiaries interviewed by the ET reported stereotypes and competition between Jordanians in these communities and Syrian refugees that were persistent. This suggests that the model developed by the Usharek program and implemented over this short time period as Ana Usharek Mujtam3i was not able to meet the challenging community cohesion and resilience circumstances in the targeted communities. IRI ACTIVITIES IRI’s approach of working with both the demand and supply sides of municipal government was helpful. By doing this, IRI was able to identify community needs and to increase communication between the civil society and government at the municipal level. In addition, IRI was able to increase understanding between these two groups which this led to better policy outcomes and activities. IRI’s work at the municipal level was strengthened because IRI engaged in several different tactics. IRI conducted training and capacity building workshops for Citizens Committee members, advised and consulted with mayors and local government employees, organized activities such as cleaning up or repainting neighborhoods and streets, and used Baldytak to gather information about the concerns of ordinary citizens. This led to a synergy in which these different components complemented with each other. Some mayors were not happy with the municipal governance program, but their concerns seem to have risen from a more general disdain for IRI or western supported NGOs. This is inevitable to some extent, but it indicates that the success of the municipal government program depends, in substantial part, upon choosing the right mayors with which to work. For the most part, IRI chose these partners carefully. However, this inevitably limits the potential scope of the program because it precludes working in cities with mayors who are not positively predisposed to collaboration with IRI. CEPPS OBJECTIVE 6: PROMOTING THE TRANSPARENCY AND INTEGRITY OF ELECTION PROCESSES AND BUILDING THE CAPACITY OF THE IEC TO CONDUCT TRANSPARENT AND CREDIBLE ELECTION PROCESSES. NDI NDI had two separate objectives in electoral assessment. Objective 6, “Promote the transparency and integrity of Jordan’s electoral process through international assessment of the pre-election, election day, and immediate post-election period,” focused on assessment. Objective 7, “Strengthen the electoral process by identifying real or potential problems, including any irregularities, logistical, or implementation problems and impediments from external actors, and offering recommendations on how these problems can be resolved,” focused on electoral observation. In practice NDI pursued these two objectives in 50 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV tandem through the same method, international election observation missions (EOMs), and reported on them together. NDI organized three missions under the award. These assessed the preparations for parliamentary elections in 2010, 2013, and 2016 and monitored the elections as well as their immediate aftermath. NDI stressed that 2010 was the first time the Government of Jordan allowed international observation, an important precedent for Jordan. The 2016 EOM was conducted jointly with IRI. NDI has extensive experience organizing and managing EOMs around the world to support transparency and integrity of electoral processes by providing accurate, impartial information through observation considering international experience. EOMs bring former senior elected officials from democratic countries around the world, with staff to support them, to meet with local stakeholders and observe the quality of electoral and political processes. EOMs make recommendations throughout the observation process, in particular through public statements and reports at key intervals. NDI deployed increasingly in-depth international EOMs in 2010, 2013, and 2016 (jointly with IRI). These EOMs are seen as a routine in many countries but were innovative in Jordan as the first international EOM in the country was the 2010 effort. NDI was able to build on this initial experience to engage in deeper election observation in subsequent years; these featured more reach across the country and a longer time frame for the observation of electoral practices. NDI interviews noted the important precedent set for domestic and international election monitoring in Jordan through the project. The CEPPs partners reported on developing EOMs as an important part of their political engagement with key partners and stakeholders in Jordan. The use of EOMs as part of establishing credibility is understood by Jordanian stakeholders. International EOMs are seen as one of the ways to make it clear that there is a level of transparency around elections. International election monitoring was not seen by stakeholders as a large component of NDI’s work. ET interviews with former MPs, current MPs, and Jordanian partners and stakeholders did not emphasize the work of the international EOMs. EOMs is recognized as a core practice of NDI and IRI. International EOMs became more systematic and worked more in depth over time. NDI built on the 2010 precedent for EOMs to make a pre-election assessment before the 2013 elections, including deploying long-term observers of the pre-election process. In 2016, IRI-NDI conducted a joint pre-election assessment as well as long-term observers and a joint short-term election observation mission. Reporting from the international observation for 2016 was disseminated in a limited way. The final report for the joint mission with IRI does not seem to be publicly available in either English or Arabic from USAID, NDI, or IRI; this limits the use of electoral observation in lessons learned and the immediate review of electoral administration and law. This lacuna also diminishes the longer-term potential influence of international electoral observation for the further development of Jordanian elections and political competition. NDI’s support for domestic elections monitoring through the Rased coalition is appreciated by Jordanian stakeholders. Interviews with CSOs valued the NDI support for election monitoring that had built the coalition, which had monitored all Jordan-wide elections over this period. Interviews with former MPs, CSO leaders, and independent experts noted that domestic monitoring had become expected for all elections in Jordan (although some of these observers recognized that the contributions of election observation to transparent and open competition were only partial). NDI support for monitoring was seen as important in opening the door to domestic monitoring and to the initial technical capacity building USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 51 of how to monitor and report on election processes. Subsequently, NDI support has been modest financial support, reportedly only 15% of the cost of observation in 2016. Jordanian Rased partners expect to observe elections going forward. NDI has contributed to creating a new culture in Jordan of monitoring elections at all levels; now Parliament, municipalities, universities and unions are calling for monitoring of their elections. Some candidates for office felt that electoral observation had not adequately reported on practices that negatively affected electoral competition in Jordan. Some dissatisfaction with election monitoring was expressed by former MPs interviewed who asserted monitoring (both domestic and international) has not adequately publicly expressed the flaws in the practice of elections in Jordan. Some former MPs asserted that the monitoring, both domestic and foreign, did not sufficiently raise the issues with the authorities that had negatively affected their results, or even cheated them out of what they saw as “rightful” victories. These Jordanians sought more public assertiveness from international and domestic observers to support stronger democratic practice and fairer electoral competition in Jordan. IRI IRI observed the 2010, 2013 and 2016 (jointly with NDI) Jordanian parliamentary elections to ensure they were carried out according to Jordanian law and international standards and to promote the transparency and integrity of Jordan’s electoral process. In a pre-election press statement in 2016, NDI and IRI noted “the IEC has developed the administrative framework for a transparent and legitimate process. The fast-approaching election date presents significant challenges to ensuring the voters are well-informed and the election authorities have adequate preparation time. The constricted timeframe gives the IEC little time to prepare for a new and complicated electoral process, for parties to strategize and campaign, and for voter education and outreach to reach an adequate portion of the population.” IFES IFES engaged in numerous activities in pursuit of this objective, which in fact constituted the crux of their program in Jordan. The primary ways IFES sought to achieve this objective were through efforts to: • Build the IEC’s long-term institutional capacity and sustainability as Jordan’s election management body; • Strengthen the legal framework for the electoral administration process; • Strengthen the IEC’s electoral management capacities to administer national and sub￾national elections; and • Build public confidence in the IEC through the support of the IEC's public and media outreach initiatives. IFES sought to achieve these goals through several activities and approaches. These included: • Providing technical support to the IEC through ongoing direct consultations by IFES staff; • Using foreign experts for specific topics such as IT or communications; • Procuring the equipment needed by the IEC to implement elections; • Training IEC employees and Election Day workers; 52 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV • Advising the IEC about how to interpret election laws and changes to the electoral system; and • Organizing study tours and sharing best practices from around the globe with the IEC. IFES contributed substantially to the evolution of the IEC and, therefore, the improved integrity and transparency of the electoral process. When this CEPPS program began, the IEC did not exist. Today it is a functioning, if imperfect, election management body (EMB). The IFES program alone did not create the IEC but did contribute substantially to its establishment and ongoing operations. It provided extensive and ongoing technical support, having an especially significant impact during the program’s early years. One interviewee who was deeply familiar with both IFES and the IEC as well as the history of elections in Jordan summarized this by saying IFES “played a major role in bringing back the positive image of elections.” IFES technical contributions as well as the caliber of their local staff and consultants were seen as critical aspects to their success. Several current and former IEC staff stated that the local IFES staff and consultants were excellent and had pertinent knowledge both of global best practices and other technical information as well as the political and electoral context in Jordan. A consistent theme that emerged in the interviews was that the foreign consultants that IFES brought in to Jordan were not helpful. There was a perception among IEC staff interviewed that many of the foreign consultants lacked an understanding of Jordan and Jordan electoral system. From the perspective of the current and former IEC staff interviewed, the insufficiency of understanding of the Jordanian context raised doubts about the consultants’ credibility, decreased buy-in to the consultants’ efforts, and in some cases caused offense. Interviewees pointed to instances in which consultants were trying to get staff to implement tools and “best practices” from a country that the IEC staff felt was far below Jordan in terms of electoral development. While the tools may have been useful and technically excellent, this cross￾cultural miscommunication led to resistance to their implementation. This also caused some IEC staff to view all foreign consultants through a lens of incompetence, in turn making the IEC more resistant to working with them. There were challenges to effectiveness in technical support in the later years of the program, as the most valuable contributions IFES made to the IEC were in the early years of the program. Most current and former IEC staff described IFES as being extremely helpful, able to offer many relevant examples from other countries and able to provide consistently valuable advice and guidance to the IEC from about 2012 to 2014. The work of IFES was more mixed in the remaining years of the agreement, with several people from the IEC indicating they received relatively little useful guidance or advice after about 2014. Although the IEC has grown and developed in the six-plus years since its creation, IFES representatives as well as several current and former IEC officials indicated the IEC would still benefit from additional technical support in areas such as voter education, how voters are processed at polling stations, vote counting and other areas. When asked if the IEC still needed technical support, respondents who are currently working at IEC generally indicated the answer was “yes,” especially given that every election that has taken place in Jordan has had a new elections law, which required trainings and technical assistance at all levels of IEC management including technical staff, polling stations management, and the focal points in field. Furthermore, several respondents mentioned that in recent years the leadership and management of IEC participated in many trainings in and out of the country, though the technical staff were rarely given proper trainings as IFES used to do before. Several respondents who were not currently with the IEC responded emphatically that the IEC would still benefit from technical support. One former IEC staff person implored, “I beg you to continue being there…to be the watchdog…The presence of the international community in the Commission helped the development of the Commission. IFES was able USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 53 to bring best practices that were really very helpful…There is more they still need (from IFES).” A current IEC employee told the ET that the IEC “could use more help on awareness and voter education,” adding that the IEC “would like to get help on awareness campaigns around changes on election law, but as partners.” Being responsible for both procurement and technical support sometimes hampered IFES. In addition to the technical support it provided, IFES also helped the IEC procure needed election administration-related supplies such as IT equipment, meals, election materials and measuring tapes. A new EMB such as the IEC invariably has many equipment and supply related needs which, if left unmet, will make it extremely difficult to administer an election, so this procurement work was necessary. But it also introduced tensions and complexity into the relationship between IFES and the IEC. Because of the procurement policies of IFES and USAID, and the frequent need to get signoff from IFES’s Washington offices for major purchases, the pace of procurement created problems for the IEC. Additionally, over time, particularly after the first few years, interviews reveal that the IEC began to see IFES as primarily a means for procuring equipment and less as a source of technical expertise. CONCLUSIONS: INTEGRITY AND TRANSPARENCY OF THE ELECTORAL PROCESS NDI International EOMs organized by NDI helped the Government of Jordan and particularly the IEC demonstrate that they were competent in electoral administration. Processes and reports from EOMs served a valuable function by providing an opportunity for Jordanian institutions to show their electoral competence to domestic and international audiences. International EOMs were able to increase their ability to monitor elections over time. Early EOMs faced obstacles that would have inhibited their ability to meaningfully observe elections. But EOMs were able to build the constituency for their work among the Jordanian authorities, particularly the IEC. These developments suggest that Jordan has developed a more transparent electoral system and election management body that is interested in external monitoring and validation. International EOMs need to fulfill their side of the implicit bargain on transparency and produce and disseminate comprehensive final reports on their activity in both Arabic and English. Publication of immediate interim reports the day after elections on EOM activities and initial findings are important but not sufficient for the long-term development of improved electoral practices. NDI support contributed to institutionalizing an enduring domestic election monitoring coalition able to routinely mobilize and monitor elections. NDI support through Al Hayat for the Rased coalition supported the development of the organizational capacity among CSOs to conduct domestic election monitoring. Support was sustained over the elections between 2013 and 2017; this has built a culture in which CSOs expect to monitor, and candidates expect to see, CSO monitoring. This capacity has strengthened democratic practice in Jordan. IRI IRI election monitors contributed to increasing voter confidence in the election. The presence of IRI and NDI election monitors helped increase voter confidence in the polls, but also allowed USAID to see the progress made by the IEC as well as some of the remaining needs. 54 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV IFES At an important moment in the development of the IEC, IFES was able to play a very crucial role. It is unlikely that efforts to support the IEC would have been as successful if IFES has been initially brought in at a later point in the IEC’s evolution. Bringing IFES in at the inception of the IEC made it possible for the IEC to hit the ground running and not to lose time making false starts or initial mistakes. That role became more complex over time as the kind of technical support, and the relationship between IFES and the IEC, became more complex. Despite early good relationships and results, IFES and the IEC did not work as well together through the later years of the program, beginning around 2015. This is in part due to personnel changes at both the IEC and IFES, but also to the evolving nature of the technical support that the IEC needed and of the IEC’s understanding of their needs. The IEC has made substantial progress since it was created six years ago but restricting technical support now could slow down that progress. Technical support to the IEC has already been curtailed. While this is unlikely to rapidly undermine the progress that the IEC has made, the absence of this support will make it a bit more difficult for the IEC to build on previous successes and could begin to threaten existing gains. A related conclusion is that the notion of making the IEC a regional hub for learning about how to administer election can be described as premature. When asked about this, IFES personnel dismissed the idea outright, while former IEC staff did not think the IEC was yet ready for this. One interviewee observed that every EMB in the region thinks that they are the best, so the IEC would struggle to persuade neighboring EMBs that they had superior technical skill to offer. That same interviewee believed that, despite appearances, there was insufficient expertise at the IEC for such a venture. He compared this discrepancy between appearance and reality to a “village in a spaghetti western.” Foreign consultants can be valuable, but it is imperative to choose them wisely and carefully. It is apparent that foreigners can, and did, give valuable guidance and technical support to the IEC, but that came primarily from IFES staff who lived in Jordan during the program. Shorter-term foreign consultants were much less effective, according to numerous IEC respondents. At an EMB like the IEC, particularly one with a strong sense of itself and what it can do, it frequently becomes difficult for foreign consultants, working on shorter projects, to have an impact. These consultants were, according to the IEC, not qualified for what they were supposed to do, unaware of the Jordanian context and chosen with no input from the IEC. Once initial technical support had been delivered, the IEC became increasingly interested in procurement as they needed resources to do their work. This almost inevitably created stresses on the working relationship with IFES and pushed technical questions to a lower priority. Providing technical support is always difficult, but the constant distraction of procurement issues created even more difficulties. Many IFES personnel reported that by year three or so procurement issues dominated conversations with the IEC. IEC interviewees supported this idea by telling the ET about procurement-related problems and delays, sometimes in response to questions about technical support. FINDINGS: PROGRAM GOAL: STRENGTHENING THE DEVELOPMENT OF MORE DEMOCRATIC AND OPEN POLITICAL PROCESSES AND THE PARTICIPATION OF CANDIDATES, ACTIVISTS, MONITORS, AND VOTERS IN ELECTIONS The ET did not collect much data directly relevant to the program goal in the evaluation’s fieldwork; instead, most of the findings above are directly relevant to the achievement of the various objectives of the CEPPS program. The ET has identified some independent findings (in the current section) and USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 55 conclusions (in the following section), however, on NDI, IRI, and IFES activities as well as the gender work of CEPPS that fit most appropriately with the Program Goal. NDI Youth programming reaches large numbers of beneficiaries. NDI’s youth programming through the Ana Usharek family of programs has had an extensive reach across the country over more than seven years, reaching more than 25,000 students, and is recognized as an important contribution to democratic development by many stakeholders. Sustained engagement has built a sustainable domestic election monitoring coalition. NDI’s early technical support plus financial assistance over successive elections has contributed to the institutionalization of an enduring domestic election monitoring coalition able to routinely mobilize for and monitor elections. Gender Women have benefited from the national and international experts provided by CEPPS partners. NDI and IRI all brought in foreign experts whom female participants found particularly helpful. This indicates that NDI and IRI took gender into consideration when choosing consultants, thus increasing the reach and quality of their programs. Women found trainings on conducting door-to-door visits and public communication and community outreach particularly useful. Through programs such as Empower, YLA and the Ana Usharek programs, many women were exposed to these tactics that can be used in political or advocacy campaigns. Many women respondents mentioned how valuable they thought these tools were. CONCLUSIONS: CEPPS PROGRAM GOAL: STRENGTHENING THE DEVELOPMENT OF MORE DEMOCRATIC AND OPEN POLITICAL PROCESSES AND THE PARTICIPATION OF CANDIDATES, ACTIVISTS, MONITORS, AND VOTERS IN ELECTIONS IRI IRI provided information, resources and skills to women, young people and PWD that makes it more possible for them to engage in political processes. Several IRI activities brought people into political life, in many cases for the first time, by providing them with essential skills and knowledge. In some cases, IRI beneficiaries included populations that are generally very difficult to reach, such as low-income women living in poverty pockets or PWDs. This made their work very valuable, but also limited the scope of their work somewhat, as these groups are difficult to reach and require a substantial commitment of resources. This work is only indirectly involved with elections as many of these people will not run for office, but many have become involved in political campaigns and advocacy groups in their communities. IRI election monitors have increased the transparency of elections. When this CEPPS program started, there were no real foreign election monitors. IRI monitors have played a growing role in international election observation in Jordan, contributing to the transparency of elections. 56 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV IFES Through its work with the IEC, IFES improved the electoral process while strengthening confidence in it and made it easier for people to vote. When this program started, Jordan had no independent EMB, as elections were run from the Ministry of the Interior. In significant part due to the support of IFES, the IEC, founded in 2012, developed into a competent EMB. The IEC is imperfect and should work to improve in some areas, but over the last six years, it has grown and developed substantially and has been instrumental in helping build confidence in the electoral process among the Jordanian people. The IEC changed Jordanian policies and became much more welcoming to election monitors. Today elections in Jordan are monitored by competent domestic NGOs as well as by NDI, IRI and European organizations. However, before this program foreign election observers were not allowed in Jordan. In the early years of this program monitors were allowed but given little access. That has changed. GENDER Although both NDI and IRI also had programs targeting women specifically, they also considered gender questions in other areas of their work. By including gender considerations when choosing foreign consultants and study trip participants, NDI and IRI were able to make some progress towards mainstreaming gender in their programming. This suggests a meaningful commitment to gender equality in their programming. Helping women develop a battery of community outreach-related skills allows them to become more involved in their community and in political life. IRI and NDI helped women develop these skills that can be used for many different political and community related endeavors and for a broad range of issues. This is significant because many of these tactics (for example, going door to door or using social media for community outreach) require women to take a public position and to make themselves more visible. These are important first steps to developing political skills, running for office or assuming influential positions in the government. EQ2: HOW DID THE STRATEGY AND IMPLEMENTATION APPROACH ENHANCE OR WEAKEN ACHIEVEMENT OF THE INTENDED OUTCOMES? ARE THERE CERTAIN AREAS/ACTIVITIES AND APPROACHES THAT HAVE BEEN MORE EFFECTIVE? WHY? Each of the CEPPS partners is a strong, well-established United States civil society organization that works all around the world. The CEPPS partners have strategies and implementation approaches that they often pursue in countries around the world. The main approaches and underlying strategies behind them used by NDI, IRI, and IFES in Jordan are evaluated here based on the fieldwork done for the evaluation. The section on EQ2 provides findings on the strategy and implementation approaches used by NDI, IRI, and IFES in implementing the CEPPS award. The three partners’ individual strategies and implementation approaches are taken in order: NDI, IRI, and IFES. The conclusions section provides the ET’s analysis of these main findings by CEPPS partner. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 57 NDI STRATEGIES NDI noted the benefits of working through a strategy of working with the most interested, willing participants. Participants in NDI’s programs are self-selected (for example, some university students choose to participate in Ana Usharek); the programs do not reach all members of a demographic group (such as all university students). NDI managers suggested that given the challenging context for strengthening the development of more democratic and open political processes in the Kingdom, NDI chose to work with key groups of Jordanians that were likely to be most receptive to NDI’s approaches. Program participants were those that chose to engage with NDI across the range of programming (university students, school teachers, fellows, MPs). This strategy of working with Jordanians who are most interested in advocacy and civic engagement was asserted to be appropriate given the difficult context in the region and in Jordan for democratic development. NDI management and staff note the value of connecting programs. NDI programming has developed incrementally, “from one success to another” as some staff explained. This is apparent, for example, in the development of the Ana Usharek “family” of programs from the initial adaptation of NDI’s civic forum model. This model of development was seen to build on learning and staff experience, as well as continue to expand the benefits that accrued to some participants. This is apparent in the more advanced skills development and experience that some Ana Usharek alumni received as these strongest participants were selected for Usharek+. Working with the most interested Parliament members and most active committees helps increase the effectiveness of program implementation, but this approach has effects on the scope and scale of assistance. NDI’s work with Parliament did not reach the entire institution, but instead the parts of Parliament that were most interested in working with NDI. This did not produce ways for the institution to own the programming or expand the tools and techniques used by NDI more broadly to other non￾assisted committees or members. NDI’s engagement with the Ministry of Education has the advantage of working with state institutions. The Ana Usharek schools program is implemented in a different way through the MoE. NDI was able to get their buy-in. The MoE felt that the program fit within the Ministry since it was based on MoE curriculum and structure. By connecting participation to the bonus and promotion structure for teachers, the MoE built in incentives for teachers to participate. There may also be challenges of working with state institutions that were not discussed by MoE interviewees or NDI. NDI has blended the approach of working through their own staff for the Ana Usharek family of programs and Parliament with an approach of working through the leadership of partner civil society organizations. The NDI program thus works in two different modes: through NDI staff and through sub￾grants. The partnership and sub-grant component is smaller, some 20% of the total effort. The focus on own staff and the need for strong direct management of decentralized staff may make it a challenge to focus on building partnerships and the capacity of government and CSO partners. IRI’S STRATEGIES A significant aspect of IRI’s approach was that they sought to use research and data to craft and improve their work. During KIIs with the IRI team, IRI explained to the ET how they used their internal evaluations and research, including public opinion research, effectively to determine the strategic directions for their 58 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV programs and activities. This included using a range of tools including national public opinion data, evaluation tools meant to capture the views of participants at various trainings, workshops and reflection sessions to discuss the evaluations by both participants and trainers. By using these M&E tools to improve their programming in an ongoing way, IRI has helped make their programs stronger, but has also made programming less subjective. IRI’s approach makes it less likely that their programming will be a reflection of the impressions or ideas of a single IRI person and that there will be more continuity in IRI’s programming even when they experience leadership changes. IRI trainers found that one-on-one mentorship over a longer-than-average program time frame, one year, was instrumental in the effectiveness of the Empower program and in engaging poor and historically disenfranchised women in civic engagement and politics and public life. This finding applies specifically to the Empower program, but it also demonstrates the need for a flexible approach to programming. The Empower program sought to work with women living in poverty pockets who for social and economic reasons were unlikely to participate in or benefit from larger trainings or similar activities. The individual treatment and support these women got from the Empower program was absolutely essential to the success of that program. The ET conducted group discussions with Empower participants who stressed that the IRI trainers cared about them as individuals. One even described the IRI trainers as “grand, awesome, pleasant and down to earth.” IRI found that improving local governance and civic engagement was best addressed by working on improving supply (mayors and staff) and demand (Citizens Committees) together. IRI’s work on the local level employed a multi-level approach. They worked with both local government and civil society in several cities to improve governance, increase accountability of local elected officials and demonstrate the value of democratic processes at the local level. They also used Baldytak, an app developed for mobile phones that allowed local elected officials to collect information from citizens and citizen’s groups. Baldytak made it possible for mayors to get a better sense of what their constituents wanted and what their priorities were. IFES STRATEGIES IFES’s strategy of being housed within the IEC helped them establish close working relations with that institution, at least at first. During the early years of the IEC, a period when IFES provided a great deal of technical support, having IFES physically at the IEC made it possible to build strong relationships between IFES and the IEC and for IFES to more effectively implement trainings and other programs while at the same time being available to answer questions that might arise. Several people who worked at the IEC at that time said that senior IFES people had very deep knowledge of electoral processes and were therefore able to answer virtually any question that arose. Similarly, IFES staff indicated that in the early years one of the reasons that IFES was able to be effective was because it was relatively easy to have access to different people at the IEC. Some of the interviewees mentioned that one of IFES’ national staff members was not sufficiently professional. They mentioned that they reported this to IFES’ regional office but nothing had changed. By the end of the program, as the IEC became less interested, according to IFES, in the expertise IFES had to offer, it was less useful to have them housed at the IEC. People who worked at the IEC during this period also indicated that by around 2016 they felt no need to have such intense interaction with IFES. Combining the technical support and procurement work for the IEC in one organization raised some challenges. This became evident through KIIs with both IFES and IEC staff. IFES personnel noted that by USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 59 midway through the program, discussions of how to provide technical support, what kind of technical support was needed, or the details of training events were frequently sidetracked by the IEC concerns about procurement. In the views of IFES, the IEC was too concerned about procurement to focus on possibly valuable technical support. This view was mirrored by IEC personnel who even when asked in a KII about IFES training quickly turned the discussion to procurement. In some cases, this was to cite the value of IFES’ procurement assistance, but in other cases, the IEC cited procurement related problems, primarily around the speed of procurement. Procurement is always difficult because of IFES and USAID policies around procurement, frequently requiring multiple bids, and usually taking some time. The items that IFES procured, or sought to procure, for the IEC were frequently important to the work the IEC was doing, but this sometimes created more conflict. IFES representatives noted that challenges related to procurement were mostly concentrated during the final year of the program and often related to IEC requests that had timelines that were not feasible and would have required IFES to bypass USG procurement regulations. For example, one senior IEC official described how printed material needed for an IEC project needed to be delivered by a given date, but IFES could not move quickly enough. Some of the respondents were concerned of the transparency and the integrity regarding IEC taking direct financial assistance for procurement. They insisted that this is critical support for elections that can threaten the elections integrity of the process if any of the procedures were not followed properly. CONCLUSIONS: STRATEGY AND IMPLEMENTATION APPROACHES NDI STRATEGIES NDI’s strategy of working with most interested parties produces expandable programs. NDI’s approaches seek out interested Jordanian counterparts, rather than work with all members of a category of counterpart. This strategy can be and has been affirmed, as the most interested beneficiaries seek to continue in these areas of their interest. This is most evident, for example, in continuing to work with the most interested Ana Usharek alumni through Usharek+ (and hiring some as Ana Usharek coordinators or even NDI staff in Amman). NDI’s approach of working through own staff presents challenges to sustainability and Jordanian ownership. Most of the implementation of NDI’s program is done by NDI staff. NDI can and has successfully used this implementation approach to develop and expand the institute’s programming. This is most evident in the Ana Usharek program, which is directly managed by a network of project staff. However, this implementation approach depends on NDI’s management and systems. It does not operate through local institutions (with stronger implications for sustainability), in contrast to programs that operate through government systems (like Ana Usharek schools and the Ministry of Education) or parliamentary monitoring (through Al Hayat and Al Quds). There is a desire for greater Jordanian ownership in the areas where NDI works from CSOs, universities, and Government of Jordan institutions. Interviews with NDI stakeholders and partners demonstrated interest in more institutional routes to Jordanian engagement in NDI’s areas of assistance. While interviews with these organizations noted appreciation for NDI’s work, counterparts noted that their institutions had less at stake in these programs than NDI. These Jordanian stakeholders felt that they or their institutions were capable of taking the examples and programs of NDI and managing them going forward. The ET did not assess the capabilities of partners or their capacity to manage these programs with less NDI engagement. 60 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV IRI STRATEGIES For IRI, strategy was very important, but this sometimes limited the scope of their work. IRI used many different tools, including activity evaluations and public opinion research, to inform their activities and program direction. In general, this led to positive outcomes and effective work, but it also is a process that in addition to being resource-intensive, sometimes made it more difficult for IRI to engage in programs that were larger in scope. The major reason for this is that the strategic approaches favored by IRI tended not to be easily scalable. For example, the Empower program was very labor- and time- intensive, but that was probably necessary given the women with whom Empower worked. IRI’s political party work was different, but the party representatives with whom the team conducted KIIs indicated that IRIs most important party work occurred when they worked closely and intensely with political party leaders. This would also be difficult to scale up. IRI’s decision to eschew, for the most part, larger, more frequent trainings in favor of smaller, more intense ones was central to their work. Several of IRI’s most significant programs, such as the PWD Empowerment, Empower and YLA programs were built around ongoing work with a relatively small number of groups of participants. This strategic approach was appropriate given the nature and constituents of each of these programs. However, even when working with easier-to-reach groups of people, this approach is still the more valuable one. Across IRI programs, participants appreciated the ongoing relationship with IRI and IRI trainers and indicated that they believed that IRI genuinely cared about them. This is valuable because of the long-term nature of CEPPS involvement in Jordan. It is not possible to know what will come of the relationships and goodwill that this approach, in addition to being a better way to communicate information, will bring, but it is likely in the long run to be helpful. IRI needs to address the question of party work. IRI reduced its political party work as the myriad difficulties associated with working with parties in Jordan became clear. These challenges are indeed daunting, but there are also problems associated with not working with parties. Parties, even in a country like Jordan, play an important role in politics and democratic reform. Given the amount of resources that CEPPS has committed to supporting democratic elections in Jordan, party work is particularly important. Parties, as IRI pointed out on many occasions, are weak in part because of the power of tribal or clan￾based identities, but this is also why parties are essential if democracy is to move forward or if elections are to be meaningful in Jordan. In a system like this one, weak parties mean that the choices offered in elections are less distinct, thus inadvertently strengthening other identities. IFES STRATEGIES Physical proximity made it easier for IFES to establish a strong working relationship with the IEC in the early years of the IEC. Because the IFES project was focused almost entirely on supporting the IEC, and because the IEC was founded only in 2012, housing IFES at the IEC was very helpful. In the early period of the IEC was such that being physically present made it easier for IFES to identify IEC needs as they emerged, provide advice on both major and minor issues and to be able to answer questions informally as they arose. However, over time this became less helpful as the IEC became less interested in the kind of support IFES had to offer. By the end of the IFES project, their presence at the IEC may have contributed to the less than ideal relations between the IEC and IFES. Too frequently, efforts to provide technical support were sidetracked by concerns and questions about procurement. There is a logic to linking technical support and procurement as the two ideas are related. Technical needs drive procurement needs, so having on organization do both can streamline the USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 61 procurement and also act as a check to make sure that the procurement needs are real. However, because many beneficiaries are often more interested in concrete procurement-related needs than in technical support when one organization does both, technical support is usually not the focus of the beneficiary. This is what occurred with regards to the IEC and IFES. This was made more difficult because many at the IEC felt that procurement was too slow, making it more difficult for them to do their work effectively. Although the goal of the IFES program was to enable the IEC to become a competent and independent EMB, the close of the IFES program came too soon. There is no question that by 2016, the relationship between IFES and the IEC had frayed and needed to be restructured. The advances made by the IEC with help from IFES, as well as the desire within the IEC to be more independent, meant that the period of intense cooperation had run its course. Nonetheless, given the remaining technical needs, the close of the IFES assistance came too soon. Another program can be established but time will have been lost. A better approach would have been to reduce the IFES program, house it somewhere else and build in a different technical assistance partnership with the IEC. EQ3: HOW HAS THE PROGRAM ADAPTED TO CHANGES AND HOW HAS COLLABORATION WITHIN CEPPS AND OTHER DRG PARTNERS/DONORS INFLUENCED ACTIVITIES AND THE ACHIEVEMENT OF RESULTS? This section provides findings from the evaluation’s field work on the two key themes of the EQ: adaptation and collaboration. The section first examines adaptation and NDI’s approaches before examining collaboration by NDI, IRI, IFES, and between the CEPPS partners and other USAID partners. The section ends with conclusions based on the ET’s analysis of these accumulated findings. ADAPTATION NDI NDI has had tremendous program growth through modifications over 2010 through 2017. The initial award evaluated in this evaluation was a January 2010 grant of $1.4 million through CEPPS for NDI to implement a 15-month program. The final Modification 21 to the award in July 2016 funded NDI’s program through June 2017, a seven-and-a-half-year program at a level of almost $30 million. NDI demonstrated a willingness to develop new programs with the new opportunities that developed. This willingness to take on new challenges was apparent in the expansion of the programming in areas where activities were already underway such as parliament but also for new programming in the wake of the Arab Spring in the region in new areas such as universities. NDI was also willing to expand to additional substantive areas such as community cohesion. NDI adapted a worldwide NDI model for advocacy and discussion, the civic forum model, to Jordan and then expanded this model. NDI initially developed Ana Usharek to work at universities, and then expanded this program to more universities as well as to support stronger participants from Ana Usharek through Usharek +. NDI also revised the Ana Usharek curriculum designed for university students in order to reach school children through Ana Usharek Schools. The program was developed for grades 5 to 7 and 8 through 10 in public schools in Jordan. NDI then expanded the number of schools, both boys’ and girls’ schools, reached by the program. Finally, NDI developed the Ana Usharek Mujtam3i initiative to reach and harmonize relationships between communities hosting Syrian refugees and the refugees within them. 62 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV NDI staff and Ana Usharek/Usharek+ coordinators/managers noted ways that the feedback from participants led to the development of specific new modules based on student interest. Staff noted that the youth programming changed over time based on feedback from students at the end of Usharek sessions. This entailed adding new modules at the start of a new year’s implementation. NDI’s initially larger portfolio of CSO partners became more focused over time on Al Hayat, Al Quds and a set of core activities: election observation and parliamentary monitoring. NDI became more concentrated in its civil society work in Jordan. NDI’s work with CSO partners declines over time in the number of CSOs that are reached by the Institute. This adjustment focused NDI’s assistance on two core areas, in contrast to the expansion of NDI’s youth activities above. NDI attributed this change to the development of other USAID approaches to civil society strengthening, USAID guidance that NDI should therefore be less engaged in civil society, and NDI’s assessment that some CSOs were better partners than others. NDI continued to rely heavily on its own staff for program implementation throughout the awards. NDI developed Ana Usharek through a model that hired and trained NDI staff to be the trainers and coordinators of groups of University students. The expansion of the program took the same form, with Usharek + and Ana Usharek Mujtam3i (while the Ana Usharek Schools program used a smaller staff to train school teachers at both boys’ and girls’ schools in Jordan). IRI Over the course of the CEPPS program, IRI deemphasized its political party program as it grew increasingly apparent that placing a lot of effort into party work in Jordan was not fruitful. The IRI program began with a strong political party component, but over time political parties became a smaller part of IRI’s portfolio. IRI claimed some successes with parties, helping them “improve their way of thinking and develop their skills,” as one IRI staff person phrased it, but over time this work became more difficult. One political party leader identified the crux of the problem IRI encountered as being that the parties are so leadership￾driven that anything short of direct and consistent work with the party leader was not going to be effective. IRI as well as USAID representatives and other observers also noted the very bad reputation (the word that was used frequently IRI’s written reports was “stigma”) facing political parties in Jordan. This made it very difficult for IRI to do effective political party support despite efforts to revise their program, work with different parties and other changes. Given the slow progress, IRI indicated that USAID asked that they stop working with parties in order to conduct research to identify more effective approaches for engaging political parties in Jordan. IFES Many at the IEC reported that coordination between IFES and the UNDP was not smooth. There was a lot of competition between these organizations, leading to the IEC having to spend time coordinating the assistance between these organizations. UNDP had a technical support program at the IEC for most of the time IFES was there. Because UNDP is not a USAID-supported organization, the existing USAID￾supported implementing partner coordination mechanisms were not in place. This contributed to very poor coordination between IFES and UNDP that created problems and was time-consuming for the IEC. Several IEC staff reported that they were never clear as to which organization was responsible for what, and that the two organizations often appeared to be in competition to host activities and to enhance their visibility. The coordination fell to an already very busy IEC. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 63 IFES struggled to adapt to changes in IEC. The IEC underwent leadership changes; it also simply developed and grew as an organization during the course of the program. Major leadership changes at the IEC created problems for the relationship between IFES and the IEC because the new leaders had different styles and were, in some areas, less interested in working with IFES. Additionally, over time the IEC became stronger and more independent. While there were still critical issues that required IFES support and no major changes to workplans were executed by USAID or IEC leadership, it seemed apparent that IEC staff wanted a different relationship with IFES. Several IEC interviewees indicated that at first IFES acted as mentors or teachers but were not able to adapt to the role of partner, which is what IEC wanted from them in the later years of the program. COLLABORATION WITH OTHER USAID PROJECTS OR OTHER DONORS Interviews found that most of the coordination within CEPPS and between the CEPPS partners with other DRG partners happens informally at the Country Director or Chief of Party (COP) level. COPs noted to the ET that there were formal channels of coordination across DRG partners mainly during the period of elections planning. The formal coordination is directed by the USAID DRG team. Informal coordination was more frequent but did not reach further into organizations than COPs or Country Directors. The three CEPPS partners worked well together and have had little to no issues of coordination. The CEPPS partners have substantial experience with dividing issues and partners in countries around the world to avoid problems of overlap. IFES has a clear mandate within the CEPPS partnership to do electoral administration assistance, which IRI and NDI do not do. Of the two political party institutes, IRI has developed specialization in areas of technical assistance, such as public opinion polling. This specialization has been used as part of focusing the portfolios of the two organizations in Jordan. Close communication between IRI and NDI at level of Country Director was noted in ET interviews. CEPPS partners and other DRG IPs generally had few conflicts with coordination and collaboration. Interviews with COPs and with CEPPs did not find issues with coordination or collaboration. IRI and NDI interviewees reported working in areas of democratic development and then reducing their engagement in these areas as other USAID programming focused on these areas, such as civil society organization capacity building for NDI and local government for IRI. There is a desire from USAID and other DRG partners to increase collaboration and coordination with CEPPS partners. This coordination was seen as desirable in general, including in areas outside of election periods. DRG partner COPs interviewed by the ET noted that their programs might benefit from closer engagement and learning. The expectation was that their programs could benefit from working with the partners that NDI and IRI had worked with over their long-term engagement in Jordan. As long-term grantees, NDI and IRI were recognized to have been continuously engaged in Jordan, unlike the shorter￾term project-based support of DRG’s partners under contracts. NDI leadership asserted that there was sufficient coordination already. Other than that, no one else was satisfied with the level or quality of coordination and instead saw this area as one for potential improvement. 64 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV CONCLUSIONS: ADAPTATION AND COLLABORATION NDI NDI has continued to rely heavily on its own staff for program implementation throughout the awards. In the 2010-2017 award and particularly in the current award, the preponderance of activities is implemented by NDI’s own staff rather than through Jordanian CSOs or government institutions. IRI IRI was able to adapt in some respects, but sometimes this led them to limit their programs. During the period of the program, the evolving political environment in Jordan led to some openings, for example with regards to election monitoring missions. IRI was adept at changing their programming so that they could take advantage of these changes. Similarly, IRI’s work with PWD grew out of the recognition both that this constituency was underserved by existing programs and that other channels, notably party work, had become more difficult. The other side of this adaptability was that some of the more difficult work, again mostly around parties, was ultimately reduced significantly. This occurred because the work was decreasingly fruitful, not because it wasn’t needed. There is a logic to this, but it also can be taken too far. IFES The IFES program was established for the early period of the IEC and struggled to adapt to a changing dynamic after these initial years. From 2016 on, the IEC no longer felt it needed, and very clearly did not want, the kind of strong, hands-on guidance that had previously been so valuable. IFES was not able to adapt to that smoothly, but the nature of the program and the beneficiary made that particularly difficult. Coordination between IFES and UNDP was an ongoing problem. IEC respondents spoke of overlapping responsibilities and poor communication between IFES and UNDP. Several IEC respondents said that the IEC had to spend a lot of time and effort coordinating the efforts of these two technical support providers CEPPS Collaboration between the CEPPS partners and USAID DRG Partners has been driven by USAID DRG. The coordination in the USAID DRG portfolio discussed in the ET’s fieldwork has come from USAID, which has called meetings for information sharing (particularly around the elections). COPs and USAID interviews both noted these meetings, their value, and their limitations. Interviews suggested that there was potential value to moving beyond information-sharing and building on the work of other programs. Collaboration between the CEPPS partners has been mainly at the COP level. The COPs of the CEPPS partners routinely appear to have collaborated and shared information. The only joint activity noted, however, was the 2016 international EOM. EQ4: WHICH INTERVENTIONS ARE MOST LIKELY TO SUSTAIN OVER TIME (AND WHICH WILL BE DIFFICULT TO SUSTAIN)? WHY AND HOW? WHAT USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 65 SHOULD BE CONSIDERED TO ENHANCE SUSTAINABILITY AND LOCAL OWNERSHIP? The section develops findings from the ET’s fieldwork on sustainability. Findings are organized by CEPPS partner (NDI, IFES, and IRI). Accumulated findings are then analyzed by the ET to determine key conclusions on sustainability for each of the three CEPPS partners which conclude the section. FINDINGS NDI NDI’s approach has used NDI staff for project implementation with youth. NDI’s development and expansion of Ana Usharek and Usharek + is implemented exclusively by NDI staff. NDI has obtained the support of universities to implement its own activities but not enlisted university staff in implementation. This use of NDI staff and management of these staff has come at a cost to national ownership and sustainability. NDI partners recognize that NDI’s implementation of projects itself does not support CSOs’ building their own relationships and sustainability. Jordanian CSOs recognize that much of the NDI portfolio is not implemented through CSOs or Jordanian institutions. NDI’s CSO partners that implement programming funded by NDI have built their own relationships that enable their work to continue (e.g., in parliamentary monitoring and election monitoring). NDI support for CSO partners in election and parliamentary monitoring has enabled sustainability. NDI’s sustained technical and financial support for Al Hayat and Al Quds has allowed for enduring effects of this assistance. It is now expected that Jordanian CSOs will do election monitoring (even with only modest financial support from NDI) and parliamentary monitoring. CSO partners continued to depend on donor funding – including through CEPPS – for engagement in elections and policy dialogue. Former CSO partners in this area report doing less research and having less potential impact on policy without NDI assistance. These former CSO partners seek support for policy research. University deans think they could sustain Ana Usharek if funded. Some university deans interviewed were interested in having their universities operate Ana Usharek and Usharek+ through their own management and staff. These deans noted that they would need financial and technical support to pursue this national ownership. A potential funder noted was the King Abdullah Fund. This capacity to fund or manage without NDI has not been tested to date. Interest and institutional capacity may make the Ministry of Education a potential owner of Ana Usharek Schools. The ET’s interview with the MOE suggested that the relevant department in the Ministry was interested in owning and operating civics education using the training techniques and manuals of the Ana Usharek Schools program. The capacity of the MoE to sustain the program has not been explored or tested to date. 66 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV IRI The Citizens Committee program may be able to sustain itself somewhat as committees now frequently identify and recruit new members, according to IRI. IRI reported that over time Citizen’s Committees in various cities began to recruit new members on their own. In one city where the ET conducted a group discussion, Citizens Committee members reported that they had recruited a “few” new members. This reflects that members of the Citizens Committees understand what they are doing with IRI as valuable and that the Citizens Committees are viewed positively in their communities. This qualifies as only partial sustainability, because even if Citizens Committee membership can be renewed with little effort from IRI, new members will still need, and want, the trainings and workshops that IRI has provided to the Citizens Committees. IRI representatives have indicated that IRI understands this issue and plans to develop the capacity of new members of the Citizen Committees based on the requests and plans of the Citizen Committees themselves moving forward. The skills and capacities that participants in various IRI youth-oriented programs have gained will go with them if they continue to be involved in political life. The YLA program, Empowerment and PWD Empowerment, as well as to lesser extent the Citizen’s Committee programs all sought to build capacity and skills among their members. Programs like these help individuals develop skills that can be used throughout their lives in various endeavors. Better leadership skills, communication ability or other general skills can be used in many different kinds of endeavors and are by no means just limited to politics. Thus, there will almost certainly be some sustained impact from this part of IRI’s program, but that impact will be diffuse and difficult to measure. For many of the participants, they will simply have a few more skills with which to go through life. Seeking to improve sustainability, IRI shifted to working with mayoral staff as well as elected mayors. Mayors are susceptible to losing elections, moving to a different government position or souring on their relationship with IRI. By working only with mayors as part of their municipal government project, IRI was putting a lot of weight on individual relationships. After a few years of that, IRI decided that rather than working just with mayors, they would work with mayoral staff directly and in coordination with local Citizen’s Committees. Thus, the capacity that IRI was building was being delivered to a larger range of individuals. IFES The IEC was a functioning and competent EMB towards the end of the IFES program and remained one after IFES’ work concluded. The EU monitoring mission that observed the 2016 parliamentary election, which took place during the end of the IFES program, wrote that “the Independent Election Commission (IEC) delivered a well-administered and inclusive election.” Additionally, the IEC successfully administered the 2017 local elections after the completion of the IFES program. This is an indication that the IEC can competently administer elections now. This notion was reinforced both by KIIs with IEC and IFES, but also through observations of the IEC offices. The creation and development of the IEC is a clear and positive indicator of some of the work that CEPPS did in Jordan. Most KII respondents indicated that the IEC would still benefit from some technical support. The sustainability of the IEC does not mean that it is as technically strong as it could be. In this regard, while the IEC is itself sustainable, and indeed sustaining, technical support for the IEC is USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 67 not sustainable since IFES left. Current programs supporting the IEC, funded by the EU, do not have the impact, according to the IEC, that IFES did during its early years. If continued technical support to the IEC does not occur, the sustainability could be threatened. CONCLUSIONS: SUSTAINABILITY AND LOCAL OWNERSHIP NDI NDI depends heavily on its own staff for program implementation; this poses challenges for sustainability and national ownership. Implementing most programming through NDI staff comes at a cost for sustainability and national ownership. While most staff are Jordanian, the program is implemented through NDI systems and procedures rather than national counterpart organizations, with the exception of NDI’s work with Jordanian CSOs. NDI support for parliamentary monitoring and elections monitoring through Jordanian organizations has built sustainable practices among Jordanian CSOs. NDI’s work with Al Hayat on parliamentary and election monitoring (the Rased coalition) and Al Quds on parliamentary monitoring have been sustained over years and electoral cycles. NDI has supported a culture of election monitoring and organizations and a network that expects to monitor all elections going forward. These CSO partners now have this capacity and receive what they consider to be modest levels of funding from NDI for parliamentary monitoring. These organizations anticipate continuing these endeavors. Jordanian counterparts in areas where NDI works through its own staff are potential partners. NDI programming implemented by its own staff is implemented in areas that are managed by Jordanian institutions in many cases. Jordanian institutions are potential partners that could implement programming with NDI support and assistance. There are potential Jordanian partners that are interested in continuing NDI’s work. Interviews noted interest by Jordanian counterparts in taking a larger role in or managing the kinds of programming implemented through NDI staff themselves. University deans expressed interest in managing Ana Usharek and Usharek + and the MoE expressed interest in managing Ana Usharek Schools. Parliamentary leaders expressed an interest in working along more institutional lines with NDI to support parliamentary strengthening. IRI Citizens Committees, with some more support, could become an important and enduring part of municipal governance. In some cities, according to IRI and members of the Citizens Committees, new members of those committees are being recruited directly by existing members. This suggests a continuity within the Citizens Committees that is not dependent upon IRI, and is clearly a reflection of the role that the Citizens Committees play in these communities. Institutionalizing civil society organizations that liaise between local government and the people who live in those areas is way to improve both democracy and governance. However, IRI and USAID should be careful not to project too much onto this conclusion. There remains a need for IRI to be engaged in the Citizens Committee program so it can build a stronger foundation and link it to the strong women from Empower as well as the PWD program. 68 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Although IRI programs will need continued support from USAID, the longer-term sustainability will likely be real, but hard to measure. Several major IRI programs have invested heavily in building the capacity of individuals, primarily young people, PWD and women. Some of the beneficiaries of these programs are already becoming more involved in civic and political life, but the extent to which these programs have sustainability will depend on these people continuing to be involved, presumably in more influential roles and positions. Thus, the true sustainability of these programs cannot be known yet, but in other countries there have been cases of beneficiaries of similar programs remaining involved in politics and playing a positive role in civic life. Working with municipal staff is valuable but getting buy-in from more mayors would also be helpful. IRI’s decision to work with mayoral staff as well as mayors helped ensure that IRI was reaching more people, able to weather a mayor leaving office and making it easier for Citizens Committees to work with municipal governments. Nonetheless, mayors, particularly in smaller communities where they might not have large staff, are the lynchpin of municipal governance. Currently, IRI’s program depends upon mayors being cooperative and interested in reforming how the government works. This necessarily limits the municipalities where IRI can effectively work. Therefore, building and maintaining more relationships with mayors will be important as this program moves forward. IFES IFES is no longer in country, but the IEC is still functioning and employing key skills and concepts gained during the IFES program such as the ability to properly assess polling stations. Given this, the IEC is currently well positioned to administer Jordan’s next elections. This is, in an almost literal sense, the definition of sustainability. Although IFES’s last years in the country were not always smooth, and the relationship between IFES and the IEC was frequently rocky during that period, the continued existence and high functioning level of the IEC is a testament to the sustainability of IFES’s interventions, particularly during the 2012-2014 period. The IEC still has areas where it needs technical support, but with IFES out of the country there is no clear way for them to get that support and guidance. This suggests that while the institution will continue, it will not be able to grow and develop as quickly or get the support it still needs, thus indicating that there are some limits to the sustainability of IFES’s work. More significantly, the absence of ongoing technical support threatens to undermine the sustainability of the IEC. If the IEC does not receive further technical support, there is a danger that they could stop improving or that their technical capacity could begin to backslide due to changes in key personnel or the absence of high-quality experts who can help with new challenges. EQ5: WHAT ARE SOME KEY LESSONS LEARNED THAT CAN INFORM THE ACTIVITY AND THE MISSION GOING FORWARD? Evaluation question 5 first outlines findings from the ET’s fieldwork that are directly relevant to questions of lessons learned. The findings from document review, interviews, and the survey on lessons learned are listed by CEPPS partner. This section finishes with the conclusions from the ET’s analysis of these accumulated findings, listed by CEPPS partner. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 69 NDI NDI’s M&E, analysis, and reporting has had little to say about lessons learned. NDI’s reporting mechanisms have not been used to develop or disseminate lessons learned. For example, the Final Report for the 2010-2017 CEPPS award includes a section on lessons learned with no NDI lessons. However, NDI interviews noted ways that NDI had adapted and changed in the course of developing programming under the CEPPS award. The substantial adaptation in the program may be driven at least in part by lessons learned, but these lessons do not seem to be presented to USAID in ways that can be transferred to partners for broader potential impact. NDI asserted plans for analysis and learning as part of expanding Ana Usharek, Usharek +, and Ana Usharek Schools in its 2017 proposal and award. NDI developed plans to conduct research on its programming and develop knowledge products as part of its proposal for the current award. NDI’s monitoring mechanisms focused on pre-post comparisons that measure whether attitudes or behavior had changed over the period of working with NDI on selected activities (such as Ana Usharek or Usharek +). These methods do not assess whether attitudinal or behavioral changes are enduring. NDI reporting and interviews emphasized that sustained efforts and a long-time period were needed to build support for change in context of Jordan. Interviews with MPs concurred in areas such as creating a Women’s Caucus, which was seen to have taken persistent efforts. NDI also noted that given the culture in Jordan and electoral system, developing a cadre of women with campaign techniques that can choose to run and potentially win in elections was a long-term proposition. NDI’s long-term engagement helps support organizations and practices that are likely to persist. Consistent NDI support for Al Hayat and Al Quds in the areas or parliamentary monitoring and election monitoring was seen by these organizations to have developed a culture of monitoring that would persist even without NDI support. IRI Overall, making it possible for IRI to be a true partner so that IRI could develop its own strategy and ideas has led to better programming. During the eight-plus years of this program the political context and needs in Jordan evolved and changed. IRI sought to stay ahead of this through a research-driven approach to program development. Respondents from IRI stated that USAID was a very supportive partner that encouraged IRI to develop appropriate programs and adapt to changing political realities. This was essential for the successes achieved by IRI. IFES The success of IFES’ work with IEC was due to the timing of their work with the IEC, demonstrating that intensive work with EMBs works well in the early stages. Because IFES was present from the beginning of the IEC they were able to make critical early interventions and rapidly develop a sense of the ongoing and evolving technical needs of the IEC. In some respects, IFES was indeed phased out from its work with the IEC, but that was due primarily to tensions in the working relationship between the two organizations, rather than 70 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV through any planned action. As the relationship soured, IFES did less work with the IEC until eventually the program was completed, but this was not a structured or strategic phase out. CONCLUSIONS: LESSONS LEARNED NDI NDI appears to recognize the need for stronger M&E, analysis, and stakeholder engagement in programming. There is substantial room for longer-term monitoring of the influence of NDI’s work on participants as well as for comparison through randomized trials and control groups of attitudes and behaviors of NDI beneficiaries compared to Jordanians that have not benefitted from NDI’s programming. Research through experimental or quasi-experimental methods would yield significant learning. Given the relatively substantial resources required to conduct such research, it may be necessary for USAID to support such an initiative. NDI, USAID, and Jordanian partners and stakeholders would benefit from stronger analysis and learning products based on this analysis. The M&E under the past award does not provide a basis for measuring longer-term influence of NDI programming, for example on youth, or expanding/contracting NDI programming based on evidence of enduring effects or their absence. IRI The structure of IRI’s agreement and its relationship with USAID made it possible for the program to evolve, adapt and take advantage of opportunities as the context in Jordan changed. Many of the lessons learned from IRI’s work in Jordan have been addressed elsewhere in this report but is still significant that KII respondents from both IRI and USAID observed that the relationship between the two organizations was strong and fruitful. This is particularly important given the nature of the work IRI did throughout these years in Jordan. Working in the Jordanian political space requires a flexibility that is less essential in other areas. It also requires constantly processing new information and political realities and adjusting programs accordingly. In general, this freedom allowed them to thrive. IFES The major lessons that can be learned from IFES’s work in Jordan involve timing. IFES was able to play a valuable role, but a key lesson from looking at the IEC now is that EMBs need technical support beyond their first three or four years. Additionally, organizations offering that support need to change the ways they do that over time. Because the IFES contract came to a close, the IEC lost out on valuable time they could have been using to improve their vote counting protocols, voter outreach, gender programs and other important components of running elections. Similarly, by seeking to continue to support IEC after the commission’s fourth year as it had in earlier years, IFES was not sufficiently sensitive to changing attitudes at the IEC. V. RECOMMENDATIONS The USAID Evaluation Policy states that recommendations should be action-oriented, practical, and specific. Based on the findings and conclusions provided in this report, recommendations were generated collaboratively with USAID to ground-truth their applicability to Agency procedural and resource realities. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 71 The extensive list of recommendations is for consideration regarding CEPPS at this stage, and for future Agency investments of a similar scope. The emphasis is on lessons learned and areas where improvements can be made. A. FOR SUPPORTING THE IEC AND ELECTION MONITORING A1. USAID should renew technical support to the IEC, but not with the same structure and intensity of the IFES program. The IEC is unlikely to welcome assistance on the scale or style that IFES provided in the past, so future support needs to have a lighter touch and less of a full-time presence at the IEC, while still providing needed technical support. USAID should also make certain that any international support to the IEC is done in the spirit of partnership and is respectful of what the IEC has already accomplished. A2. USAID should fund an independent needs assessment of the IEC. This will help define what kind of technical support will be most helpful. An independent assessment, not by the IEC or IFES, could help move past the controversy over who provides assistance to what kinds of support the IEC needs to fulfill its strategic plan. It would also be an opportunity for the leadership of the IEC to become more comfortable with their need for additional technical support, thus making it easier for whatever organization is eventually brought in to do that work. A3. The organization offering technical support to the IEC should not be responsible for procurement. Based on the experience of IFES and the IEC from 2012-17, it is evident that the IEC is not positioned to work on technical issues with an organization that is also providing physical goods for them because the need for materials always seemed more urgent than the technical support. This made it extremely difficult for IFES and the IEC to craft and implement technical support activities and strategies. It should be noted that, by separating procurement and technical support, adding layers of bureaucracy may be necessary to ensure both types of support are carried out effectively. But it would allow whichever organization is charged with the technical side of the project to focus on that. A4. USAID should support a gender mainstreaming strategy for the IEC as well as a full￾time gender specialist at the IEC to help incorporate gender related issues into the work of the IEC. This will insure that gender issues become mainstreamed and a priority for the IEC that are incorporated into all aspects of the IEC’s work. A5. A process should be created so that the IEC is involved with selecting any foreign consultants with which they will work. In general, the IEC found the foreign consultants selected by IFES to be ineffective and of little value. However, many reported that the longer-term IFES staff were in some cases quite helpful. This suggests that foreign experts still have something to offer the IEC, but that they need to be selected with more care. If the IEC is involved in working with the technical support provider or donor to identify the foreign consultants, the IEC will have the opportunity to help identify their needs as well as review resumes and provide input on the selection candidates who they think are most likely to fit those needs. The technical support provider would retain the final authority to select the consultant to ensure that contractual and financial requirements are met, and there is an acceptable level of impartiality from the consultants. A6. USAID should continue to support both domestic and foreign election monitors. Recent elections in Jordan have been observed by strong domestic and foreign efforts. This has been critical for 72 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV building confidence in the electoral system and making the electoral process more democratic and transparent. Continuing to support these efforts is essential for continuing to improve elections in Jordan. A7. Rather than seek ways to quickly grow the IEC into a regional training body, USAID should focus on shoring up the gains the IEC has already made. On several occasions people at the IEC indicated that they were interested in developing the IEC into a regional training center for other EMBs in the Middle East. This is an ambitious, and perhaps worthy, goal but it should be understood as a longer-term project rather than something for the near future. USAID should encourage the IEC to strengthen itself, continue to get the technical support it needs and focus on improving the quality of elections in Jordan before turning its attention to broader regional goals. A8. USAID should create formal coordination structures between any organization it supports to assist the IEC and existing international groups doing the same. USAID was able to create valuable coordinating structures for members of the CEPPS consortium as well as for the larger group of recipients of USAID funds. However, these structures were not applicable at the IEC. IFES was the only USAID-supported organization providing significant support to the IEC, so there was no question of coordination of USAID funded organizations working with the IEC. However, there were other organizations working at the IEC supported by other donors, specifically the UNDP and later the EU. Interviewees at the IEC were in a virtual consensus that these groups did not sufficiently coordinate with IFES, thus creating extra work for the IEC, unhelpful competition between the groups and in some cases overlapping programs. If USAID restarts its support for the IEC, it should include structures, such as regular meetings with donors and implementers working with the IEC. USAID cannot force other donors to attend these meetings, but it is in everybody’s interest to coordinate better and USAID is well positioned to take the lead on that. B. FOR IMPROVING COORDINATION B1. USAID should create opportunities for CEPPS partners, as well as other DRG partners, to coordinate at levels below that of Chiefs of Party. The formal coordination structures between USAID partners only engage Chiefs of Party. While this means that Chiefs of Party know what other USAID partners are doing, this information does not always get to other people within these organizations. Creating opportunities for coordination below the Chief of Party level through, for example, meetings of people holding similar positions, such as Deputy Chief of Party or M&E Officer, would alleviate this problem. Opportunities for senior and mid-level staff from USAID partners to hear a talk or a presentation would also be helpful. C. GENDER, YOUTH AND PWD C1. USAID should support programming that seeks to build the capacity of women in government ministries and agencies. Women are underrepresented in government ministries and agencies, but the women who already work in government would benefit from capacity building and developing skills that would both strengthen their ability to do their work and strengthen ties to the communities they represent or serve. By supporting programs like this, USAID will make it possible for these women to be more easily promoted and assume increasing influence and decision-making power. C2. USAID should support efforts to develop a code of conduct that endorses women, youth, refugee and PWD-specific needs. This should be done in partnership with CSOs, media, USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 73 parliamentarians and other key stakeholders. A code of conduct among CEPPS partners would ensure that the needs and interests of all these key constituencies are incorporated into all programming done by CEPPS partners. It would also help standardize best practices and other key considerations across these organizations. C3. IRI should track YLA, Empower, PWD Empowerment participants for years after they finish. IRI implemented several activities that built capacity in individuals in the hopes that in the coming years they will begin to play a role in Jordan’s political and civic life. Determining the extent to which the knowledge and skills that were transmitted to these people was sustained can only accomplished by tracking these participants not just for two or three years, but for significantly longer. Additionally, tracking these beneficiaries over the years can help inform DRG programming more broadly because it will show the benefits, or lack thereof, of working closely with small groups of people. It will also allow USAID to get a better sense of how their interventions influence the political engagement of individual citizens. C4. IRI should develop phase II programs for youth, women and PWD programming. In addition to tracking the involvement of these participants in politics and civil society, IRI should also create next-phase programs for PWD empowerment and Empower. These programs already exist for YLA, so in that case alumni programs would be helpful. IRI invested substantial time and resources into PWD Empowerment and Empower and made these investments intensively in a relatively small handful of people. Given that, continuing to work with these people, thus increasing the chances of them becoming, or remaining, active in politics, would be helpful for development outcomes and cost-effective. D. POLITICAL PARTIES AND PUBLIC OPINION D1. IRI should restart its party program, but with more modest, attainable goals and less intensive activities. IRI’s political party program encountered numerous difficulties over the years. Moreover, political parties are not well liked, or even well-known, in Jordan. It is true, as IRI has indicated, that they are extremely leadership-driven, disliked by most people and lacking cohesive visions, goals or ideologies. This does not change the reality that political parties are an integral and essential part of democracy. Without parties, legislatures cannot function well, and elections become about patronage, ethnic and tribal loyalty and worse. For these reasons, IRI should pursue a new political party program, but one that is grounded in more modest goals and that begins with activities in which parties will participate. This might include things like multi-party meetings to discuss issues or things like elections or to hear from a qualified foreign expert. It could also include regular meetings with a small handful of party leaders either individually or in groups, or other similar activities. A major goal of all of this would to increase dialog with political parties and to build relationships and gain the confidence of those parties. D2. IRI should work with some political parties and relevant government offices to reform the political party law. The current political party law creates little incentive for political parties to compete in, and do well in, elections. A new party law could ensure that parties that are committed to political engagement are recognized and given resources, while others are not. This would be an important first step to helping political parties overcome the many barriers they currently face. IRI could do his by hosting roundtables and dialogues both among parties and between parties and the government. D3. New IRI political party programming should have a clear gender component. This can be done by developing gender-sensitive party curricula that unifies and clarifies the stages of political party development from the perspective of gender. This could include institutionalization, community outreach, 74 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV messaging, and development of practical programs. Women have been particularly receptive to this kind of approach in other IRI activities, so applying this with regard to political parties would be helpful. Additionally, working directly with female party leaders may be a useful way to reframe the political party program. D4. IRI should determine goals for public opinion research program. IRI’s public opinion polling program sought to do many things, including providing information about Jordanian political opinion to the USG; helping IRI craft its programs; helping political parties develop their programs; and familiarize Jordanian political parties and CSOs with the value of polling as a political tool. This had led to IRI investing a lot of effort into public opinion research and achieving several partial successes. It would be helpful to clarify what the purpose of the IRI polls are, and to limit that to one or two major goals. Once that happens, the polling program can be refined and improved for maximum effectiveness. E. LOCAL GOVERNANCE E1. IRI should craft a strategy for working in municipalities with less cooperative mayors. IRI’s work on municipal governance brought together CSOs (i.e., the Citizens Committees) and local government, primarily through individual mayors, to strengthen governance. This approach was productive, but it too frequently depended on the goodwill of the mayor. Mayors who were open to working with IRI and interested in improving governance were drawn to the program, while others were not. This is natural and was addressed somewhat when IRI began to work with mayoral staff as well. Nonetheless, IRI was limited by only working with cooperative mayors, so the program was in little value to people who lived outside of these cities. IRI should find a way to bring more mayors, although some likely will never be interested, into this program through directly building relationships with a larger pool of mayors. F. CIVIL SOCIETY ENGAGEMENT IN MORE DEMOCRATIC AND OPEN PUBLIC POLICY PROCESSES F1. USAID should work through NDI to support the engagement of more CSOs in a variety of areas of engagement in policymaking. The civil society portfolio of NDI has become focused on Al Hayat and Al Quds for targeted activities in parliamentary monitoring and support for electoral monitoring through the Rased coalition. Civil society organizations should be encouraged to have more expansive roles in public policy and greater engagement with the public around policy. To stimulate more engagement from a larger number of CSO partners and greater diversity of CSO activities that encourage more democratic and open political processes, NDI should develop and manage a competitive process providing grants to civil society organizations seeking to encourage public discussion, debate, and input into public policy making. The size of awards should be determined by the capacity of these organizations, and the magnitude of their engagement in policy processes should be limited to what is manageable for CSOs in the remaining period of implementation of the current CEPPS award. NDI should be encouraged to award as many grants to different organizations as possible to increase the engagement of individual CSOs with constituents and in their communities in public policy making. The competition should not prescribe permissible areas for CSO engagement but instead leave the choices of areas to engage in up to CSOs. NDI should encourage many CSOs to make proposals. F2. NDI should consider developing and holding networking and information-sharing events for civil society organizations engaged in public policy or civic engagement. NDI USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 75 could further develop democratic and open political processes by encouraging CSO interactions and sharing knowledge between organizations with shared interests in public policy and working more with citizens. NDI could serve as a convener of sessions that bring together organizations in the sector around information sharing, networking, and new ideas and practices. This engagement could include networking them with government counterparts or international CSOs. F3. USAID should encourage NDI to continue to engage with Jordanian CSOs to maintain and support a culture of parliamentary and election monitoring. Through USAID-support, NDI has developed the capacity and practice of Jordanian CSOs and networks in the monitoring of parliament and domestic election monitoring. These two areas of CSO engagement have become established practices that are key contributions to more democratic and open political processes in Jordan. NDI should continue to support this capacity by supporting ongoing parliamentary monitoring and maintaining the ability of Rased to mobilize for domestic election monitoring. G. PARLIAMENTARY STRENGTHENING G1. USAID should work with NDI to assess the opportunities for working with Parliament and Members of Parliament in a comprehensive way and develop activities that both strengthen Parliament as an institution as well as the activity of diverse blocs and individual Members of Parliament. USAID and NDI have substantial experience working with Parliament and Members of Parliament. Approaches have worked with Parliament as an institution and with individual members that have different perspectives from the Speaker and Secretary General. NDI should consider what is feasible in the remaining period of the successor award in a comprehensive way in both of these different tracks. NDI should then develop a few priority areas to develop more democratic and open processes with the leadership of the institution and to support individual Parliamentarians and blocs. H. SUSTAINABILITY AND NATIONAL OWNERSHIP H1. NDI should consider a broader range of approaches to support sustainability and national ownership for all of its programming. NDI programming has depended heavily on NDI staff and the personal engagement of expatriate leadership with Jordanian counterparts to develop and maintain political space for program operations. These modes of operations do not support program sustainability through Jordanian ownership. NDI should systematically explore how to increase the roles of Jordanians in the more political aspects of NDI engagement. NDI should explore ways to hand over program successes and learning to sustainable Jordanian organizations. I. SCHOOLS PROGRAMMING I1. NDI should work towards and prepare to hand over Usharek Schools to the Ministry of Education. By working with the Ministry, Usharek schools could be expanded through national ownership to have a larger footprint across the country. NDI should work closely with the Ministry of Education to develop the constituency for the Ministry to increase its role in the Usharek Schools program. NDI should work with Ministry officials to develop plans for the Ministry to incorporate civics and the lessons of Usharek Schools into the Ministry’s curriculum and programs – or even take over and incorporate the program (or a revised version) into schools across Jordan through the Ministry. 76 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV J. ENGAGEMENT WITH UNIVERSITY YOUTH J1. USAID should work with NDI to increase the sustainability of NDI’s work with university youth by encouraging national ownership. NDI should explore and test alternative methods to incorporate Ana Usharek and Usharek+ programming into Jordanian institutions to encourage national ownership and sustainability. NDI should prepare to hand over program materials and the capacity for implementing and managing Ana Usharek and Usharek+ to Jordanian counterparts with the potential to sustain these programs with reduced or no USAID funding. The processes of exploring, testing, and eventually handing over the capability and responsibility for civics education for university youth should be transparent to key Jordanian stakeholders. USAID and NDI should develop roles for civil society organizations and leaders in the management, monitoring, and reporting on Ana Usharek and Usharek+’s activities and results. J2. NDI should consider how to expand the number of Ana Usharek and Usharek+ participants beyond those already interested in civic participation. The reach and size of NDI’s youth programming through Ana Usharek and Usharek+ have grown substantially. However, the program still is implemented in ways that focus on reaching the most interested youth at universities. NDI should explore and test ways to increase the range of students that participate in Ana Usharek to provide useful knowledge and experience in civic engagement to students that are not as interested in civic engagement. J3. NDI should increase efforts to network Ana Usharek alumni and strengthen program implementation to encourage current alumni networking. NDI has reached more than 25,000 university students through Ana Usharek since starting program implementation in 2012. This engagement has the potential to create enduring links around the value of and benefits from civic activism. NDI should increase its efforts to maintain the links of youth reached by the program to NDI and to each other through an alumni network. NDI should work to reach out to and create a network of past participants. NDI should develop structured ways to continue to reach current student participants after their Ana Usharek training has ended. J4. NDI should conduct a thorough review of its Ana Usharek and Usharek+ experience to deepen student engagement in the program. NDI asserted that the Institute would conduct a review of its programming as part of the CEPPS follow-on award for 2017-2020. NDI should conduct systematic research into the work of the programs and their effects on short- and long-term civic engagement of participants and alumni. The results of this research should be used to strengthen the impact of the program. K. WOMEN CANDIDATES AND PROSPECTIVE CANDIDATES TRAINING PROGRAMS K1. USAID should work with NDI to develop training programs for prospective and declared candidates for public office that target the varying experience and capacity levels of prospective candidates. NDI has had extensive experience in supporting the aspirations of women to run for elected office. Jordanian women vary in their levels of preparation to participate in electoral processes. To better support the range of women that are interested in running for electoral office, NDI should develop and implement more than one set of trainings. NDI’s Jordanian candidate training programs should adapt trainings by differentiating according to women’s existing experience and needs for training. Trainings should be conducted at different times in the electoral cycle, targeting different USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 77 skills and practices as well of levels of experience and capacity. Women with less experience need training to further encourage their interest and start building their campaign skills long before the period of elections. More experienced and skilled potential candidates need more advanced skills targeted to particular times in the electoral cycle as well as individual meetings and consultations. L. TRANSPARENCY AND OPENNESS L1. USAID should work with IRI and NDI to broaden the public understanding of USAID’s support for strengthening the development of more democratic and open political processes in Jordan. USAID, NDI, and IRI should routinely translate and make publicly available materials on program design, implementation, and results on their own websites and those of their Jordanian partners. The knowledge of what USAID, NDI, and IRI are doing to support Jordanian institutions and individuals engage in public and open discussion and debate should be more widespread in Jordan. Jordanians need to understand the extent and content of USAID-funded development assistance to support more democratic and open political processes to demystify these activities and encourage more participation and partnerships. The tools and techniques developed with the support of USAID should be made more easily available to support interested individuals and groups in Jordan. M. MONITORING, EVALUATION, AND LEARNING M1. NDI should consider whether its management structure can be strengthened to better support learning and adaptation from program implementation. NDI has continued to operate through a Country Director and Deputy Director, and a division of responsibilities around technical areas of the program. This organization has been retained regardless of whether the program was small in size or had large numbers of program coordinators, leading activities in a decentralized manner across Jordan. NDI should consider how its organizational structure could more strongly support monitoring, evaluation, and learning. NDI should strengthen dissemination of program learning to the public in Jordan. NDI should also be encouraged to support innovation and the testing and evaluation of alternative program approaches to support the development of more democratic and open political processes in Jordan. M2. NDI should conduct research on the longer-term effects of Ana Usharek and Usharek+ and incorporate the results of this research into program implementation to seek to have more enduring effects on alumni behavior. NDI has implemented civics education and activities in Jordan since 2012. Monitoring and evaluation tools have not analyzed or tracked the long-term effects of sustained engagement with university youth and the large numbers of youth reached through the programs. The survey of youth beneficiaries of NDI and IRI programs conducted as part of this evaluation suggests a number of unexplored puzzles about program alumni attitudes, behaviors, and the links between them. NDI should develop and carry out more research into the effects of civic engagement programming and make this monitoring research publicly available. These survey results can be used to strengthen the impacts of the two programs. Research through experimental or quasi￾experimental methods would yield significant learning. Given the significant resources required to conduct such research, it may be necessary for USAID to support such an initiative. 78 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV ANNEX A. EVALUATION STATEMENT OF WORK INTRODUCTION USAID/Jordan requests an external performance evaluation of the Consortium for Elections and Political Processes Strengthening (CEPPS) program, with a total value of $53.3 million, covering the performance period of January 2010 – June 2017. Through this period, this Activity was implemented by the National Democratic Institute (NDI), the International Republican Institute (IRI) and the International Foundation for Electoral Systems (IFES). USAID/Jordan also requests the evaluation of the decentralization elections support efforts provided under the follow-on CEPPS activity, which was recently re-awarded, covering the performance period of July 2017 – December 2020, with a total value of $19.2 million. The Activity is now implemented by NDI and IRI. A separate direct Government-to-Government (G2G) award of $1.5 million was provided in May 2017 to the Independent Election Commission (IEC) for the administration of the Decentralization elections which was increased to $2.05 million in August 2017. BACKGROUND AND ACTIVITY DESCRIPTION Jordan has been on a sluggish but steady course towards political reform. Over the last 10 years, Jordan has implemented various structural reforms that aim at engaging a greater proportion of citizens in the political process. The country’s unique experience during the Arab Spring has produced the first amendments to the constitution in decades and a new legal framework for elections and political parties. Most recently, parliamentary elections were carried out in 2016, and local elections were carried out in August 2017 under new municipalities and decentralization laws ushering in new councils that can bring change through more effective community engagement. However, regional developments, the influx of Syrian refugees, and a struggling economy continue to cast long shadows over the social, economic and political landscape in the country leaving it vulnerable to conflicts across its borders. A key goal of USG foreign policy in Jordan is to help ensure that Jordan becomes increasingly responsive to citizens and supports civil and political rights. This goal requires supporting the development and consolidation of increasing pluralistic, fair, broad-based, and representative elected institutions. CEPPS contributes to USAID/Jordan’s current Country Development Cooperation Strategy (2013-2019), as amended and extended, under Development Objective #2: “Democratic Accountability Strengthened” and Special Development Objective #4: “Gender Equality and Female Empowerment Enhanced.” In particular, it contributes to IR 2.1 “Accountability of, and Equitable Participation in, Political Processes Enhanced,” and IR 2.3 “Civil Society Engagement and Effectiveness Increased.” The CEPPS Activity is intended to support domestic election monitoring, increase participation in election processes, and train candidates and political parties in effective campaigning and polling, and provide technical and in-kind assistance to the Independent Election Commission (IEC). USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 79 CEPPS activity, which is the primary focus of this evaluation: • Development Objective: Democratic Accountability Strengthened • Activity Title: Consortium for Elections and Political Processes Strengthening • Award Number: 278-A-00-10-00407 • Leader Cooperative Agreement NO. DFD-A-00-08-00350-00 • Award Dates: January 2010 – June 2017 • Funding: $53.3 million • Implementing Partners: NDI, IRI and IFES • AOR: George Kara’a • Alternate AOR: Talar Karakashian Follow-on CEPPS activity, where decentralization elections support work was provided during the first few months of the award: • Development Objective: Democratic Accountability Strengthened • Activity Title: USAID/Jordan Elections and Political Processes • Project (EPP) • Award Number: AID-278-LA-17-00001 • Leader Cooperative Agreement No. AID-OAA-L-15-00007 • Award Dates: July 2017 – Dec 2020 • Funding: $19.2 million • Implementing Partners: NDI and IRI • AOR: Talar Karakashian • Alternate AOR: George Kara’a PURPOSE OF THE EVALUATION The objective of this evaluation is to provide USAID with findings and strategic recommendations related to the effectiveness of the Activity’s interventions and implementation approach. The evaluation will inform USAID’s decisions on the future course of work under existing activities to better integrate current programming with other parts of the democracy and governance portfolio, inform future Mission strategy, and to better support the Jordanian-led reform agenda. Specifically, the evaluation will focus on assessing efforts to build capacity of the IEC, electoral administration, awareness raising and citizen engagement in the electoral process, especially women, youth and Persons with Disabilities, capacity building targeted towards national and local government representatives and communication between the elected representatives, the executive and citizens. The evaluation will also assess the most recent effort by partners around the Decentralization elections in addition to the program’s overall progress towards achieving intended objectives and document lessons learned and best practices to inform future DRG programming. OBJECTIVES AND THEORY OF CHANGE The CEPPS program aims to strengthen the development of more democratic and open political processes in the kingdom and, specifically, to support the participation of candidates, activists, monitors, and voters in elections. To this end, the program worked to achieve specific objectives including: 80 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV ● Improving the ability of civil society organizations (CSOs) to build grassroots demand and effectively advocate for a new legal framework for elections; ● Developing avenues to engage youth in the election process and civic engagement; ● Strengthening political alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels; ● Encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls; ● Bolstering public demand for candidates’ commitment to open and more accountable performance as future members of parliament; and ● Building the capacity of the IEC to conduct transparent and credible election processes. Success in electoral management, and the provision of assistance to electoral managers, does not only depend on preparation for electoral events. Electoral assistance is more effective if delivered during the complete electoral cycle as it enables election management bodies to identify and apply lessons learned from previous electoral processes in the country itself. Through long-term programs, electoral assistance providers and election managers are better able to improve key elements that help ensure credible elections. Institutional management structures and strategic plans can be reviewed and improved. Professional development programs can be developed and implemented to advance the knowledge and skills of its permanent and temporary staff. Legislation, regulations and executive instructions that govern elections can be reviewed and improved. Better quality operational plans and operational training programs can be developed for specific elections. And, perhaps most importantly, effective public outreach and awareness programs can be developed to involve all electoral stakeholders in the electoral process and boost the trust of all stakeholders in the electoral process. An empowered and effective Parliament will build trust among constituents and promote accountability with the Government. Public opinion of political parties may begin to change if parties can focus on issues of importance and talk about solving them, rather than about personal attributes. If youth are more aware of opportunities within the political sphere and have an increased understanding of political engagement, then they are more likely to participate in political activity. Local elected officials will be better equipped to deal with the pressures of managing limited resources and growing demands on municipal services, and better positioned to handle decentralization when that opportunity arises, if they engage in constructive dialogue with citizens about priorities and needs. Women will be more likely to vote their conscience if they have access to information about the voting process and know their vote is anonymous and will be more likely to eventually run for office, if they are encouraged by and learn from other independent women in elected office and other leadership positions with similar backgrounds. Faith in Jordanian public institutions will increase when citizens have a deeper understanding of democratic processes and decision-making structures; when processes for legislative drafting and appropriation of public funds are more transparent and reflective of input from civil society and the citizenry; when Parliament and the government are held accountable through a combination of mutual oversight and robust civil society monitoring; and when barriers to the political participation of marginalized populations, such as women and youth and persons with disabilities, are reduced. Furthermore, electoral processes will gain legitimacy with the adoption of a new, fairer legal framework. Grassroots, civil society-led advocacy campaigns will help ensure that the new election-related legislation takes into account input from a broad range of stakeholders. Tensions between Syrian refugees and host Jordanian communities will decrease if both groups have a platform for dialogue and joint civic engagement through which they can address issues of mutual concern. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 81 EVALUATION QUESTIONS In addressing the stated purpose, the evaluation will explicitly answer the questions stated below: EFFECTIVENESS & RELEVANCE 1. What is the overall effectiveness of CEPPS for achieving its goals and objectives? 2. How did the strategy and implementation approach enhance or weaken achievement of the intended outcomes? Are there certain areas/activities and approaches that have been more effective? Why? 3. How has the program adapted to changes and how has collaboration within CEPPS and other DRG partners/donors influenced activities and the achievement of results? SUSTAINABILITY 4. Which interventions are most likely to sustain over time (and which will be difficult to sustain)? Why and how? What should be considered to enhance sustainability and local ownership? LEARNING 5. What are some key lessons learned that can inform the Activity and the Mission going forward? EVALUATION DESIGN AND METHODOLOGY The external evaluation will use quantitative and qualitative approaches to assess how effective the Activity was in meeting its objectives and to provide recommendations on how to make future interventions with similar objectives more effective and more sustainable. In responding to the evaluation questions, the evaluation team should highlight, to the extent possible, demographic differences- e.g. sex, age, geography, socioeconomic status, and others as deemed relevant. Special attention should be placed on gender and youth. Data collection and evaluation methodology should include: 1. Review of secondary data including basic program documents, such as the Cooperative Agreement, monitoring and evaluation plan, annual work plans, quarterly reports, list of deliverables, surveys and polls, training materials, and others. 2. Focus groups and individual interviews with key stakeholders and beneficiaries of key activities and initiatives; for example, the Empower Initiative for women, Citizens Academies, Youth Leadership Academy and Ana Usharek groups. Interviews should also be conducted with USAID staff, USAID implementing partners4, relevant CSOs, IEC officers and elected and non-elected government officials. 4 IFES team no longer has an office in Jordan; some of the team members are currently working for IEC or other USAID or donor Activities. It may be necessary for the evaluation team to interview IFES staff and solicit documentation from IFES HQ in Washington, D.C. 82 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV 3. A questionnaire or guide of key questions for the interviews should be used to ensure consistency in data collection. Rigorous data analysis methods should be used to ensure evaluation questions are addressed and data analyzed. 4. Quantitative survey of respondents from the various program activities, such as beneficiaries of the Ana Usharek, Usharek+ and Ana Usharek Mujtama’i as well as select beneficiaries of other USAID programs implementing election-related interventions. This survey should gauge beneficiary perceptions on the effectiveness, relevance, and sustainability of CEPPS and other USAID programs related to elections. Additionally, the evaluation team should consider the use of other available quantitative data such as the national and local level surveys conducted by IRI, and the upcoming General Population Survey being developed by MESP. GENDER AND SOCIAL INCLUSION In line with USAID’s Gender Equality and Female Empowerment Policy and Automated Directives System (ADS) 205, USAID/Jordan’s DO4, the evaluation should consider the gender-specific and differential effects of CEPPS. The evaluation team will seek to go beyond simple male/female comparisons, and analyze variation within gender groups, as well. The evaluation team will disaggregate data by gender at multiple points in the causal pathway to analyze the potential influence these effects have on the aforementioned outcomes. Different types of female respondents, including widows, young unmarried women, single mothers, affiliation with tribes, wives in male-headed households, and women in female-headed households, will be targeted for their perspectives, when possible. Additionally, in line with Mission and Activity priorities, the evaluation should consider the effects on and perceptions of youth, persons with disabilities, and other marginalized groups. EVALUATION TEAM COMPOSITION In order to meet the requirements of team composition, ensure data quality, and contribute to building capacity of local evaluation specialists, the following is suggested for team composition: 1. Team leader (subject-matter experience, USAID evaluation experience) 2. Subject-Matter Expert (Local Jordanian expert and/or international expert with country-specific experience) (up to 2) 3. Evaluation Specialist (USAID evaluation experience) 4. Gender and Social Inclusion Expert 5. USAID Political Participation Expert The MESP M&E Specialists, Evaluation Assistants and Chief of Party will also support the evaluation team. A political participation expert from the USAID/DRG team in Washington may join the evaluation team, especially for the design and data collection phases of the evaluation. If confirmed, USAID will cover all expenses related to the participation of the USAID staff member. Larger data collection (qualitative and/or quantitative), in case necessary, will be conducted by Mindset, MESP’s data collection partner. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 83 PERFORMANCE PERIOD The evaluation will be conducted from January 2018 through May 2018, with initial design of the evaluation developed in January – February 2018, data collection conducted in February/March 2018 and final report submitted by April 2018. Logistics for the assessment and evaluation will be provided by MESP, except for the costs associated with the participation of a USAID staff member. DELIVERABLES AND TIMELINE MESP finalize SOW, in-brief meeting via video or in-person upon arrival where definitions of the evaluation areas and the SOW will be discussed, begin desk review of CEPPS, develop work plan. January 2018 In-brief meeting with Mission In-brief meeting with CEPPS Partners Field Work March 2018 MESP develop evaluation methodology and tools, finalize work plan and submit evaluation design report for feedback from Mission and CEPPS partners; final approval will be provided by USAID Field Work March 2018 Debriefing presentation for USAID and CEPPS on evaluation findings, initial conclusions and recommendations May 2018 USAID and Evaluation team collaboration on developing the recommendations ● USAID Participation in the team FCR Session ● Meeting/s between the evaluation team and USAID to develop actionable recommendations May 2018 MESP submit draft report June 2018 MESP submit final evaluation report, incorporating USAID feedback June 2018 84 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV ANNEX B. EVALUATION DESIGN INTRODUCTION USAID/Jordan has requested an external performance evaluation of the Consortium for Elections and Political Processes Strengthening (CEPPS) program, with a total value of $53.3 million, covering the performance period of January 2010 – June 2017. CEPPS was recently re-awarded, covering the performance period of July 2017 – December 2020, with a total value of $19.2 million. The activity is now implemented by NDI and IRI. USAID/Jordan has also requested that the evaluation consider, where possible, the support for the decentralization elections under the follow-on CEPPS activity. A separate direct Government-to-Government (G2G) award of $1.5 million was provided in May 2017 to the Independent Election Commission (IEC) for the administration of the Decentralization Elections -which was increased to $2.05 million in August 2017. The evaluation will thus include the first quarter of the current CEPPS grant through September 2017 but will not examine the current grant to the IEC. BACKGROUND AND PROJECT DESCRIPTION Jordan has been on a sluggish but steady course towards political reform. Over the last 10 years, Jordan has implemented various structural reforms that aim at engaging a greater proportion of citizens in the political process. The country’s unique experience during the Arab Spring has produced the first amendments to the constitution in decades and a new legal framework for elections and political parties. Most recently, parliamentary elections were carried out in 2016, and local elections were carried out in August 2017 under new municipalities and decentralization laws ushering in new councils that can bring change through more effective community engagement. However, regional developments, the influx of Syrian refugees, and a struggling economy continue to cast long shadows over the social, economic and political landscape in the country leaving it vulnerable to conflicts across its borders. Key goals of USG foreign policy in Jordan are to help ensure that Jordan becomes increasingly responsive to citizens and that it becomes more supportive of civil and political rights. These goals require supporting the development and consolidation of increasing pluralistic, fair, broad-based, and representative elected institutions. CEPPS contributes to USAID/Jordan’s current Country Development Cooperation Strategy (2013-2019), as amended and extended, under Development Objective #2: “Democratic Accountability Strengthened” and Special Development Objective #4: “Gender Equality and Female Empowerment Enhanced.” In particular, it contributes to IR 2.1 “Accountability of, and Equitable Participation in, Political Processes Enhanced,” and IR 2.3 “Civil Society Engagement and Effectiveness Increased.” The CEPPS Activity is intended to support domestic election monitoring, increase participation in election processes, and train candidates and political parties in effective campaigning and polling, and provide technical and in-kind assistance to the Independent Election Commission (IEC). The CEPPS program aims to strengthen the development of more democratic and open political processes in the kingdom and, specifically, to support the participation of candidates, activists, monitors, and voters in elections. To this end, the program worked to achieve specific objectives including: USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 85 ● Improving the ability of civil society organizations (CSOs) to build grassroots demand and effectively advocate for a new legal framework for elections; ● Developing avenues to engage youth in the election process and civic engagement; ● Strengthening political alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels; ● Encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls; ● Bolstering public demand for candidates’ commitment to open and more accountable performance as future members of parliament; and ● Building the capacity of the IEC to conduct transparent and credible election processes. EVALUATION QUESTIONS The Scope of Work (SOW) for the evaluations provides five evaluation questions in three areas. EFFECTIVENESS & RELEVANCE 1. What is the overall effectiveness of CEPPS for achieving its goals and objectives? 2. How did the strategy and implementation approach enhance or weaken achievement of the intended outcomes? Are there certain areas/activities and approaches that have been more effective? Why? 3. How has the program adapted to changes and how has collaboration within CEPPS and other DRG partners/donors influenced activities and the achievement of results? SUSTAINABILITY 4. Which interventions are most likely to sustain over time (and which will be difficult to sustain)? Why and how? What should be considered to enhance sustainability and local ownership? LEARNING 5. What are some key lessons learned that can inform the activity and the Mission going forward? UNDERSTANDING THE EVALUATION QUESTIONS The evaluation team has developed its understanding of the EQs through our initial reading of CEPPS project documentation, our initial discussions with USAID and internal team discussions. The EQs focus on key components of performance evaluations: effectiveness & relevance; sustainability; and lessons learned. The ET’s task is to develop its understanding of the work of the CEPPS project and how to most effectively evaluate its work to answer all of the EQs asked by USAID. The ET has developed our understanding by unpacking the EQs and developing our understanding of the CEPPS project so that we can fit the main activities of the partners in Jordan into the evaluation. This is necessary to evaluate the work of the project fairly – and to evaluate in a clear, well-organized way that answers all of the evaluation questions. 86 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV EFFECTIVENESS & RELEVANCE The SOWs first three questions focus on the effectiveness and relevance of the CEPPS program. The first EQ asks the ET to evaluate the overall effectiveness of CEPPS for achieving the goals and objectives of the program. Evaluating effectiveness has different implications for different types of evaluations. USAID clearly distinguishes between an impact evaluation and a performance evaluation like this one.5 To consider effectiveness through performance evaluation goals and methods, the ET will focus on how the CEPPS partners implemented activities towards the program’s goal and objectives, what the program has achieved, and how these implementation approaches and achievements are perceived by USAID, the three main CEPPS partners, other CEPPS partner organizations, stakeholders, subject matter experts and beneficiaries. Overall effectiveness clarifies that the ET will focus on the main strategies, approaches, and activities of the CEPPS partners towards achieving the goal and objectives of the program. Specifically, the ET will target its efforts on the four major strategic themes of the CEPPS program as identified by USAID: 1. Electoral process reform and election monitoring; 2. Parliamentary monitoring; 3. Local governance; 4. Civic engagement The CEPPS program has evolved over the eight-year period of performance, between January 2010 and June 2017, of the cooperative agreement. The initial cooperative agreement was for a 23-month program and committed less than $4 million to IRI and NDI. The program evolved with changes in Jordan to become a $53.3 million partnership which also brought IFES into the country to support the new Independent Electoral Commission (IEC) with the administration of elections. The cooperative agreement evolved into one where IFES reported on four objectives, IRI seven objectives, and NDI 12 objectives. While the evaluation primarily focuses on CEPPS’ performance in the period between 2010 and 2017, the evaluation will also evaluate the first quarter’s work of the current CEPPS cooperative agreement. The current agreement is a $19.2 million award to IRI and NDI for the period July 2017 – December 2020. The CEPPS work to be evaluated from the current award includes interventions focused on the September 2017 municipal and decentralization elections. Including this quarter’s work, the evaluation can cover three election cycles- two national parliamentary election cycles in 2013 and 2016 and one special election cycle for the decentralization and local elections in August 2017. The ET has expanded the six objectives from the SOW to ensure that these objectives explicitly include all of the 23 objectives of the cooperative agreement for the 2010-2017 period. The ET will use the six expanded encompassing objectives of the program below: 5 The two are usefully defined and distinguished in the August 2017 “USAID Evaluation and Monitoring Terms” by USAID’s Learning Lab (https://usaidlearninglab.org/sites/default/files/resource/files/usaid_program_cycle_terms_aug_2017.pdf, accessed March 20, 2018). USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 87 CEPPS Objective 1 - Improving the ability of groups of politically active citizens and civil society organizations (CSOs) to participate in the electoral process, build grassroots demand and effectively advocate for a new legal framework for elections. CEPPS Objective 2 - Developing avenues to engage women, youth and the disabled in the election process and civic engagement CEPPS Objective 3 -Strengthening political parties and alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels CEPPS Objective 4 - Encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls CEPPS Objective 5 - Bolstering public demand for local self-government and candidates’ commitment to open and more accountable performance as future members of parliament CEPPS Objective 6 - Promoting the transparency and integrity of election processes and building the capacity of the IEC to conduct transparent and credible election processes. The three CEPPS partners each work towards some or all of these objectives. The following section describes which partners work towards each of the six objectives of the CEPPS program specified in the evaluation SOW. This information is also provided in Annex VIII: CEPPS Objective Map. CEPPS Objective 1 covers IRI’s work with groups of politically active citizens and efforts to help voters make informed choices based on increased knowledge of election procedures, candidates, and parties. The objective includes NDI’s work with civil society on advocacy, parliamentary monitoring, strengthening the relationship between parliament and civil society/their constituents, and encouraging a more participatory political process working with government institutions. CEPPS Objective 2 encompasses IRI’s work with women and youth, including the activities with parties and rural and urban communities, as well as NDI’s work to engage youth and women. We will also include IFES’s work with the IEC on disability access under this objective. CEPPS Objective 3 covers only IRI’s Objective 1: Political parties in Jordan develop issue-oriented platforms and policy positions and communicate them to citizens. CEPPS Objective 4 refers only to IRI Objective 7 on polling (Jordanian citizens, political parties and organizations have greater access to qualitative and quantitative public opinion research). CEPPS Objective 5 encompasses both IRI and NDI. The Objective covers three NDI objectives: 5 Bolster public demand for ensuring candidates' commitment to a more open and accountable performance as future members of parliament; 9 work to improve citizens understanding of local self￾governance (LSG); and 11 work with communities hosting refugees on common concerns. The Objective also covers IRI’s Objective 3, Groups of politically active citizens increased their capacity to interact with and advocate to elected officials and governmental bodies for the needs of their communities 88 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Finally, we have edited CEPPS Objective 6 to encompass IRI and NDI’s work on electoral observation, an objective of each of these CEPPS partners, as well as the four objectives of IFES. The evaluation will also ask directly about implementation, achievement, and perceptions of the partners’ work toward the CEPPS goal of strengthening the development of more democratic and open political processes in the kingdom and, specifically, to support the participation of candidates, activists, monitors, and voters in elections. The ET will not ask key informants or participants in group interviews about performance and program achievements in objectives that they do not work in. The second EQ asks the ET to evaluate how the strategy of the program and the implementation approaches of the CEPPS partners promoted or detracted from the achievement of the intended outcomes of the program. The ET will approach this question directly by evaluating the strategy and main approaches of the CEPPS partners. The ET will also approach this question by investigating the sub-question in the SOW on whether there are certain areas and approaches that have been more effective – and what made these activities more or less effective? The third EQ asks the ET to explore how the program has adapted and changed over the course of the cooperative agreement (including in the successor award and first quarter of its implementation at the end of FY 2017). The ET will focus on understanding how IFES, IRI and NDI and responded to changes in Jordan, in US government approaches, and the region, and how this informed CEPPS partner’s strategies, approaches, and activities. The ET will also explore how the CEPPS partners collaborated within the partnership, as well as with other USAID DRG partners and donors in Jordan. The ET will focus on IFES, IRI, and NDI collaboration with USAID’s other DRG activities and partners within this sub-question. SUSTAINABILITY The fourth EQ asks the ET to evaluate progress towards sustainability. The ET will answer this question through a focus on perceptions – which activities and approaches are seen as more likely to be sustained going forward – as well as whether and how interventions may have continued without CEPPS support. When possible, the ET will examine whether beneficiaries have been or expect to be able to sustain their activities after the conclusion of IFES, IRI, and NDI support. The ET will ask about specific approaches in used by the CEPPs partners to build sustainability. The ET will also explore what CEPPS partners and USAID could consider doing to further build local ownership and enhance sustainability. LEARNING The fifth and final EQ asks the ET for key lessons learned from the implementation and achievements of the CEPPS program. The ET will explore this question through all of its evaluation methods by asking informants about what they think the main lessons learned from the program are. The ET will explore and validate these findings in our work on answering the other four EQs. The ET will also analyze accumulated findings from the other EQs to identify other lessons learned. APPROACH TO ANSWERING THE EVALUATION QUESTIONS The CEPPS program includes work with three principal partners, IFES, IRI and NDI, over a period of eight years. The program encompasses work in numerous areas related to democracy, governance and USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 89 political processes including assistance for election administration, training and deploying election monitors, strengthening civil society, municipal governance and support for parliament. Given the substantial scope of this evaluation, with regards to both duration and breadth of the programs, the team will utilize a range of evaluation tools to obtain broad, objective and representative evidence while remaining focused on the evaluation questions. The team will use these approaches to evaluate the performance of the three CEPPS partners and determine findings (facts) about CEPPS performance and achievements, triangulate and analyze these findings to draw conclusions, and make recommendations for future USAID programming in Jordan. The ET proposes to focus the evaluation on the period 2012 through the 2017 local and municipal council elections. The reasons for this focus are that changes in Jordan and subsequent changes in the program mean that the context in 2010 and 2011 was so different from the period 2012 to 2017. This context change has made the activities of this period substantially different from the earlier period as to be not usefully comparable. Simply put, the effects of the Arab Spring across the region and in Jordan have meaningfully changed the context and program. Another implication of these two changes is that informants will not remember much about the period prior to these changes, and what they remember from this period will be influenced by these changes and post-Arab Spring developments. This means that retrospectively respondents are unlikely to report out accurately on their experiences from that time or how they understood things at the time. The ET understands that USAID and IRI are assessing the environment in Jordan for the development of political parties based on the experience of IRI is supporting political party development to date through the CEPPS mechanism. USAID and IRI have suspended work with political parties while they carry out this assessment. Since USAID and IRI are already reassessing the programs work with political parties and what can be done going forward with political parties under the current CEPPS award, the ET will not focus its work in this area. The ET will examine IRI’s work with parties as one of the objectives IRI pursued under CEPPS. The ET will evaluate this work within the context of the entire IRI program since it is necessary to explain IRI’s strategy under CEPPS, IRI’s implementation approaches, and the achievements of the program. The ET will not put any additional focus on political parties since USAID and IRI are already actively reassessing assistance under CEPPS in this area. The evaluation team will use several different qualitative and quantitative approaches to gather the data needed for the evaluation. Qualitative approaches will include the desk review, key informant interviews (KIIs), group interviews (GIs), and site visits. The desk review will focus on the primary program related documents produced by the CEPPS partners including quarterly reports, the final report, inception documents and modifications. The ET will use these data to ensure that we understand the approaches, activities, and reported progress towards objectives and the goals of the CEPPS program. KIIs will be held with relevant USAID staff, the Country Directors of IFES, IRI, and NDI, IEC leaders, the leaders and staff of key program partners, and subject matter experts. KIIs will be used to gather new, independent data on program approaches, performance, and achievements directly relevant to answering the evaluation questions. KII protocols for these groups of informants are included in Annex VI. In regard to KIIs with IP staff, the ET has been notified that in interviews with staff, both IRI and NDI have requested that that senior staff including the chief of party be present. The ET notes a number of challenges associated with this approach, which include the fact the presence of senior management may 90 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV significantly impact the validity and reliability of information gathered during the interviews with staff. With supervisors attending interviews, staff may feel less inclined to express their views freely. Respondents may adjust their responses so as not to be perceived negatively by supervisors or jeopardize their standing within the organization. Group interviews will be held with three different groups of informants. GIs will also gather new, independent data on program approaches, performance, and achievements that are directly relevant to answering the evaluation questions. At NDI and IRI’s insistence, the ET will not conduct KIIs with most NDI and IRI senior program staff. Instead, the ET will hold group interviews to learn from senior staff about their experience implementing the program, the main achievements of their work, collaboration and coordination, sustainability, and lessons learned. The ET will also hold a group interview with the USAID AORs and CORs for other partners of the DRG Office that implemented activities around the 2016 and 2017 elections. This GI will focus on collaboration across USAID partners. The third group of informants for group interviews are beneficiaries of IRI and NDI programs. The ET will hold separate group interviews with groups of similar beneficiaries of specific IRI and NDI programs (e.g. participants in IRI’s Youth Academy program, participants in particular types of Ana Usharek activities). Site visits will be used to get more information about how trainings and other workshops have been conducted. The team will endeavor to link site visits to holding GIs with beneficiaries of the program. Other GIs will be held without site visits. We recognize these visits will occur after the period the team has been asked to evaluate so will only make site visits to programs that were part of the program during the time in question. Site visits will also give the team the opportunity to conduct GIs with some program beneficiaries who were also part of the program in 2016-2017, or earlier. The quantitative approach employed by the team will be a survey of program beneficiaries. Per the evaluation SOW, the ET was tasked to conduct a quantitative survey of respondents from CEPPS program activities as well as select beneficiaries of other USAID DRG programs that implemented election-related interventions related to the 2016 and 2017 decentralization and parliamentary elections. These activities include Takamol, the Civics Initiatives Support (CIS) Program, and the Cities Implementing Transparent Innovative and Effective Solutions (CITIES). The proposed survey was intended to ask general questions to explore if respondents feel they have political agency and more confidence in advocating for their interests. After further consideration, the ET believes that the most appropriate target respondents for the survey are the beneficiaries of NDI and IRI activities and that the beneficiaries of related USAID DRG activities should be excluded. The ET has conducted an initial review of the Takamol, CIS and CITIES programs and has concluded that their engagement with beneficiaries differs enough from CEPPS’ engagement that including them in the survey pool would create a significant methodological challenge. Unlike with CEPPS activities, the primary beneficiaries of the non-CEPPS activities were CSOs and CBOs and not individuals. These beneficiary CBOs and CSOs conducted a wide range of activities and had varying levels of engagement with their own individual beneficiaries. Additionally, when the evaluation team attempted to retrieve comprehensive lists of the beneficiaries with contact information that were engaged by beneficiary CBOs and CSOs, the team discovered that a number of the organizations did not maintain such lists and trying to create such lists would require significant time and resources. Given this information, the evaluation team proposes that survey focus solely on beneficiaries of IRI and NDI activities. The evaluation team is still reviewing beneficiary lists from NDI and IRI to create a more USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 91 concrete sampling criteria and design. The team will share this and a draft of the proposed questionnaire in the next couple of days for review and comment by both USAID and relevant IPs. ETHICS The ET understands the complexities that arise when conducting research on human subjects. As such, the ET will adhere to a strict code of ethics when conducting individual key informant interviews, group interviews, and the survey in order to prevent any potential harm to respondents. Prior to conducting any interview or the survey, the ET will provide detailed information to respondents about the nature of the research and how information gathered will be used to inform the overall evaluation. Respondents will also be notified that all information gathered will be confidential and that no direct attributions will be made to them without their consent. Interviewers will obtain informed consent before commencing an interview or survey. Respondents will be notified that they will not be obligated to answer any question they do not wish to and may terminate the interview/survey at any point. The ET will adhere to this established ethics protocol and work to create an environment in which the respondent feels comfortable responding to questions and expressing their opinions freely. GENDER AND SOCIAL INCLUSION APPROACH In gathering data to answer the evaluation questions, the team will take into consideration gender￾related issues primarily by assessing how gender was integrated in the CEPPS design, implementation, and monitoring and evaluation. The evaluation will cover how CEPPS partners targeted various interventions at both male and female beneficiaries. The inclusion of both genders serves an integral part of the process of change to foster a participative political environment by ensuring that all Jordanians, regardless of gender, feel more empowered in decision-making and that they have control over their choices. The evaluation will identify and assess the barriers to, and opportunities for, voting, becoming a candidate, participating in capacity building activities, strengthening the capacity of the civil society, electoral system and IEC’s management from the gender perspective. The focus will be on NDI, IRI and IFES and their engagement in seeking solutions to the challenges that were identified or encountered in project design and implementation. The evaluation will look not only at cases of positive engagement (for example female beneficiaries of candidate development trainings who ran for office or who hold office) but also cases of non-political engagement (for example, trained female beneficiaries who did not run for office). The team will examine the extent to which the CEPPS partners integrated vulnerable populations into their initiatives, including women, youth and persons with disabilities. The evaluation will look into how they assess the implementation and achievements of the program and at their sense of agency as well. The team will work to mainstream gender and social inclusion indicators and considerations throughout the data collection methods and the data analysis. All data will be disaggregated by age, gender and other social inclusion demographic indicators such as poverty level and disability status. 92 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV DATA COLLECTION METHODS DESK REVIEW The ET will review the key planning, implementation, reporting, and monitoring and evaluation documents of the CEPPS program. The initial purpose of the desk review is to ensure that the ET understands the goals and objectives of the program, the theory of change for the program, CEPPS approaches, and the plans of the CEPPS partners under the award and how these objectives, plans, approaches, and activities change and evolve over the eight-year period from January 2010 through September 2017. The desk review will also distill an understanding of key partner approaches and the main achievements of the program. This kind of understanding is essential to evaluating the program comprehensively and fairly through a focus on the main interventions and approaches of the CEPPS partners and other program partners. The desk review will also help provide the team with key partners for KIIs. QUALITATIVE SAMPLING APPROACH The team will purposively select between 65-85 individuals for KIIs and GIs. KII subjects will be drawn from the following eight categories of informants: • USAID • Implementing Partners • IEC • CEPPS NGO and Civil Society Partners • Government Officials and MPs • Political Party Representatives • Other DRG Partners, International Donors and International Implementers • Subject Matter Experts The team will consult with the IPs to determine the most fruitful opportunities for conducting site visits and GIs. At NDI and IRI’s request, rather than KIIs, the ET will conduct group interviews with key program staff with the presence of the COP. The ET will conduct group interviews with the USAID AORs/CORs of other DRG programs that targeted the 2016 elections. We will also seek to conduct GIs with beneficiaries from several different NDI and IRI activities, when possible this will be linked to a site visit to an activity such as a training or a workshop. QUANTITATIVE SAMPLING APPROACH The survey anticipates using beneficiary information made available to the team from NDI and IRI on activities from similar interventions with similar demographic groups around the 2016 and 2017 elections. The proposed survey will ask general questions of satisfaction with CEPPS activities, how these activities may be improved, and to explore if respondents feel they have political agency and have more confidence in advocating for their interests. The team will request lists of participants from the relevant programs from NDI and IRI. From those lists, the team will endeavor to construct a representative sample. The ET will randomly sample a significant number of individual beneficiaries from lists of participants maintained by NDI and IRI. If we are unable achieve this, or if there are significant biases in the data, such as respondents being overwhelmingly male, we will consult with the USAID team regarding the feasibility, risks and limitations associated with conducting such a survey. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 93 A full survey design and methodology report will be delivered separately before the start of survey fieldwork and will be included as an annex to the final report. TABLE I: SAMPLE DATA COLLECTION METHODS FOR THE FIVE EVALUATION QUESTIONS: Data Collection Methods Evaluation Questions Desk Review 1,2,3,4,5 Key Informant Interviews 1,2,3,4,5 Survey 1,2,4,5 Group Interviews 1,2,4,5 Site Visits 1,2,4,5 DATA ANALYSIS METHODS The evaluation team will analyze all data collected through the qualitative and quantitative research through four different methods: descriptive analysis, content analysis, trend analysis and crosstabs. DESCRIPTIVE ANALYSIS: The team will use descriptive analysis to analyze both quantitiatve and qualitative data. Descrpitive analysis will draw on all of the different methodological approaches to describe the programs and begin to frame responses to the evaluation questions. CONTENT ANALYSIS: The team will review the contents of KIIs, site visits, GIs, and relevant program documents to understand findings relevant to answering the evaluation questions. Content analysis will also be used to identify program activities that are particularly relevant to the evaluation as well as to provide illustrative examples that help explain the findings. TREND ANALYSIS: The team will review data from the IPs or USAID that provides information about the program over time. This includes measures like quantitative indicators about program participants as well as descriptions of how the programs and activities have evolved over the course of the project. KIIs with other respondents who have sufficient institutional memory will be used for trend analysis as well. Additionally, we will examine IRI’s polling data for findings that can be compared over the course of two or more polls. FREQUENCIES: The team will draw on frequency data from the survey of beneficiaries to help answer questions regarding the effectiveness of project activities. These data will be compared through content and desriptive analysis to identify patterns or inconsistincies in the findings. CROSSTABS: By using crosstabs from the public opinion survey, the team will seek to show how specific groups such as women or youths viewed programs activities. This approach will make it possible for the team to use learn more about the IPs performance with these key groups. 94 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV LIMITATIONS All evaluation designs and methodologies face some limitations. These limitations may affect the quality and quantity of data the ET can collect and analyze. The ET has identifed the following limitations in developing the design and methods: 1. Limited information on some program staff and beneficiaries: Key staff from earlier periods of program implementation may no longer be engaged in this same work, with the same IP, or in the country. It may thus be difficult to identify and interview them. It also may be difficult to acquire accurate and complete lists of participants in DRG electoral programs from IRI, NDI, CITIES, Takamol, and CIS partners from 2017 and 2018 with current phone numbers. This could create problems for the survey 2. Limited independence of NDI and IRI staff: NDI and IRI insist that the ET not conduct individual KIIs with key program staff other than the COPs. USAID has instead approved group interviews with key program staff which will be held with senior NDI and IRI staff present. The group setting and presence of their management may inhibit some NDI and IRI staff from speaking freely with the ET under these conditions. 3. Recall bias: Key informant interviewees, group interviewees, and survey respondents may systematically have difficulties remembering details about the past relative to the present. This presents potential issues of bias in their recall of information. Recall biases may particularly make it difficult to accurately use trend analysis to compare situations before and after interventions. Recall bias may also affect their ability to remember particular interventions, which may affect our ability to compare between interventions. 4. Selection bias: Some key informants may decline to be interviewed or surveyed. This presents the possibility of selection bias within our purposive sampling. Respondents who choose to be interviewed or surveyed might differ from those who do not in terms of their attitudes and perceptions or other areas. A telephone survey may add additional potential biases related to differences in telephone or mobile phone access among different groups must be considered. 5. Halo bias: KII, GI, and survey respondents may under or overreport socially undesirable answers and alter their responses in accordance with what they perceive as prevailing social norms. The extent to which respondents will be prepared to reveal their true opinions may also vary for questions that call upon the respondents to assess the performance of partners that provide them with benefits. 6. Absence of baselines: The evaluation design is not able generate baselines to understand the situation prior to the CEPPS program. We will learn about the achievements of the program, but cannot compare them to what the situation was pre-intervention. 7. Inability to assess attribution. The ET cannot assess causality – whether any changes in individuals or organizations can be attributed to particular interventions or the work of the project. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 95 8. Difficulty assessing sustainability: CEPPS interventions are ongoing, which in many cases will make it difficult for the ET to estimate whether reported/observed achievements are likely to continue after CEPPS support ends. 9. Size of the CEPPS program: The CEPPS partners have completed many interventions towards the 23 objectives of the program over eight years of implementation from January 2010 through September 2017. The many interventions, large number of partners, and long time frame make it difficult for the ET to identify and focus on the main CEPPS interventions towards achieving the goal and objectives of the program. The evaluation design addresses these limitations through a variety of methods: 1. The ET will be explicit about the limitations in the evaluation’s design and methods as well as how these limitations are managed throughout the evaluation in the Evaluation Report. 2. The ET will guarantee all individual KII, non-NDI and IRI group interview, and survey respondents informants anonymity and non-attribution to help reduce halo biases. Anonimity does not exist for NDI and IRI staff interviewed in group settings with management present. 3. The ET will not ask questions to NDI and IRI staff in group interviews where the management is present that may make participants feel uncomfortable answering these questions. 4. The ET will triangulate evidence from different qualitative and quantitative data sources, which serves to increase the credibility of findings via validation by multiple data sources. 5. The ET will work closely with the CEPPS partners to make sure that we have sufficient information on program staff – particularly past staff – partners, stakeholders, and beneficiaries. The ET will collaborate closely with USAID and the CEPPS partners to facilitate introductions and mobilization of partner, beneficiary, and stakeholder participation in the evaluation. Mobilization helps ensure that partners, beneficiaries, and stakeholders are willing to participate and are well informed of the purposes of the evaluation. The ET will hold KIIs and GIs in venues where participants are comfortable. The ET will organize GI that are as homogenous as possible in terms of participants. 6. Telephone surveys create a certain amount of selection bias as the distribution of access to telephones is not equal across the population. The ET will examine the demographic and program data from the random sample of beneficiaries done in the telephone survey and check for issues of bias, and report on any issues in the evaluation report. 7. The ET will focus on the most robust findings from qualitative data and analysis. These are the findings that appear with relatively greater frequency across multiple stakeholders. 8. The ET will conduct systemic data analysis of the quantitative data from the survey using well￾established statistical methods and software. The ET will also use questions from IRI polling in the beneficiary survey and compare findings from these data to the findings of IRI polls. 9. The ET proposes to focus the evaluation on the most recent work of CEPPS partners in the survey which minimizes recall bias. The team will remain cognizant of the difficulties of working 96 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV further back in time and ask specific questions about interventions from earlier in the period of performance. The ET also proposes to focus KIIs on CEPPS work from 2012 onward rather than the pre-Arab spring period to reduce both potential recall and halo biases. 10. The ET will continue to emphasize that the evaluation is not designed to and will not focus on causality or the attribution of achievements to particular interventions. 11. The ET will focus on the likelihood of sustainability in the future (for interventions and achievements of interventions that have not ended) and informants’ perceptions of what is likely to be more or less sustainable going forward. Questions about why interventions or achievements are seen as more or less sustainable will be asked to support these views. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 97 GETTING TO ANSWERS MATRIX Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appropriate) Methods for Data Collection e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey6 Sampling or Selection Approach (if one is needed) Data Analysis Methods E.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Analysis Data Source(s) Method 1. What is the overall effectiveness of CEPPS for achieving its goals and objectives? Yes/No • Program documents • USAID staff • IP staff • Partners • Stakeholders e.g. MoPPA • Beneficiaries • Desk review • KIIs • Site visits • Group Interviews (GIs) • Survey • Purposive • Descriptive analysis • Content analysis • Trend analysis • Frequencies • Cross￾tabulations X Description X Comparison X Explanation 1. How did the strategy and implementation approach enhance or Yes/No • Program documents • USAID staff • Desk review • KIIs • Purposive • Descriptive analysis X Description 6 Data from evaluations are a deliverable and methods should indicate how data will be captured, i.e., for focus groups USAID requires a transcript. 98 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appropriate) Methods for Data Collection e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey6 Sampling or Selection Approach (if one is needed) Data Analysis Methods E.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Analysis Data Source(s) Method weaken achievement of the intended outcomes? a. Are there certain areas/activities and approaches that have been more effective? b. Why? X Comparison • IP staff • Partners • Beneficiaries • Group Interviews (GIs) • Survey • Content analysis • Trend analysis • Frequencies • Cross￾tabulations X Explanation 2. How has the program adapted to changes and how has collaboration within CEPPS and other DRG partners/donors influenced activities and the achievement of results? Yes/No • Program documents • USAID staff • IP staff • Partners • Beneficiaries • Desk review • KIIs • Purposive • Descriptive analysis • Content analysis • Trend analysis X Comparison X Explanation X Description USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 99 Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appropriate) Methods for Data Collection e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey6 Sampling or Selection Approach (if one is needed) Data Analysis Methods E.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Analysis Data Source(s) Method 3. Which interventions are most likely to sustain over time (and which will be difficult to sustain)? a. Why and how? b. What should be considered to enhance sustainability and local ownership? Yes/No • Program documents • USAID staff • IP Staff • Partners • Stakeholders e.g. MoPPA • Beneficiaries • Desk review • KIIs • Site visits • Group Interviews (GIs) • Survey • Purposive • Descriptive analysis • Content analysis • Trend analysis • Frequencies • Cross￾tabulations X Description X Comparison X Explanation 5. What are some key lessons learned that can inform the Activity Yes/No • Program documents • USAID staff • Desk review • KIIs • Purposive • Descriptive analysis X Description 100 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appropriate) Methods for Data Collection e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey6 Sampling or Selection Approach (if one is needed) Data Analysis Methods E.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Analysis Data Source(s) Method and the Mission going forward? X Comparison • IP Staff • Partners • Stakeholders e.g. MoPPA • Beneficiaries • Site visits • Group Interviews (GIs) • Survey • Content analysis • Trend analysis • Frequencies • Cross￾tabulations X Explanation USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 101 WORK PLAN The team’s schedule of planned activities is listed below. The inception phase ends with the approval of the Evaluation Design. In the subsequent phase, the team will be involved in data collection through secondary data collection and primary interviews, ending approximately the end of November. The final phase encompasses the analysis of data, leading to findings, conclusions and co-generation of recommendations along with USAID. During this phase, additional data collection may continue to fill in gaps discovered during the F/C/R process. The team will present preliminary findings to USAID at periodic stages of the assignment, including prior to finalizing the report. Activity Responsible Dates Location Design and Kick Off Phase Team Planning Meeting Evaluation Team March 14 Amman In-Brief with USAID DRG Team Evaluation Team March 15 Amman Draft Evaluation Design submission Evaluation Leader March 28 Amman Data Collection Phase Data Collection through key informant interviews (KII), site visits, small group discussions and beneficiary survey Evaluation Team March 26 – May 13 Amman and relevant governorates Analysis, Briefings and Report Development Initial data analysis of qualitative and survey data Evaluation Team May 14 – June 10 Amman/Remote Midterm briefing with USAID DRG Team Evaluation Team June 10 Amman/Remote Final data analysis of qualitative and survey data Evaluation Team June 10 – June 23 Amman/Remote De-Brief with USAID DRG Team Evaluation Team June 24 Amman Co-Generation of Recommendations Evaluation Team/USAID July 1 Amman/Remote Submission of Draft Report Team Leader July 29 Amman/Remote 102 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV DETAILED PROPOSED CONSULTATION LIST USAID AND IMPLEMENTING PARTNERS USAID • Deputy Mission Director • Director, Democracy, Rights & Governance Office • Deputy Director, Democracy, Rights & Governance Office • DRG Technical Director, Democracy, Rights & Governance Office • Program Management Specialist/CEPPS AOR • Program Development Specialist/Office of Program Management • DG AORs and CORs from other DRG political process/elections focused activities NDI • Senior Country Director • Deputy Chief of Party • Program Manager, Youth Program • Program Officer • Program Manager, Schools Program • Gender Advisor /Manager with focus on Gender • Usharek Field Coordinators/Regional Coordinators • Former relevant personnel (if needed) IRI • Country Director • Senior Governance Advisor • Former Country Director • Deputy Chief of Party • Resident Program Officers/Program Managers • M&E Officer/Manager\ • Gender Advisor • Former relevant personnel (if needed) IFES • Former Chiefs of Party/Country Directors • Senior Washington Program Staff • Former Deputy Chief of Party • Former Senior Program Staff • Former Gender Advisor • Former M&E Officer • Former governmental advisor* USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 103 IEC • Chair • Department head electoral process • Department head policies and institutional development • Department head communications information and awareness • Department head information systems and IT • Department head administrative financial • Department head human resources and legal affairs • Selected board members NON-CEPPS DRG IPS • Chief of Party, CIS • Chief of Party, Takamol • Chief of Party, CITIES • Chief of Party CEP NGO AND CIVIL SOCIETY PARTNERS OF CEPPS • Al-Hayat o Center Director o Strategic Planning Director o RASED Parliament Program Manager • Identity Center o Center Director o Executive Director o Programmes Manager • Al-Quds Center o Director of the Center o Executive Director o Director of Research • National Human Rights Center o Commissioner • Other Civil Society Partners** o National Women’s Council o Eduardo Frei Foundation GOVERNMENT OF JORDAN AND PARLIAMENT • Government Partners o MoMA*** o Municipal Governments (2-3 outside Amman)*** o Ministry of the Interior*** o Local Government Unit*** o Ministry of Parliament and Political Affairs*** o Ministry of Education o Ministry of Higher Education 104 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV o University Deans • Parliament o Speaker and Secretary General of Parliament o Parliamentary leadership *** o Members of Parliament*** o Parliamentary Staff*** o Members of the Women’s Parliamentary Caucus*** POLITICAL PARTY REPRESENTATIVES TBD based on further consultations with IRI and USAID SUBJECT MATTER EXPERTS • Former MP and currently director of Nissan Center • Former MP and columnist, political parties activists • Former MP and President of Teachers Association • Columnist and currently dean of Jordan Media Institute • Columnist and political scientist at Jordan • Columnist and political analyst NON-USAID DONORS AND IMPLEMENTERS • Other Donors***: o EU o DIFD • Other Implementers***: o UNDP o Westminster Foundation DATA COLLECTION INSTRUMENTS KEY INFORMANT INTERVIEWS USAID Key Informant Protocol Note This protocol is written so that it can be used for a range of key informants from different organizations. Individual interviews will stress parts of the questionnaire that directly focus on the engagement of each informant with the project (e.g. interviews with the IEC will focus on the evaluation question on how support built the capacity of the IEC). The interviewer will follow up these broad questions with targeted follow-up questions that gather more detail on the initial responses of informants that provide specific information on areas of the project with which they have been engaged with. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 105 Introduction Thank you for taking the time to talk with us today. My name is _______________ and I work for MSI with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. Our team is conducting a performance evaluation of the program implemented in Jordan by the International Foundation for Electoral Systems or IFES, International Republican Institute or IRI, and National Democratic Institute or NDI. We would like to learn from your experience engaging with/managing these partners over the period 2010 through 2017. There are a standard set of best practices in evaluation that we will use, including anonymity, non￾attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your name – or use the information you provide in such a way that can be traced back to you. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT INFORMANT EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the program? As a performance evaluation, we are focused on the effectiveness of program implementation. We are most interested in how these activities were implemented in Jordan. 1. How long have you been with USAID? 2. Please tell us about your position, background, and responsibilities in the organization? Effectiveness, Relevance, Coordination, And Sustainability 3. In your experience, how effective do you think IRI and NDI have been in implementing programs to improve the ability of groups of politically active citizens and civil society organizations (CSOs) to: a. Participate in the electoral process b. Build grassroots demand c. Effectively advocate for a new legal framework for elections d. Why have they been more effective or less effective? • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? 106 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 4. In your experience, how effective do you think IRI, NDI, and IFES have been in implementing programs in developing avenues to engage the following groups in the election process and civic engagement: a. women b. youth c. the disabled d. Why? • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 5. In your experience, how effective do you think IRI has been in implementing programs strengthening political parties and alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels? Why? • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 107 o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 6. In your experience, how effective do you think IRI has been in implementing programs encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls? Why? • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 7. In your experience, how effective do you think IRI and NDI have been in implementing programs bolstering public demand for local self-government and candidates’ commitment to open and more accountable performance as future members of parliament? Why? • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 8. In your experience, how effective do you think IRI and NDI have been in implementing programs promoting the transparency and integrity of election processes - and how effective do you think IFES has been in implementing programs building the capacity of the IEC to conduct transparent and credible election processes? Why? 108 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 9. In your experience, how effective do you think IFES, IRI, and NDI have been in implementing programs strengthening the development of more democratic and open political processes and the participation of candidates, activists, monitors, and voters in elections? Why? [PROGRAM GOAL] • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? Lessons Learned 10. Based on your experience with these programs, what do you think are some key lessons learned that can help NDI, IRI, the IEC, and their partners under the current program? 11. Based on your experience with these programs, what do you think are some key lessons learned that can help the USAID Mission in these areas in your future strategy? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 109 USAID DRG AORS/CORS (NON-CEPPS) Small Group Interview Protocol Introduction Thank you for taking the time to talk with us today. My name is _______________ and I work for MSI with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. Our team is conducting a performance evaluation of the program implemented in Jordan by the International Foundation for Electoral Systems or IFES, International Republican Institute or IRI, and National Democratic Institute or NDI. We would like to learn from your experience engaging these partners over the period 2010 through 2017. There are a standard set of best practices in evaluation that we will use, including anonymity, non￾attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your name – or use the information you provide in such a way that can be traced back to you. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT ALL INFORMANTS EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the program? Background 1. How long have you been with USAID? 2. Could you please describe your current position and responsibilities within the USAID DRG Office? Effectiveness and Relevance 3. Could you describe any collaboration your activities have had with CEPPS? Have these interactions been effective? 4. What challenges have you encountered? 5. How has collaboration within CEPPS and with other partners and donors influenced activities and the achievement of results? 6. Based on your experience, are there any ways in which collaboration with CEPPS could be improved? 110 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV IMPLEMENTING PARTNERS (NDI, IRI AND IFES) Key Informant Protocol Note This protocol is written so that it can be used for a range of key informants from different organizations. Individual interviews will stress parts of the questionnaire that directly focus on the engagement of each informant with the project (e.g. interviews with the IEC will focus on the evaluation question on how support built the capacity of the IEC). The interviewer will follow up these broad questions with targeted follow-up questions that gather more detail on the initial responses of informants that provide specific information on areas of the project with which they have been engaged with. Introduction Thank you for taking the time to talk with us today. My name is _______________ and I work for MSI with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. Our team is conducting a performance evaluation of the program implemented in Jordan by the International Foundation for Electoral Systems or IFES, International Republican Institute or IRI, and National Democratic Institute or NDI. We would like to learn from your experience engaging with these partners over the period 2010 through 2017. There are a standard set of best practices in evaluation that we will use, including anonymity, non￾attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your name or the name of your organization – or use the information you provide in such a way that can be traced back to you or your organization. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT INFORMANT EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the program? As a performance evaluation, we are focused on the effectiveness of program implementation. We are most interested in how these activities were implemented in Jordan. We would appreciate it if you can focus your responses on the programs of your organization. 1. How long have you been with IRI/NDI/IFES? 2. Please tell us about your position, background, and responsibilities in the organization? Effectiveness, Relevance, Collaboration, And Sustainability 3. (Not for IFES) In your experience, how effective do you think your organization has been in implementing programs improving the ability of groups of politically active citizens and civil society organizations (CSOs) to: a. Participate in the electoral process b. Build grassroots demand c. Effectively advocate for a new legal framework for elections d. Why have you been more effective or less effective? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 111 • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 4. In your experience, how effective do you think your organization has been in implementing programs developing avenues to engage the following groups in the election process and civic engagement? Why? a. women b. youth c. the disabled • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 5. (For IRI)In your experience, how effective do you think IRI has been in implementing programs strengthening political parties and alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels? Why? • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? 112 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 6. (For IRI) In your experience, how effective do you think IRI has been in implementing programs encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls? Why? • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 7. (For IRI and NDI) In your experience, how effective do you think IRI and NDI have been in implementing programs bolstering public demand for local self-government and candidates’ commitment to open and more accountable performance as future members of parliament? Why? • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 113 • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 8. In your experience, how effective do you think your organization has been in implementing programs promoting the transparency and integrity of election processes and [for IFES only] building the capacity of the IEC to conduct transparent and credible election processes? Why? • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 9. In your experience, how effective do you think your organization has been in implementing programs strengthening the development of more democratic and open political processes and the participation of candidates, activists, monitors, and voters in elections? Why? [PROGRAM GOAL] • What approaches have you relied on over time? What has worked? What has not worked? Why? • How do you monitor and evaluate your work in this regard? o How have you approached learning in this area? • Were there any major shifts that you had to undertake? Why? • How would you characterize your ability to adapt to the changing context in Jordan? o What factors, in case any, have allowed you to adapt effectively? o What factors, in case any, make it difficult? • Do you have a formal approach to ensuring the sustainability of your work in this area? If so, please elaborate? o [If relevant]: Has this approach to sustainability been successful in your experience? Why or why not? • When it comes to this objective, how would you characterize the overall coordination between your team and other relevant USAID activities? o Are there are any formal and informal mechanisms for this? If so, please describe them? o Are there ways in which the coordination between USAID activities in this area can be improved? If so, how? 114 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Lessons Learned 10. Based on your experience with these programs, what do you think are some key lessons learned that can help NDI, IRI, the IEC, and their partners under the current program? 11. Based on your experience with these programs, what do you think are some key lessons learned that can help the USAID Mission in these areas in their future strategy? IEC Key Informant Protocol Note This protocol is written so that it can be used for a range of key informants from different organizations. Individual interviews will stress parts of the questionnaire that directly focus on the engagement of each informant with the project (e.g. interviews with the IEC will focus on the evaluation question on how support built the capacity of the IEC). The interviewer will follow up these broad questions with targeted follow-up questions that gather more detail on the initial responses of informants that provide specific information on areas of the project with which they have been engaged with. Introduction Thank you for taking the time to talk with us today. My name is _______________ and I work for MSI with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. Our team is conducting a performance evaluation of the program implemented in Jordan by the International Foundation for Electoral Systems or IFES, International Republican Institute or IRI, and National Democratic Institute or NDI. We would like to learn from your experience engaging with these partners over the period 2010 through 2017. There are a standard set of best practices in evaluation that we will use, including anonymity, non￾attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your name or the name of your organization – or use the information you provide in such a way that can be traced back to you or your organization. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT INFORMANT EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the program? We are focused on the effectiveness of program implementation. We are most interested in how these activities were implemented in Jordan. We would appreciate it if you can focus your responses on the programs implemented by IFES, IRI, NDI and their work directly with you and your organization. 1. How long have you been with the IEC? 2. Please tell us about your position, background, and responsibilities in the IEC and electoral administration? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 115 Effectiveness and Relevance 3. In your experience, how effective do you think IFES was in implementing programs to build the capacity of the IEC to conduct transparent and credible election processes? Why? a. How effective was IFES in implementing programs helping build the long-term institutional capacity of the IEC? b. How effective was IFES in implementing programs helping strengthen the IEC’s legal framework for electoral administration processes? c. How effective was IFES in implementing programs helping strengthen the IECs electoral management capacities to administer national and sub-national elections? d. How effective was IFES in implementing programs helping build public confidence in the IEC through the support of the IEC's public and media outreach? [Follow up to ask about different techniques and activities if not raised by key informants] 4. In your experience, do you think the implementation of some IFES activities with you were particularly effective or ineffective? What made the implementation of these particular activities more or less effective? 5. In your experience, how has the IFES program adapted to changes in the environment in which they worked? [follow up – How about changes in the: Jordanian legal and political context, USG context, Jordanian public activism, Jordanian civil society community] Sustainability 6. What IFES work with you do you think is more likely to be sustainable over time? What makes this work more sustainable? 7. What IFES work with you do you think is less likely to be sustainable over time? What makes this work less sustainable? 8. What should be considered to enhance the sustainability and local ownership of IFES’s past work with you? Lessons Learned 9. What do you think are some key lessons from IFES’s engagement with the IEC? NGOS AND CIVIL SOCIETY PARTNERS Key Informant Protocol Note This protocol is written so that it can be used for a range of key informants from different organizations. Individual interviews will stress parts of the questionnaire that directly focus on the engagement of each informant with the project (e.g. interviews with the IEC will focus on the evaluation question on how support built the capacity of the IEC). The interviewer will follow up these broad questions with targeted follow-up questions that gather more detail on the initial responses of informants that provide specific information on areas of the project with which they have been engaged with. Introduction Thank you for taking the time to talk with us today. My name is _______________ and I work for MSI with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. 116 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Our team is conducting a performance evaluation of the program implemented in Jordan by the International Foundation for Electoral Systems or IFES, International Republican Institute or IRI, and National Democratic Institute or NDI. We would like to learn from your experience engaging with these partners over the period 2010 through 2017. There are a standard set of best practices in evaluation that we will use, including anonymity, non￾attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your name or the name of your organization – or use the information you provide in such a way that can be traced back to you or your organization. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT INFORMANT EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the program? We are focused on the effectiveness of the program implementation, which is how these activities were implemented in Jordan. We would appreciate it if you can focus your responses on how the programs implemented by IFES/IRI/NDI worked directly with you and your organization. 1. How long have you been with your organization? 2. Please tell us about your position, background, and responsibilities in the organization? Effectiveness and Relevance 3. In your experience, how effective do you think NDI/IRI has been in implementing programs improving the ability of groups of politically active citizens and civil society organizations (CSOs) to: a. Participate in the electoral process b. Build grassroots demand c. Effectively advocate for a new legal framework for elections 4. Why have you been more effective or less effective? (Not for IFES) 5. In your experience, how effective do you think NDI/IRI/IFES has been in implementing programs developing avenues to engage the following groups in the election process and civic engagement? Why? a. women b. youth c. the disabled 6. In your experience, how effective do you think IRI has been in implementing programs encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls? Why? 7. In your experience, how effective do you think IRI has been in implementing programs strengthening political parties and alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels? Why? 8. In your experience, how effective do you think NDI has been in implementing programs bolstering public demand for local self-government? Why? 9. In your experience, how effective do you think IFES, IRI, and NDI have been in implementing programs strengthening the development of more democratic and open political processes and the participation of candidates, activists, monitors, and voters in elections? Why? [PROGRAM GOAL] 10. In your experience, do you think some of the work NDI or IRI did with your organization was USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 117 particularly effective or ineffective? What made the implementation of these particular activities more or less effective? 11. In your experience, how has the NDI/IRI program adapted to changes in the environment in which they work? [follow up – How about changes in the: Jordanian political context, USG context, Jordanian public activism, Jordanian civil society community] Sustainability 12. What NDI/IRI/IFES interventions do you think are more likely to be sustainable over time? What makes them more sustainable? 13. What NDI/IRI/IFES interventions do you think are less likely to be sustainable over time? What makes them less sustainable? 14. What should be considered to enhance sustainability and local ownership? Lessons Learned 15. Based on your experience with these programs, what do you think are some key lessons learned that can help NDI, IRI, and their partners under the current program? 16. Based on your experience with these programs, what do you think are some key lessons learned that can help the USAID Mission in these areas in their future strategy? GOVERNMENT OFFICIALS AND MPS Key Informant Protocol Note This protocol is written so that it can be used for a range of key informants from different organizations. Individual interviews will stress parts of the questionnaire that directly focus on the engagement of each informant with the project (e.g. interviews with the IEC will focus on the evaluation question on how support built the capacity of the IEC). The interviewer will follow up these broad questions with targeted follow-up questions that gather more detail on the initial responses of informants that provide specific information on areas of the project with which they have been engaged with. Introduction Thank you for taking the time to talk with us today. My name is _______________ and I work for MSI with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. Our team is conducting a performance evaluation of the program implemented in Jordan by the International Foundation for Electoral Systems or IFES, International Republican Institute or IRI, and National Democratic Institute or NDI. We would like to learn from your experience engaging with these partners over the period 2010 through 2017. There are a standard set of best practices in evaluation that we will use, including anonymity, non￾attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your name or the name of your organization – or use the information you provide in such a way that can be traced back to you or your organization. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT INFORMANT EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] 118 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the program? We are focused on the effectiveness of the program. We are most interested in how these activities were implemented in Jordan. We would appreciate it if you can focus your responses on the programs implemented by IFES, IRI, NDI and their work directly with you and your organization. 1. How long have you been with your organization? 2. Please tell us about your position, background, and responsibilities in the organization? Effectiveness and Relevance 3. In your experience, how effective do you think NDI/IRI has been in implementing programs improving the ability of groups of politically active citizens and civil society organizations (CSOs) to: a. Participate in the electoral process b. Build grassroots demand c. Effectively advocate for a new legal framework for elections d. Why have you been more effective or less effective? (Not for IFES) 4. In your experience, how effective do you think NDI/IRI/IFES has been in implementing programs developing avenues to engage the following groups in the election process and civic engagement? Why? a. women b. youth c. the disabled 5. In your experience, how effective do you think IRI has been in implementing activities strengthening political parties and alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels? Why? 6. In your experience, how effective do you think IRI has been in implementing activities encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls? Why? 7. In your experience, how effective do you think IRI and NDI have been in implementing activities bolstering public demand for local self-government and candidates’ commitment to open and more accountable performance as future members of parliament? Why? 8. In your experience, how effective do you think NDI, IRI, and IFES has been in implementing activities promoting the transparency and integrity of election process and building the capacity of the IEC to conduct transparent and credible election processes? Why? 9. In your experience, how effective do you think IFES, IRI, and NDI have been in implementing activities strengthening the development of more democratic and open political processes and the participation of candidates, activists, monitors, and voters in elections? Why? [PROGRAM GOAL] 10. In your experience, do you think the implementation of some NDI, IRI, and IFES activities was particularly effective or ineffective? What made the implementation of these particular activities more or less effective? 11. In your experience, how has the NDI/IRI/IFES program adapted to changes in the environment in which they work? [follow up – How about changes in the: Jordanian political context, USG context, Jordanian public activism, Jordanian civil society community] Sustainability 12. What NDI/IRI/IFES interventions do you think are more likely to be sustainable over time? What makes them more sustainable? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 119 13. What NDI/IRI/IFES interventions do you think are less likely to be sustainable over time? What makes them less sustainable? 14. What should be considered to enhance sustainability and local ownership? Lessons Learned 15. Based on your experience with these programs, what do you think are some key lessons learned that can help NDI, IRI, the IEC, and their partners under the current program? 16. Based on your experience with these programs, what do you think are some key lessons learned that can help the USAID Mission in these areas in their future strategy? POLITICAL PARTY REPRESENTATIVES Key Informant Protocol Note This protocol is written so that it can be used for a range of key informants from different organizations. Individual interviews will stress parts of the questionnaire that directly focus on the engagement of each informant with the project (e.g. interviews with the IEC will focus on the evaluation question on how IFES implemented activities to support built the capacity of the IEC). The interviewer will follow up these broad questions with targeted follow-up questions that gather more detail on the initial responses of informants that provide specific information on areas of the project with which they have been engaged with. Introduction Thank you for taking the time to talk with us today. My name is _______________ and I work for MSI with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. Our team is conducting a performance evaluation of the program implemented in Jordan by the International Foundation for Electoral Systems or IFES, International Republican Institute or IRI, and National Democratic Institute or NDI. We would like to learn from your experience engaging with these partners over the period 2010 through 2017. There are a standard set of best practices in evaluation that we will use, including anonymity, non￾attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your name or the name of your organization – or use the information you provide in such a way that can be traced back to you or your organization. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT INFORMANT EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the program? We are focused on the effectiveness of programs. We are most interested in how these activities were implemented in Jordan. We would appreciate it if you can focus your responses on the programs implemented by IFES, IRI, NDI and their work directly with you and your organization. 120 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV 1. How long have you been with your party? 2. Please tell us about your position, background, and responsibilities in the party? Effectiveness and Relevance 1. In your experience, how effective do you think NDI/IRI has been in implementing programs improving the ability of groups of politically active citizens and civil society organizations (CSOs) to: a. participate in the electoral process b. build grassroots demand c. Effectively advocate for a new legal framework for elections d. Why have you been more effective or less effective? 2. In your experience, how effective do you think IRI has been in implementing activities strengthening political parties and alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels? Why? a. Were there any IRI activities that were particularly useful to you? b. Were there any IRI activities that were less useful? c. Do your party members frequently draw on the knowledge they gained from working with IRI 3. In your experience, how effective do you think IRI has been in implementing activities encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls? Why? 4. In your experience, how effective do you think IFES, IRI, and NDI have been in implementing activities strengthening the development of more democratic and open political processes and the participation of candidates, activists, monitors, and voters in elections? Why? [PROGRAM GOAL] 5. In your experience, do you think the implementation of some IRI activities was particularly effective or ineffective? What made the implementation of these particular activities more or less effective? 6. In your experience, how has the IRI program adapted to changes in the environment in which they work? [follow up – How about changes in the: Jordanian political context, USG context, Jordanian public activism, Jordanian civil society community] Sustainability 7. What IRI interventions do you think are more likely to be sustainable over time? What makes them more sustainable? 8. What IRI interventions do you think are less likely to be sustainable over time? What makes them less sustainable? 9. What should be considered to enhance sustainability and local ownership? Lessons Learned 10. Based on your experience with these programs, what do you think are some key lessons learned that can help IRI, and their partners under the current program? 11. Based on your experience with these programs, what do you think are some key lessons learned that can help the USAID Mission in these areas in their future strategy? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 121 OTHER INTERNATIONAL DONORS AND INTERNATIONAL IMPLEMENTERS Key Informant Protocol Note This protocol is written so that it can be used for a range of key informants from different organizations. Individual interviews will stress parts of the questionnaire that directly focus on the engagement of each informant with the project (e.g. interviews with the IEC will focus on the evaluation question on how support built the capacity of the IEC). The interviewer will follow up these broad questions with targeted follow-up questions that gather more detail on the initial responses of informants that provide specific information on areas of the project with which they have been engaged with. Introduction Thank you for taking the time to talk with us today. My name is _______________ and I work for MSI with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. Our team is conducting a performance evaluation of the program implemented in Jordan by the International Foundation for Electoral Systems or IFES, International Republican Institute or IRI, and National Democratic Institute or NDI. We would like to learn from your experience engaging with these partners over the period 2010 through 2017. There are a standard set of best practices in evaluation that we will use, including anonymity, non￾attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your name or the name of your organization – or use the information you provide in such a way that can be traced back to you or your organization. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT INFORMANT EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the programs? As a performance evaluation, we are focused on the effectiveness of program implementation. We are most interested in how these activities were implemented in Jordan. We would appreciate it if you can focus your responses on the programs implemented by IFES, IRI, NDI and their collaboration with you and your partners. 1. How long have you been with your organization? 2. Please tell us about your position, background, and responsibilities in the organization? Effectiveness and Relevance 3. How aware are you of the USAID funded CEPPS program here Jordan? How do you know about these programs and implementers? 4. Are there any areas where your work overlaps with that of NDI, IRI, and IFES? 5. How do you coordinate with these organizations in the implementation of activities in Jordan? 6. Have there been any challenges in coordinating with these organizations? If so, how have these challenges been addressed? 122 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV 7. In general, how would you describe the relationships and coordination between your organization with IRI, NDI, and IFES? 8. What should be considered to enhance sustainability and local ownership? ELECTIONS AND GOVERNANCE STAKEHOLDERS Draft Key Informant Protocol Note This protocol is written so that it can be used for a range of key informants from different organizations. Individual interviews will stress parts of the questionnaire that directly focus on the engagement of each informant with the project (e.g. interviews with the IEC will focus on the evaluation question on how the implementation of IFES activities supported building the capacity of the IEC). The interviewer will follow up these broad questions with targeted follow-up questions that gather more detail on the initial responses of informants that provide specific information on areas of the project with which they have been engaged with. Introduction Thank you for taking the time to talk with us today. My name is _______________ and I work for MSI with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. Our team is conducting a performance evaluation of the program implemented in Jordan by the International Foundation for Electoral Systems or IFES, International Republican Institute or IRI, and National Democratic Institute or NDI. We would like to learn from your experience engaging with these partners over the period 2010 through 2017. There are a standard set of best practices in evaluation that we will use, including anonymity, non￾attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your name or the name of your organization – or use the information you provide in such a way that can be traced back to you or your organization. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT INFORMANT EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the program? We are focused on the effectiveness of programs. We are most interested in how these activities were implemented in Jordan. We would appreciate it if you can focus your responses on the programs implemented by IFES, IRI, NDI and their work directly with you and your organization. 1. How long have you been with your organization? 2. Please tell us about your position, background, and responsibilities in the organization? Effectiveness and Relevance 3. In your experience, how effective do you think NDI/IRI has been in implementing programs improving the ability of groups of politically active citizens and civil society organizations (CSOs) USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 123 to participate: a. in the electoral process b. build grassroots demand c. Effectively advocate for a new legal framework for elections d. Why have you been more effective or less effective? 4. In your experience, how effective do you think NDI/IRI/IFES has been in implementing programs developing avenues to engage the following groups in the election process and civic engagement? Why? a. women b. youth c. the disabled 5. In your experience, how effective do you think IRI has been in implementing programs strengthening political parties and alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels? Why? 6. In your experience, how effective do you think IRI has been in implementing programs encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls? Why? 7. In your experience, how effective do you think IRI and NDI has been in implementing programs bolstering public demand for local self-government and candidates’ commitment to open and more accountable performance as future members of parliament? Why? 8. In your experience, how effective do you think NDI and IRI have been in implementing programs promoting the transparency and integrity of election process and IFES in building the capacity of the IEC to conduct transparent and credible election processes? Why? 9. In your experience, how effective do you think IFES, IRI, and NDI have been in implementing programs strengthening the development of more democratic and open political processes and the participation of candidates, activists, monitors, and voters in elections? Why? [PROGRAM GOAL] 10. In your experience, do you think the implementation of some NDI, IRI and IFES activities were particularly effective or ineffective? What made the implementation of these particular activities more or less effective? 11. In your experience, how has the NDI, IRI and IFES programs adapted to changes in the environment in which they work? [follow up – How about changes in the: Jordanian political context, USG context, Jordanian public activism, Jordanian civil society community] Sustainability 12. What NDI, IRI and IFES interventions do you think are more likely to be sustainable over time? What makes them more sustainable? 13. What NDI, IRI and IFES interventions do you think are less likely to be sustainable over time? What makes them less sustainable? 14. What should be considered to enhance sustainability and local ownership? Lessons Learned 15. Based on your experience with these programs, what do you think are some key lessons learned that can help NDI, IRI, the IEC, and their partners under the current program? 16. Based on your experience with these programs, what do you think are some key lessons learned that can help the USAID Mission in these areas in their future strategy? 124 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Group Interview Protocol For Individual IRI/NDI Beneficiaries Thank you for taking the time to talk with us today. My name is _______________ and I work with the USAID Monitoring and Evaluation Support Project (MESP). MESP provides monitoring and evaluation support to the US Agency for International Development (USAID) in Jordan. [SECOND PERSON] My name is ____________. I am also working with MESP. Our team is conducting an evaluation of the program implemented in Jordan by the International Republican Institute or IRI and National Democratic Institute or NDI. We would like to learn from your experience engaging with these partners over the last two years. Doing an evaluation is a standard way of learning about how programs work, which is used by organizations all around the world. There are a standard set of best practices that we will use, including anonymity, non-attribution, and informed consent. We will take notes on our conversation. But in writing the report, we will not use your names– or use the information you provide in such a way that can be traced back to you. This interview is voluntary, and you have the opportunity to end the discussion at any time. Our discussion will take about an hour. Do you agree to participate in our interview under these conditions? [ENSURE THAT INFORMANT EXPLICTLY ANSWERS YES TO AGREE TO PARTICIPATE] Do you have any questions before we start with a set of questions that USAID has asked us to focus on about the programs you participated in with IRI or NDI? 1. Please tell us a little bit about how engaged you have been with the IRI or NDI program. • How many trainings have you have attended? • How many activities you have participated in? 2. Please describe how you learned about this program and why you decided to participate. 3. How has this program been valuable for you? 4. What were some of the things that NDI or IRI did that was particularly interesting or valuable for you? 5. Was there anything NDI or IRI did in the activities you participated in that you thought could have been done better? How can the partners improve the implementation of their activities? 6. What kind of activities or training would you like to see NDI or IRI do in the future? 125 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV CEPPS Objective Map The ET has mapped the 23 objectives of the 2010-2017 CEPPS award to the six objectives as written in the evaluation SOW. This has entailed editing the objectives from the SOW to ensure that they explicitly cover all 23 objectives pursued under the CEPPS program. The ET has done this exercise to ensure that the evaluation covers all the work of the CEPPS partners. The ET’s edits for the expanded objectives are in italics below. CEPPS EVALUATION OBJECTIVE 1 - Improving the ability of groups of politically active citizens and civil society organizations (CSOs) to participate in the electoral process, build grassroots demand and effectively advocate for a new legal framework for elections IRI – OBJECTIVE (0)4 Jordanian voters made informed choices in elections based on increased knowledge of election procedures and processes, as well as through improved platforms, messaging and campaigning by candidates and parties NDI –01 Strengthen civil society’s capacity to effectively advocate for a new legal framework on decentralization, elections and political parties 04 Build civil society’s capacity to strategically and effectively monitor parliament and national and sub‐national elections; 08 strengthen the relationship between parliament and civil society, and between parliamentarians and their constituents (because done by working with CSOs), 010 encouraging a more participatory political process working with government institutions CEPPS OBJECTIVE 2 - Developing avenues to engage women, youth and the disabled in the election process and civic engagement IRI – 02 Women and youth demonstrate that they are valued members of political parties by promoting the party’s agenda and profile; 06 Women from rural and urban communities become politically active as community leaders, educators, candidates, campaign workers, and activists NDI –02 Develop civil society’s capacity to engage youth in electoral and political processes 03 Expand and strengthen women’s participation CEPPS OBJECTIVE 3 -Strengthening political parties and alliances and the ability of candidates to articulate, organize and implement clear political alternatives at the national and sub-national levels IRI – 01 Political parties in Jordan develop issue-oriented platforms and policy positions and communicate them to citizens CEPPS OBJECTIVE 4 - Encouraging issue-based campaigns informed by public opinion research, and the completion of nationwide pre-election and exit polls IRI – 07 polling (Jordanian citizens, political parties and organizations have greater access to qualitative and quantitative public opinion research) CEPPS OBJECTIVE 5 - Bolstering public demand for local self-government and candidates’ commitment to open and more accountable performance as future members of parliament 126 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV IRI - 03 Groups of politically active citizens increased their capacity to interact with and advocate to elected officials and governmental bodies for the needs of their communities NDI – 05 Bolster public demand for ensuring candidates' commitment to a more open and accountable performance as future members of parliament 09 work to improve citizens understanding of LSG, 011 work with communities hosting refugees on common concerns CEPPS OBJECTIVE 6 - Promoting the transparency and integrity of election processes and building the capacity of the IEC to conduct transparent and credible election processes. IFES - 01 Build the IEC’s long-term institutional capacity and sustainability as Jordan’s election management body; 02: Strengthen the IEC’s legal framework for the electoral administration process 03: Strengthen the IECs electoral management capacities to administer national and sub-national elections 04: Build public confidence in the IEC through the support of the IEC's public and media outreach IRI – 05 assessment and observation (Jordanian elections are carried out according to Jordanian Law and international standards) NDI – electoral assessment (O6 Promote the transparency and integrity of Jordan’s electoral process through international assessment of the pre-election, election day, and immediate post-election period; Electoral observation (O7 - Strengthen the electoral process by identifying real or potential problems, including any irregularities, logistical, or implementation problems and impediments from external actors, and offering recommendations on how these problems can be resolved). Note IRI changed objectives from those used in the early period of the cooperative agreement. The ET used the Objectives from FY 2012 Q2. The objective map will help the ET’s organization of the evaluation, particularly where to place our data, findings, and conclusions. One example of how we shall address this overlap challenge is in NDI Objective 9, for example here Indicator 9.1.3: Number of youth that participate in discussions on effective local government as part of the CEPPS/NDI’s Ana Usharek program. These activities and data could fit with the work on local self-governance in CEPPS Objective 5, but we will primarily put these data in CEPPS Objective 2 focused on youth based on our understanding that NDI’s work under Ana Usharek is primarily on youth engagement. The data for this example comes from the Final Report, “Over the course of the program, 7,376 Jordanian youth in total participated in guided discussion sessions on local governance led by CEPPS/NDI coordinators as part of the Ana Usharek program.” (p. 129). 127 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV ANNEX C. SURVEY QUESTIONNAIRE SURVEY MODULES A. Respondent background (pre-coded) 127 B. TRAINING EXPOSURE AND PERCEPTIONS 128 C. GENERAL OUTLOOK 130 D. ENGAGEMENT AND AGENCY 131 E ATTITUDES AND PERCEPTIONS 138 F. DEMOGRAPHICS 142 A. RESPONDENT BACKGROUND (PRE-CODED) # Question Response A1 Questionnaire No. A2 Respondent # A3 Respondent Name A4 Program(s) A5 Enumerator Name A6. NOW, CALL {RESPONDENT #}. • Hello, my name is _______ from Mindset, a research company currently conducting a study with beneficiaries of the “Ana Usharek”, “Usharek +” and “Youth Leadership Academy” programs in the last few years • We are calling you because you participated in a youth workshop or training program through Usharek or Youth Leadership Academy and we are interested in your opinions about a number of topics, such as democracy, governance and local and national elections • This survey is part of a larger effort to assess the effectiveness and results of the support to the development of more democratic and open political processes through the CEPPS program that was implemented by NDI and IRI • Your answers will help us understand the views of youth in politics and assess how youth-focused programs such as Usharek and Youth Leadership Academy are impacting beneficiaries • There is no right or wrong answer – we are only interested in your own personal thoughts • You may refuse to answer any question, and you may end the interview at any time • All answers that you do provide are completely confidential and we will never provide any information about you to anyone 128 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV • Your participation in this survey is voluntary and you will not receive compensation for taking part • The interview should last 25 minutes. May we have your permission to proceed with the interview? ⬜ 1. Accepted to participate in the interview ⬜ 2. Refused the interview ---- (END INTERVIEW) ⬜ 3. Wrong number - heard a message stating that the number is wrong ---- (END INTERVIEW) ⬜ 4. Not in use - Heard a message that the number is not used ---- (END INTERVIEW) ⬜ 5. Number is switched off - Heard a message stating that the number is closed---- (END INTERVIEW) ⬜ 6. The number is disconnected - hear a message stating that the number is disconnected ---- (END INTERVIEW) ⬜ 7. No answer ---- (END INTERVIEW) ⬜ 8. The requested person has died ---- (END INTERVIEW) ⬜ 9. The requested person has travelled ---- (END INTERVIEW) ⬜ 10. Asked to be called again at a later time ---- (END INTERVIEW) ⬜ 11. Called before ---- (END INTERVIEW) ⬜ 12. Respondent language problem ---- (END INTERVIEW) ⬜ 13. No phone number ---- (END INTERVIEW) ⬜ 14. The number is for another person ---- (END INTERVIEW) ⬜ 15. The respondent did not participate in any training ---- (END INTERVIEW) A7. IS YOUR AGE AT LEAST 18 YEARS OLD? ⬜ 1. Yes ⬜ 2. No (END INTERVIEW) - If you have any questions or comments, please feel free to contact MSI at 00962065925138 B. TRAINING EXPOSURE AND PERCEPTIONS Interviewer prompt: In this section, we will ask you some questions regarding your involvement in activities related to political participation and your view regarding their effectiveness. 1. Have you ever participated in a training or an activity related to political participations, elections, civic engagement or advocacy? ⬜ 1. Yes ⬜ 2. No (END INTERVIEW) ⬜ 98. Refused (vol.) (END INTERVIEW) 2. Have you participated in USAID activities or trainings related to political participations, elections, civic engagement or advocacy? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 129 ⬜ 1. Yes ⬜ 2. No (SKIP to B4) ⬜ 98. Refused (vol.) (SKIP to B4) 3. About how many USAID political-participation activities or trainings have you participated in since January 2016? ⬜ 1. 0 ⬜ 2. 1-2 ⬜ 3. 3-5 ⬜ 4. More than 5 ⬜ 5. Cannot recall ⬜ 98. Refused (vol.) 4. In thinking about the trainings that you have attended, which of the following best describes the types of trainings/activities that you participated in (select all that apply)? ⬜ 1. Traditional/Classroom trainings ⬜ 2. One to one mentorship programs ⬜ 3. On the job trainings/internships ⬜ 4. Peer led trainings/student led trainings ⬜ 5. Online trainings ⬜ 7. Community lead trainings ⬜ 8. One day training or workshops ⬜ 9. Volunteer activities or community led initiatives ⬜ 6. Other ______________ ⬜ 98. Refused (vol.) 5. In thinking about the types of trainings you listed above, how would you rank the effectiveness of each type of training [ Interview can prompt Which do you think is the most effective? Which do you think is the second most effective? Which do you third most? Which do you think is fourth most effective? [Interviewer: Continue ranking all sources mentioned above] 130 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Responses from QB4 Rank Ref (vol.) DK (vol.) 98 99 98 99 98 99 98 99 98 99 6. Are there any specific topics or subjects related to political participation, elections, civic engagement or advocacy (for example: campaign management, electoral law, polling etc.) that you would like to have training on, either first time training or additional training? ⬜ 1. Yes ⬜ 2. No (SKIP to C1) ⬜ 98. Refused (vol.) (SKIP to C1) What topics or subjects would you like to receive training on? (Open ended, list all) C. GENERAL OUTLOOK Interviewer Prompt: In this section, we will ask you some questions regarding your thoughts and opinions on the current state of the country and where it may be headed. 1. Overall, is Jordan headed in the right or wrong direction? ⬜ 1. Things are going mostly in the right direction ⬜ 2. Things are going somewhat in the right direction ⬜ 3. Things are going somewhat in the wrong direction ⬜ 4. Things are going mostly in the wrong direction ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 2. What, in your opinion, is the single biggest problem facing Jordan as a whole? (Record verbatim and post code) ⬜ 1. Rising cost of living ⬜ 2. Unemployment ⬜ 3. Poverty ⬜ 4. Terrorism ⬜ 5. Jordan’s economic condition ⬜ 6. Corruption (administrative and financial) ⬜ 7. Refugee influx USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 131 ⬜ 8. Widespread drug use ⬜ 9. Income ⬜ Other: ____________________ ⬜ 98. Refused (vol.) 3. How would you describe your current household economic situation? ⬜ 1. Very good ⬜ 2. Good ⬜ 3. Bad ⬜ 4. Very bad ⬜ 98. Refused (vol.) ⬜ 99. Don’t know (vol.) 4. In the next 12 months do you think your household economic situation is likely to get: ⬜ 1. A lot better ⬜ 2. Somewhat better ⬜ 3. Somewhat worse ⬜ 4. A lost worse ⬜ 5. Stay about the same ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 5. To Interviewer: Do not read. This is the first break off point. If the respondent wants to discontinue the interview, select "break off" and save the responses that have been collected up to this point. ⬜ 1. Continue ⬜ 2. Break off - Respondent does not want to continue (END INTERVIEW) D. ENGAGEMENT AND AGENCY Interviewer prompt: Now, we will ask a set of questions about your ability to participate in the political process as well as your own recent engagement with local, regional and national politics. 1. In the past year, have you ever volunteered your time to address an immediate need facing your community? Examples could include helping people in need, or removing litter on the street, or joining advocating for a change in your community. ⬜ 1. Yes ⬜ 2. No ⬜ 98. Refused (vol.) 2. In the past year, have you attended a public meeting with public officials at any level of government (mayor, municipal councils, MPs etc.)? 132 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV ⬜ 1. Yes ⬜ 2. No → C6 ⬜ 98. Refused (vol.) → C6 3. What type of meeting? ⬜ 1. Town hall or other public meeting ⬜ 2. Civil society organization meeting ⬜ 3. Tribal meeting ⬜ 4. Informal community meetings ⬜ 6. Other: ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 4. About how many times have you attended such a public meeting in the last year? ⬜ 1. 1-3 times ⬜ 2. 4-6 times ⬜ 3. At least 7 times ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 5. Did you speak at any of the meetings you attended? ⬜ 1. Yes ⬜ 2. No ⬜ 98. Refused (vol.) 6. In the past year, have you ever contacted or visited a public official at any level of government in order to share your opinion or express a concern? ⬜ 1. Yes ⬜ 2. No → C8 ⬜ 98. Refused (vol.) → C8 7. If yes, whom did you visit or contact (select all that apply) ⬜ 1. Local council ⬜ 2. Municipal council ⬜ 3. Mayor ⬜ 4. Governor ⬜ 5. Governorate council ⬜ 6. Member of Parliament ⬜ 7. Minister or Ministry official ⬜ 98. Refused (vol.) 8. Have you ever done any of the following activities in the past? (select all that apply) ⬜ 1. Run for office USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 133 ⬜ 2. Joined a political party ⬜ 3. Started a political party ⬜ 4. Joined an NGO/CSO ⬜ 5. Started an NGO/CSO ⬜ 6. Been actively involved in a political campaign ⬜ 98. Refused (vol.) 9. How likely or unlikely are you to do the following activities in the future? (ask all) Very likely Somewhat likely Somewhat unlikely Very Unlikely Ref (vol.) DK (vol.) Run for office 1 2 3 4 98 99 Join a political party 1 2 3 4 98 99 Start a political party 1 2 3 4 98 99 Join an NGO or CSO 1 2 3 4 98 99 Start an NGO or CSO 1 2 3 4 98 99 Become actively involved in a political campaign 1 2 3 4 98 99 10. Did you vote in the 2016 parliamentary elections? ⬜ 1. Yes → C11 ⬜ 2. No ⬜ 3. No, not eligible to vote due to age at the time of the elections ⬜ 4.Cannot recall→ C11 10 a. If no, would you please share what the main reason for not voting was?? [INTERVIEWER do not read list. Will be post-coded] ⬜ 1. I did not trust the candidates ⬜ 2. I did not see any benefits from these elections ⬜ 3. No trust in the electoral process/not convinced in the electoral process ⬜ 4. Not interested ⬜ 5. I was too busy with other commitments ⬜ 6. Family Issues ⬜ 7. Illness/indisposed ⬜ 8. No accessible polling stations by me ⬜ 9. Fear of reprisals/intimidation 134 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV ⬜ 10. Other ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 11. Did you vote in the 2017 decentralization and municipal elections? ⬜ Yes→ C12 ⬜ No ⬜ Cannot recall → C12 ⬜ 98. Refused (vol.) → C12 ⬜ 99. Don’t Know (vol.) → C12 11a. If no, what is the main reason you did not vote? [INTERVIEWER do not read list. Will be post￾coded] ⬜ 1. I did not trust the candidates ⬜ 2. I did not see any benefits from these elections ⬜ 3. No trust in the electoral process/not convinced on the electoral process ⬜ 4. Not interested ⬜ 5. I was too busy with other commitments ⬜ 6. Family Issues ⬜ 7. Illness/indisposed ⬜ 8. No accessible polling stations by me ⬜ 9. Fear of reprisals/intimidation ⬜ 10. Other ⬜ 98. Refused (vol.) ⬜ 9. Don’t Know (vol.) 12. How likely or unlikely are you to vote in the next round of national or local elections? ⬜ 1. Very likely ⬜ 2. Somewhat Likely ⬜ 3. Somewhat Unlikely ⬜ 4. Very Unlikely ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 13. How often do you discuss news and political events with your family? ⬜ 1. Daily ⬜ 2. A few times a year (2-4) ⬜ 3. Several times a year (5-7) USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 135 ⬜ 4. About once a month ⬜ 5. About once a week ⬜ 6. Never ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 14. How often do you discuss news, political events with your friends? ⬜ 1. Daily ⬜ 2. A few times a year ⬜ 3. Several times a year ⬜ 4. About once a month ⬜ 5. About once a week ⬜ 6. Never ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 15. How often do you discuss news, political events with your colleagues/peers (other students)? ⬜ 1. Daily ⬜ 2. A few times a year (2-4) ⬜ 3. Several times a year (5-7) ⬜ 4. About once a month ⬜ 5. About once a week ⬜ 6. Never ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 16. How often do you discuss news, political events on social media? ⬜ 1. Daily ⬜ 2. A few times a year (2-4) ⬜ 3. Several times a year (5-7) ⬜ 4. About once a month ⬜ 5. About once a week ⬜ 6. Never ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 17. How many sources you use to get news and information about politics, economics and social issues? 18. Where do you get news and information about politics, economics and social issues? Please tell me as many sources of news and information as you use. (Interviewer: If the respondent 136 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV cannot answer prompt with: radio, television, friends, family, social media, mobile phone, newspapers etc.) [response to be post-coded] 19. Now, thinking about the sources of news and information that you just mentioned, please tell me which of them do you trust the most to give you important news and information? Which do you trust the second most? Which do you trust the third most? Which do you trust the fourth most? Which do you trust the fifth most? [Interviewer: Continue ranking all the sources mentioned above] A6. Source of News and Information A7. Trust Ranking 20. To what degree do you care about political reform in Jordan? ⬜ 1. To a large degree ⬜ 2. To a moderate degree ⬜ 3. To a little degree ⬜ 4. Not at all ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 21. To what degree do you feel citizens should engage in the development of their community? ⬜ 1. To a large degree ⬜ 2. To a moderate degree ⬜ 3. To a little degree ⬜ 4. Not at all ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 22. To what degree do you feel you have a say in governmental decision-making on issues that directly affect you? ⬜ 1. To a large degree ⬜ 2. To a moderate degree ⬜ 3. To a little degree ⬜ 4. Not at all ⬜ 98. Don’t Know (vol.) USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 137 ⬜ 99. Refused (vol.) 23. Do you agree or disagree with this statement: “Politicians do not listen to the needs and ideas of young people”? ⬜ 1. Strongly Agree ⬜ 2. Agree ⬜ 3. Disagree ⬜ 4. Strongly disagree ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 24. To what degree do you feel that you have the ability to actively engage in the political process as a candidate? ⬜ 1. To a large degree ⬜ 2. To a moderate degree ⬜ 3. To a little degree ⬜ 4. Not at all ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 25. To what degree do you feel that you have the ability to actively engage in the political process by working with a campaign/volunteering with a campaign ⬜ 1. To a large degree ⬜ 2. To a moderate degree ⬜ 3. To a little degree ⬜ 4. Not at all ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 26. To what degree do you have the ability to work in your community to address a common problem through advocacy? ⬜ 1. To a large degree ⬜ 2. To a moderate degree ⬜ 3. To a little degree ⬜ 4. Not at all ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 27. I’m going to read out some different forms of political action that people can take, and I’d like you to tell me, for each one, whether you support people taking this action in most circumstances, support people taking this action in some circumstance or do not support people taking this action under any circumstance. 138 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV 26. Political Action 26. Response 1. Support in most circumstances 2. Support, only in certain circumstances 3. Would never support under any circumstances 98. Refused (vol) 99 Don’t Know (vol) a. Sign a petition b. Join a boycott c. contacts a government representative (letter writing campaigns, calling MPs, organizing meetings) d. Join a strike e. Join a protest 28. In the past year, did you take part in such actions? ⬜ 1. Yes ⬜ 2. No ⬜ 99. Refused (vol.) 29. To Interviewer: Do not read. This is the second break off point. If the respondent wants to discontinue the interview, select "break off" and save the responses that have been collected up to this point. ⬜ 1. Continue ⬜ 2. Break off - Respondent does not want to continue (Skip to I01: Enumerator Name) E. ATTITUDES AND PERCEPTIONS Interviewer prompt: In this section, you will be asked questions regarding your opinions and perceptions on various government institutions and political processes. 1. How much confidence do you have in [Insert item]? Is it no confidence at all, a little confidence, a moderate amount of confidence, or a great deal of confidence? None at all A little Moderate amount A great deal Ref (vol.) DK (vol.) a. Military 1 2 3 4 98 99 USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 139 None at all A little Moderate amount A great deal Ref (vol.) DK (vol.) b. Judiciary 1 2 3 4 98 99 c. Religious Institutions 1 2 3 4 98 99 d. Doctors and Hospitals 1 2 3 4 98 99 e. Schools and Universities 1 2 3 4 98 99 f. Municipal Councils 1 2 3 4 98 99 g. Parliament 1 2 3 4 98 99 h. The Prime Minister 1 2 3 4 98 99 i. Political Parties 1 2 3 4 98 99 2. In your view, how responsive are the following institutions to the priorities, preferences and needs of the Jordanians? Very Respo nsive Some what respo nsive Very unresponsive Not respon sive at all NA Ref (vol.) DK (vol.) j. Military 1 2 3 4 97 98 99 k. Judiciary 1 2 3 4 97 98 99 l. Religious Institutions 1 2 3 4 97 98 99 m. Doctors and Hospitals 1 2 3 4 97 98 99 n. Schools and Universities 1 2 3 4 97 98 99 o. Municipal Councils 1 2 3 4 97 98 99 p. Parliament 1 2 3 4 97 98 99 q. The Prime Minister 1 2 3 4 97 98 99 r. Political Parties 1 2 3 4 97 98 99 3. How effective or ineffective do you believe the current parliament is in exercising its duties of monitoring the government and issuing legislation? ⬜ 1. Very Effective ⬜ 2. Somewhat effective ⬜ 3. Somewhat ineffective 140 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV ⬜ 4. Very ineffective ⬜ 98. Don’t know (vol.) ⬜ 99. Refused (vol.) 4. In your opinion, do you think female members of parliament are more effective than male members of parliament, less effective or about the same? ⬜ 1. Women are more effective than men ⬜ 2. Women and men have the same level of effectiveness ⬜ 3. Men are more effective than women ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 5. How effective or ineffective do you believe the IEC is in ensuring fair and transparent elections? ⬜ 1. Very Effective ⬜ 2. Somewhat effective ⬜ 3. Somewhat ineffective ⬜ 4. Very ineffective ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 6. How effective do you think your [Insert Item] is/are in delivering services to the local people in your area? Is [insert item] not at all effective, a little effective, moderately effective, or extremely effective? Extremely effective Moderatel y effective A little effective Not all effective NA Ref (vol.) DK (vol.) a. Local council members 1 2 3 4 97 98 99 b. Local council head 1 2 3 4 97 98 99 c. Municipal council 1 2 3 4 97 98 99 d. Mayor 1 2 3 4 97 98 99 e. Governor 1 2 3 4 97 98 99 f. Governorate council 1 2 3 4 97 98 99 7. Have you heard of any political parties in Jordan? ⬜ 1. Yes ⬜ 2. No →9 ⬜ 99. Refused →9 8. If yes, can you please tell me what political parties you have heard of (name all that you know) 9. To what extent do you believe that elections are generally fair in Jordan? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 141 ⬜ 1. Elections in Jordan are generally very fair ⬜ 2. Elections in Jordan are generally somewhat fair ⬜ 3. Elections in Jordan are generally not fair ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 10. To what extent do you believe that elections are generally transparent in Jordan? ⬜ 1. Elections in Jordan are generally very transparent ⬜ 2. Elections in Jordan are generally somewhat transparent ⬜ 3. Elections in Jordan are generally not transparent ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 11. To what degree do you agree or disagree with this statement: “Women should be equally represented in political decision making in Jordan? ⬜ 1. Strongly Agree ⬜ 2. Agree ⬜ 3. Disagree ⬜ 4. Strongly disagree ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 12. To what degree do you agree or disagree with this statement: “Youth should be represented MORE in political decision making in Jordan? ⬜ 1. Strongly Agree ⬜ 2. Agree ⬜ 3. Disagree ⬜ 4. Strongly disagree ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 13. In your opinion, in regard to people with disabilities’ engagement in the political process, would you say that they are engaged? ⬜ 1. Less than they should be ⬜ 2. About the right amount ⬜ 3. More than they should be ⬜ 98. Don’t Know (vol.) ⬜ 99. Refused (vol.) 142 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV F. DEMOGRAPHICS Interviewer prompt: In this section, we will ask you a set of questions regarding basic demographic information. 1. How old are you? ______ 2. What is your gender? ⬜ 1. Female ⬜ 2. Male 3. In which governorate and district do you currently live? Governate: District: ⬜ 98. Refused (vol.) 4. Were you born in this governorate? ⬜ 1. Yes → D6 ⬜ 2. No ⬜ 98. Refused (vol.) ⬜ 99. Don’t know (vol.) 4a. Were you born in this district? ⬜ 1. Yes → D6 ⬜ 2. No ⬜ 98. Refused (vol.) ⬜ 99. Don’t know (vol.) 5. In what governorate/district were you born? Governorate: District: ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 6. What is your highest level of education: ⬜ 1. Some High School ⬜ 2. High School Graduate ⬜ 3. Some College/University ⬜ 4. University or College Degree ⬜ 5. Some Graduate Degree course work ⬜ 6. Graduate Degree (Masters, PhD, MBA etc.) ⬜ 7. Other USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 143 ⬜ 98. Refused (vol.) 7. Are you currently working? [Interviewer: clarify if needed, working means formally employed part time or full time]: ⬜ 1. Yes → D9 ⬜ 2. No ⬜ 98. Refused (vol.) 8. Are you currently looking for a job? ⬜ 1. Yes ⬜ 2. No ⬜ 98. Refused (vol.) 9. What is your marital status? [Interviewer do not read out options] ⬜ 1. Single ⬜ 2. Married ⬜ 3. Divorced ⬜ 4. Widowed ⬜ 5. Other ⬜ 98. Refused (vol.) 10. What is your average household income (monthly), include all sources of income: ⬜ 1. Less than 200 JD ⬜ 2. 200- 399 JD ⬜ 3. 400-599 JD ⬜ 4. 600-799 JD ⬜ 5. 800 -999 JD ⬜ 6. 1,000+ JD ⬜ 98. Refused (vol.) ⬜ 99. Don’t Know (vol.) 144 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV ANNEX D. SURVEY METHODOLOGY TECHINICAL REPORT STUDY OBJECTIVES MSI/MESP is conducting a performance evaluation of the Consortium for Elections and Political Process Strengthening (CEPPS) program. The CEPPS program encompasses work in numerous areas related to democracy, governance, and political processes, including assistance for election administration, training and deploying election monitors, strengthening civil society, municipal governance, and support for parliament. The evaluation will cover the work done by the CEPPS implementing partners, IFES, IRI and NDI, over a period of seven years (2010-2017). The evaluation also assesses the most recent effort by partners around the Decentralization elections, in addition to the program’s overall progress towards achieving intended objectives and document lessons learned and best practices to inform future DRG programming. As part of this evaluation, Mindset conducted a phone survey of beneficiaries from Ana Usharek, Usharek+ and the Youth Leadership Academy (YLA) program activities. The survey’s objective was to gauge beneficiary perceptions on the effectiveness, relevance, and sustainability of CEPPS and to provide data that can be compared to existing national poll results on issues of political engagement (including voting behavior), civic and democratic knowledge and attitudes, and perceptions of agency. STUDY OVERVIEW Mindset conducted phone interviews with 452 respondents, whereas the set target was 475 respondents. The minimum overall sample size of 475 reflects the aggregate population of approximately 12,725 registered beneficiaries, which were provided to Mindset by MSI/MESP. Please refer to Table 3 for a detailed breakdown of the sample quotas provided by MESP and the achieved quotas by Mindset. The survey timeline extended from May 2018 till July 19, 2018 when the final dataset was delivered. RESPONDENTS Mindset targeted the CEPPS program beneficiaries as provided by MSI/MESP. Respondents were asked if they participated in any political training to ensure they are in fact program beneficiaries, in order to proceed with conducting the phone interviews. SAMPLE DESIGN AND SELECTION MSI/MESP provided a list of 12,681 beneficiaries of the Ana Usharek, Usharek+, or the Youth Leadership Academy program activities; excluding the 32 beneficiaries that were interviewed in the pilot stage of the program. Once the sample lists were received from MSI/MESP, Mindset prepared the lists for calling by adding necessary variables needed to set interviews. This resulted in several steps of refinement such as: consolidating lists, removing duplicates, and listing all training programs beneficiaries participated in. Table 1 below illustrates the total universe for all segments once the lists have been sorted. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 145 TABLE 1. SAMPLE UNIVERSE Segment / Program Sample Universe Ana Usharek 10,419 Usharek+ 2,144 YLA 118 Total 12,681 Subsequently, the sample size for this survey was determined by MSI/MESP at 475 respondents in order to achieve a 95.0% level of confidence with a 7.0% margin of error. The quota per program was broken down as follows: TABLE 2. QUOTA BREAKDOWN Segment / Program Quota Ana Usharek 200 Usharek+ 200 YLA 75 Total 475 Once the quotas were set, Mindset randomized the list and initiated the data collection phase. RESPONSE RATES Mindset has successfully interviewed 452 beneficiaries; however, the YLA quota was not achieved as shown in table 4. This was due to due to insufficient sample lists. Mindset usually requests that the list of beneficiaries is at least three times the requested sample to ensure the quota is met. 146 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV TABLE 3. SUCCESSFUL PHONE INTERVIEWS BREAKDOWN Segment / Program Quota Achieved Achieved % Ana Usharek 200 201 100.5% Usharek+ 200 202 101.0% YLA 75 49 65.3% Total 475 452 95.2% Table 4 below shows that Mindset contacted 12.8% of the beneficiaries listed within MSI/MESP lists. The remaining 87.2% were not contacted because they were either unavailable by phone or the quota was already met for the program they belonged to. Out of the 1,626 respondents Mindset contacted, 452 (27.8%) agreed to participate. TABLE 4. MSI/MESP SAMPLE LIST BREAKDOWN Total sample received from MSI/MESP 12,681 100.0% Total not contacted to set an appointment through the phone 11,055 87.2% Completed quota 6,774 61.3% No answer 1,178 10.7% No phone number 7 0.1% Not in use 2,122 19.2% Number is switched off 613 5.5% The number is disconnected 336 3.0% Wrong number – wrongly formatted numbers 25 0.2% Total contacted for a phone interview 1,626 12.8% Unsuccessful phone outcomes 1,174 72.2% Asked to be called again at a later time 483 41.1% The number is for a different person 309 26.3% Respondent was contacted previously 6 0.5% Respondent language problem 1 0.1% The requested person has died 4 0.3% USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 147 Total sample received from MSI/MESP 12,681 100.0% The requested person was traveling 8 0.7% The respondent did not participate in any training 157 13.4% Partial Interview 23 2.0% Respondent refused to be interviewed (selected respondent unwilling or unable to complete the interview) 183 15.6% Successful Phone Outcome 452 27.8% Agreed to a phone interview 452 27.8% Table 5 below shows that Mindset contacted 63.6% of the YLA beneficiaries. The remaining 36.4% were not contacted because they were unavailable by phone. Out of the 75 respondents Mindset contacted, 49 (65.3%) agreed to participate. TABLE 5. YLA SAMPLE LIST BREAKDOWN Total sample received from MSI/MESP 118 100.0% Total not contacted to set an appointment through the phone 43 36.4% Completed quota 0 0.0% No answer 13 30.2% No phone number 0 0.0% Not in use 20 46.5% Number is switched off 5 11.6% The number is disconnected 5 11.6% Wrong number – wrongly formatted numbers 0 0.0% Total contacted for a phone interview 75 63.6% Unsuccessful phone outcomes 26 34.7% Asked to be called again at a later time 4 15.4% The number is for a different person 2 7.7% Respondent was contacted previously 3 11.5% Respondent language problem 0 0.0% 148 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Total sample received from MSI/MESP 118 100.0% The requested person has died 0 0.0% The requested person was traveling 0 0.0% The respondent did not participate in any training 3 11.5% Partial Interview 1 3.8% Respondent refused to be interviewed (selected respondent unwilling or unable to complete the interview) 13 50.0% Successful phone outcomes 49 65.3% Agreed to a phone interview 49 65.3% Table 6 below shows that Mindset contacted 9.4% of the Ana Usharek beneficiaries. 90.6% were not contacted because they were either unavailable by phone or the quota was already met for the program. Out of the 980 respondents Mindset contacted, 201 (20.5%) agreed to participate. TABLE 6. ANA USHAREK SAMPLE LIST BREAKDOWN Total sample received from MSI/MESP 10,419 100.0% Total not contacted to set an appointment through the phone 9,439 90.6% Completed quota 6,605 70.0% No answer 738 7.8% No phone number 1 0.0% Not in use 1,430 15.1% Number is switched off 423 4.5% The number is disconnected 226 2.4% Wrong number – wrongly formatted numbers 16 0.2% Total contacted for a phone interview 980 9.4% Unsuccessful phone outcomes 779 79.5% Asked to be called again at a later time 288 37.0% The number is for a different person 213 27.3% Respondent was contacted previously 3 0.4% USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 149 Total sample received from MSI/MESP 10,419 100.0% Respondent language problem 1 0.1% The requested person has died 2 0.3% The requested person was traveling 6 0.8% The respondent did not participate in any training 147 18.9% Partial Interview 8 1.0% Respondent refused to be interviewed (selected respondent unwilling or unable to complete the interview) 111 14.2% Successful phone outcomes 201 20.5% Agreed to a phone interview 201 20.5% Table 7 below shows that Mindset contacted 26.6% of the Usharek+ beneficiaries. 73.4% were not contacted because they were either unavailable by phone or the quota was already met for the program. Out of the 571 respondents, Mindset contacted 202 (35.4%) agreed to participate. TABLE 7. USHAREK+ SAMPLE LIST BREAKDOWN Total sample received from MSI/MESP 2,144 100.0% Total not contacted to set an appointment through the phone 1,573 73.4% Completed quota 169 10.7% No answer 427 27.1% No phone number 6 0.4% Not in use 672 42.7% Number is switched off 185 11.8% The number is disconnected 105 6.7% Wrong number – wrongly formatted numbers 9 0.6% Total contacted for a phone interview 571 26.6% Unsuccessful phone outcomes 369 64.6% Asked to be called again at a later time 191 51.8% The number is for a different person 94 25.5% 150 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Total sample received from MSI/MESP 2,144 100.0% Respondent was contacted previously 0 0.0% Respondent language problem 0 0.0% The requested person has died 2 0.5% The requested person was traveling 2 0.5% The respondent did not participate in any training 7 1.9% Partial Interview 14 3.8% Respondent refused to be interviewed (selected respondent unwilling or unable to complete the interview) 59 16.0% Successful phone outcomes 202 35.4% Agreed to a phone interview 202 35.4% QUESTIONNAIRE REVIEW MSI/MESP provided the questionnaire, which Mindset then reviewed. There were minor changes made in the questionnaire to clarify the options to respondents. 6.1 TRANSLATION PROCESS: Once the wording of the questionnaire was finalized, the translation from English to Arabic was initiated. The translation followed the steps below: • The questionnaire was translated by a professional translator. • The translation was reviewed by senior project staff and amendments were conducted accordingly as seen fit. • The translation was reviewed a second time by a different senior project member by comparing the translation with the English version of the questionnaire. In addition to producing an accurate translation, this process also ensured that key project staff are fully engaged in the questionnaire and are ready to train interviewers and answer their questions during training and research. 6.2 SCRIPTING PROCESS: The questionnaire script included rigorous controls to prevent and flag illogical answers. There is a functionality that allows monitoring of specific key questions. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 151 The tool was pre-tested and modified prior to scripting on the system and after scripting to ensure that all quality assurance rules were applied correctly. An export of dummy data was done prior to commencement of data collection for assurance that the data is compatible with the needed format. This allowed for the submission of quantitative data in SPSS and Excel formats. INTERVIEWERS AND TRAINING The interviewing team consisted of 9 enumerators and 1 supervisor. All team members underwent a structured and thorough two-day training as shown in table 8 below. TABLE 8. TRAINING SCHEDULE Session Date Attendance Location Questionnaire Training 3 June 9 enumerators 1 supervisor Mindset offices Tablet Training 4 June 9 enumerators 1 supervisor Mindset offices QUALITY CONTROL MEASURES Mindset employed research best practice in the execution of this research project. The below are the quality assurance and control measures that were used throughout this task. TABLE 9. QUALITY CONTROL SUMMARY Procedure Description Percentage Visual consistency check Questionnaires were reviewed visually for accuracy, errors in coding, logical issues, pattern response, and missing data. 100% of data Data cleaning The data processing experts performed several levels of data cleaning for cohesion, logic, and completeness of data. 100% of data CODING AND DATA ENTRY 9.1 CODING Coding of open-ended questions started on the second day of interviews. The data processing team was responsible for entering the codes daily. Senior project members reviewed and approved the codes. Moreover, during data cleaning, the data processing officer reviewed all the entered codes to ensure they are valid for each question 9.2 DATA ENTRY AND PROCESSING Data cleaning was done on an on-going basis starting the second day of data collection. 152 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV 1. Common errors are collected by the data processing officer and relayed on a daily basis to the research team. 2. Data errors are divided into three types: • Logic errors. Those are referred to the call back team for collection and verification. • Data entry errors. Those are referred to the data cleaning team for correct entry. • Open ended errors. Those are spelling mistakes which are also referred to the data cleaning team for correct entry. • Other checks that are done: o Single response: contains 1 response o Text response: contains words only o Numeric values: contains numbers only o Exclusive answers: contains 1 response only o Skips: ensure skip patterns are followed o The option “Other” in open-ended questions: response is entered if “other” is selected and response is different from original options / codes 3. After all errors were addressed and modified into the system, a final cleaning of the full dataset was done. PROJECT SCHEDULE The project started on May 7th when Mindset received the English questionnaire from MSI/MESP. Mindset fulfilled the requested sample as per original plan on the 19th of July, including the ongoing data cleaning. 7 May - 7 June Preparation •Review & translation of questionnaire •Recruitment & logistical planning •Scripting of the form •Internal testing •Training •Pilot •Amendment of form 10 June - 9 July Data Collection •Conducted data collection activities •Ongoing data cleaning •Ongoing coding •Ongoing back￾checks and verifications 10 July - 19 July Post Fieldwork •Ongoing data cleaning •Ongoing coding •Ongoing back￾checks and verifications •Delivery of cleaned dataset USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 153 TABLE 10. KEY DATES Task Date Questionnaire reviewed and translated 15 May Questionnaire training 3 June Tablet training 4 June Pilot 5-6 June Data collection 10 June – 9 July Data entry, cleaning, and processing 11 June – 19 July STUDY CHALLENGES Mindset encountered a few challenges throughout the study as listed below: • The sampling list contained issues with some of the phone numbers provided (no longer valid, wrong number, etc.). This reduced the number of available beneficiaries that Mindset was able to contact. • The study began during Ramadan, which effected the response rate and availability of the targeted respondents, as well as their willingness to participate. The study also overlapped with preparations for Eid, which also effected response rates and availability. • Several interviews were ended by respondents prior to the completion of the questionnaire (34 in total). This can largely be attributed to two factors: o The length of the questionnaire – respondents were not available to complete the questionnaire in full. o Respondents did not feel comfortable answering political questions. ▪ Mindset did not meet the target rate of 75 interviews for the Youth Leadership Academy due to insufficient sample lists. Mindset usually requests that the list of beneficiaries is at least three times the requested sample to ensure the quota is met. SURVEY METHODOLOGY APPENDIX: POST-RESEARCH DATA PROCESSING AND QUALITY CONTROL PROCEDURE CHECKLIST DATA QUALITY/CLEANING CHECKLIST SIGHT CHECKS 1. Do all SPSS variable labels and value labels in the dataset match the final questionnaire? Do they have the correct skip patterns? 154 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV 2. Does the numbering of the response options in the dataset match the numbering of the options in the final questionnaire (e.g., make sure no items were reverse-coded, etc.)? 3. Does the codebook adhere to the questionnaire? DATA CLEANING 1. Does the structure (multiple/single response) of all questions in the data match the structure in the codebook? 2. Are there any missing values that should not be missing? 3. Do any of the questions have filters that were not properly followed or administered? 4. Is there any extraneous data to remove? 5. Have missing values been recoded (e.g. applying a new code to a question: e.g. refused to answer)? 6. Have open coded questions been back-coded so that “other” responses are fit into properly categorized answers whenever data filters are not affected by these changes? PERFORM LOGIC CHECKS (MARGINAL/CROSSTABS) 1. Were filter questions or skip patterns properly executed (cross-tabulate variables to see if respondents were isolated properly using filters/skip)? o If minor errors found was there forward cleaning of data? (which may include removing extraneous data of later questions that have filters that were not properly followed or administered during the research) 2. Are questions that allow for multiple responses (such as first answer/second answer; multiple dichotomies) coded properly or in a way that makes sense? 3. Are there any outliers? CHECK PARA/META DATA 1. Are paradata and metadata variables specified in the technical specifications included in the data file? 2. Do sampling variables in the data file match the pre-survey sampling design? INTERVIEWER CHECKS 1. Are interviewer and supervisor workloads consistent with the contract/technical specifications for the project (e.g., number of interviews per interviewer, number of supervisors used)? 2. Is the daily distribution of interviews consistent with the contract and logically feasible for an interviewer (e.g., number of interviews per day)? 3. Do the dates and locations of the interviews match the stated dates and locations in the work plan? 4. Is the average time of interview reasonable given the questionnaire length? Can any excessively short or long interviews be explained satisfactorily? 5. Are there any overlapping interviews by the same interviewer on the same day? 6. Are there any instances of interviewer "teleportation" (e.g., interviewer moves across the country in a single day, in a way that is impossible)? USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 155 7. Are there any interviewers who had the same responses for particular questions across all of his/her interviews? 8. Are there interviewers with high item non-responses and missing values in the data? 9. Are there any interviews/cases that have the same answers across a series of questions? DUPLICATES 1. Does the dataset have any duplicate cases (e.g., duplicate IDs)? ANNEX E. SURVEY KEY FINDINGS REPORT CEPPS Phone Survey Findings To understand how the beneficiaries from NDI’s Ana Usharek and Usharek+ programs and IRI’s Youth Leadership Academy program interact with and understand different avenues for civic engagement, the evaluation team conducted a phone survey (N = 463) of beneficiaries from all years of each program. As noted in the methodology section of this report, the survey was not meant to examine causality between attitudinal and behavioral outcomes and the CEPPS activities. It was designed to be used as a contextual tool to understand the attitudes, perceptions and reported behaviors of this cohort of beneficiaries around civic life in Jordan. As such the survey asked a series of questions aimed at gauging respondents reported attitudes towards key political/civic themes covered in NDI’s and IRI’s programs, perceptions of the effectiveness of different civic/political institutions in Jordan and a series of questions aimed at gauging participants perceptions of their own agency in engaging in key avenues for civic participation (voting, campaigning, community activism) that have been focused on heavily by NDI and IRIs program. Many of the survey questions were taken from the 2017 and 2016 IRI national surveys to allow for comparisons between survey respondents and national averages. Additionally, the survey asked a limited number of questions on the effectiveness of the NDI or IRI programs the respondents participated in. The table below outlines the all the survey modules from the CEPPS phone survey. TABLE 3. CEPPS SURVEY MODULES In some findings, particularly around reported engagement there was a slight variance amongst the three beneficiary groups, however, most of that variance can be explained by the differences in participant background and participant recruitment in each of the three programs7. The Youth Leadership Academy recruited already engaged activists and political actors, Usharek+ recruiting the best Ana Usharek graduates and Ana Usharek had no recruitment criteria and had beneficiaries with varying levels of exposure and interest in politics. Respondents from IRI’s YLA program, expectantly, showed generally the highest levels of knowledge, agency-attitudes and engagement, while respondents from Ana Usharek 7 Given the small sample universe and nature of the phone survey (delivering a lower response rate than a face to face survey) the margins of error for each beneficiary group were relatively large. Ana Usharek and Usharek + have a +/- 7% margin of error, while YLA has a +/- 11% margin of error. 156 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV generally showed the lowest levels. There was no difference between YLA, Ana Usharek and Usharek+ beneficiaries in regard to gender bias. GENERAL PERCEPTIONS AND ATTITUDES Significantly more CEPPS survey respondents reported government corruption as the biggest challenge facing Jordan than the respondents from the IRI 2017 national poll. Government corruption ranked in the top three challenges facing Jordan amongst the CEPPS survey respondents, with 23% of respondents ranking it as the top challenge. In comparison government corruption was not near the top three challenges reported in the 2017 IRI poll, with only 6% of respondents ranking it as the top challenge. Amongst the CEPPS respondents far more men (30%) than women (16%) noted corruption as a top challenge facing Jordan. Unemployment was cited a top challenge to Jordan in both the CEPPS poll and the IRI 2017 national poll. In both the IRI and CEPPS poll unemployment was ranked as one of the top three challenges (23% in the CEPPS poll vs 25% in the IRI poll). Amongst the CEPPS respondent’s unemployment was the most frequently cited top challenge for women (28%) and the third most frequently cited challenge for men (18%). Very few CEPPS Survey respondents reported poverty or the rising costs of living as a top challenge facing Jordan compared to respondents in the IRI 2017 national poll. Amongst the CEPPS survey respondents less than 5% cited poverty or the rising cost as the top challenge facing Jordan. Comparatively 17% of IRI respondents cited poverty and 16% cited rising costs as the top challenge facing Jordan, placing these two issues as the second and third most frequently cited challenge in the IRI poll. Important Notes for Interpreting the Data: The survey universe comprised of a population that was younger, more highly educated and more economically stable than the national average. Therefore, the respondents have more privileged backgrounds then average Jordanian youth. Additionally, most of the respondents self-selected to participate in the programs targeted by the survey and two out of the three programs had screening and application processes which recruited only student with history of civic engagement and activism. Thus, this data should not be taken as representative of the Jordan population or Jordanian youth population. USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 157 FIGURE 1. TO WHAT DEGREE DO YOU CARE ABOUT POLITICAL REFORM IN JORDAN? CEPPS survey respondents reported a significantly higher interest in political reform than the national average. 93% of survey respondents indicated that the cared about political reform to moderate or large degree. In comparison 59% of respondents from the 2017 IRI poll indicated the same All YLA beneficiaries that responded to the question reported that the cared about political reform to some degree. In both Ana Usharek and Usharek+ only 2% of respondents reported not caring at all about political reform. CEPPS survey respondents were more likely to believe that citizens should engage in their community development than the national average. Nearly all CEPPS survey respondents (99%) stated that they believed citizens should engage in the development of their communities, compared to 69% of IRI 2017 national poll respondents. No CEPPS survey respondents reported that disagreed with citizen involvement in community development whereas 15% of IRI survey respondents reported such a disagreement. 75% 76% 90% 16% 18% 8% 4% 3% 2% 2% 2% 1% 2% Ana Usharek (n =204) Usharek+ (n=208) YLA (n=50) (Shown, percent surveyed by program) To a large degree To a moderate degree To a little degree Not at all Refused vol 158 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV AGENCY ATTITUDES FIGURE 2. TO WHAT DEGREE DO YOU FEEL YOU HAVE A SAY IN GOVERNMENTAL DECISION-MAKING ON ISSUES THAT DIRECTLY AFFECT YOU? Half of CEPPS survey respondents (52%) stated that they felt they had some degree of say in government decision making vs. 35% of IRI survey respondents. As shown in Figure 2, there was little variation amongst the CEPPS beneficiary groups, however, respondents from Ana Usharek more frequently reported feeling like that had little to no say in government decision making. Amongst the CEPPS survey respondents more women reported feeling like they had a voice in government than men with 59% of women surveyed felt that they had a moderate to large degree of say in governmental decision making vs. 47% of men surveyed and 17% of women surveyed felt that they had no say in governmental decision making vs. 27% of men surveyed. The avenue in which CEPPS survey respondents indicated the MOST confidence in their ability to engage was advocacy. Running for Office was the avenue survey respondents indicated the least confidence in their ability to engage. The survey asked a battery of questions on respondents’ perceptions on their ability to engage on three of the key avenues that are actively promoted by USAID and CEPPS: Campaigning, Running for office and community advocacy and activism8. 8 The fourth major avenue – voting – is explored through questions on reported voting behavior and perceptions of the electoral process. These findings can be found in the next sub-section. 22% 23% 14% 24% 36% 44% 26% 20% 18% 25% 20% 22% 2% 2% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Ana Usharek n=204) Usharek+ (n=208) YLA (n=50) (Shown, percent surveyed by program) To a large degree To a moderate degree To a little degree Not at all Don't Know USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 159 FIGURE 3. TO WHAT DEGREE DO YOU HAVE THE ABILITY TO ACTIVELY ENGAGE IN THE POLITICAL PROCESS THROUGH... The responses were similar across all three beneficiary groups, however, slightly Ana Usharek beneficiaries reported a large or moderate degree of ability to engage in each of the three avenues. There was no real difference between men and women in their senses of ability to be civically engaged through running for office, volunteering or working for a political campaign and community activism and advocacy Reported levels of support for other avenues of civic engagement varied between “formalized” avenues (organized boycott, lobbying, organized and sanctioned protests) and “informal” avenues (illegal protests/strikes). Respondents reported more support for formal avenues and little to no support for informal avenues. FIGURE 4. SUPPORT FOR POLITICAL ACTIONS REPORTED ENGAGEMENT CEPPS survey respondents reported higher levels of civic engagement than the national average, however, in some key avenues engagement is still low. 44% of CEPPS survey respondents reported 36% 44% 40% 6% 5% 47% 37% 35% 26% 21% 16% 18% 25% 68% 72% 1% 1% Boycott Contact Government… Join a Lawful/ Peaceful… Join an unofficial strike Join an unofficial Protest (shown, percent surveyed n=462) Support in most circumstances Support, only in certain circumstances Would never support under any circumstances Refused vol 55% 49% 38% 32% 31% 31% 7% 11% 14% 5% 8% 14% 2% 1% 3% Advocacy Working/Vol. for a Political Campaign Running for Office (shown: percent surveyed, n=452) To a Large Degree To a Moderate Degree To a Little Degree Not at All Ref/DK 160 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV never: running for office, volunteering for a political campaign, starting or joining a CSO/NGO, or joining/starting a political party. Furthermore 55% of surveyed respondents had not attended a public meeting with officials at any level of government in the last year Figure five below shows the percent of respondents who reported engaging in each of the key avenues of civic participation explored in the survey. In figure five the avenues are grouped thematically by color: • Electoral Avenues: Voting, Campaigning, Running for Office, Starting/Joining a Political Party (blue) • Advocacy and Community Activism Avenues: Community Based Advocacy Campaigns, Starting/Joining a CBO or NGO (Red) • Citizen Engagement Avenues: Attending Public Meetings with Government Officials, Contacting or Visiting Government Officials to Express a Concern (grey) FIGURE 5. IN THE LAST YEAR HAVE YOU ... The survey data showed a large gender gap in reported political participation and engagement. The gap is particularly strong in the “active citizen avenues” and “electoral avenues” for participation. Figure seven shows the percentages of men and women that reported having engaged in each of the key avenues of civic participation explored in the survey. 64% 36% 8% 4% 80% 44% 45% 26% Voted in the 2017 Elections Worked/Vol with a campagin Joined/started a political party Run for office Volunteered to solove a community problem through advocacy Joined/Started CBO or NGO Attended a public meeting with gov. officials Visited/contacted gov. officials to express a concern (Shown, percent surveyed answering yes, n=463) USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 161 FIGURE 7. IN THE LAST YEAR HAVE YOU... 51% of women (vs 37% of men) surveyed reported never: running for office, volunteering for a political campaign, starting or joining a CSO/NGO, or joining/starting a political party. Advocacy and Community Activism Avenues where the most frequently reported avenues of engagement. 80% of survey respondents reported that they have worked to address a need facing their community in the last year. However, it is important to remember when looking at this number, that both Usharek+ and YLA had active community campaigning and advocacy as a core part of their curriculum (in order to give beneficiaries real work experience in advocacy). Additionally, 37% of CEPPS survey respondents reported that they had joined a CBO or NGO in the past year and 72% reported that they were likely to join one. 50% of survey respondents reported that they were likely or somewhat likely to start their own CBO or NGO in the future. Community advocacy was the avenue of civic engagement that had the highest reported female participation and second lowest gender gap in participation. 76% of surveyed women reported volunteering their time address a need in their community compared to 83% of men surveyed. Voting followed community advocacy as the second most common individual avenue for reported civic engagement. CEPPS survey respondents reported voting at a much higher rate than the national average. 64% of the survey respondents voted in the 2017 decentralization election, compared to 45% in the 2017 IRI poll who stated that they intended to vote in the 2017 elections. 66% of the CEPPS survey respondents reported that they voted in the 2016 election. 38% of respondents in the 2016 IRI poll stated they intended to vote in the 2016 elections. There was no real difference amongst beneficiary groups in reported voting behavior. 73% 43% 10% 5% 83% 52% 52% 32% 56% 29% 4% 4% 76% 33% 35% 18% (shown percent surveyed by gender) Men (n=245) Women (n=207) 162 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Amongst CEPPS survey respondents far fewer women than men reported voting in either the 2016 or 2017 Election. 59% of women surveyed reported voting in the 2016 parliamentary elections vs. 73% of men. 56% of women surveyed reported voting in the 2017 decentralization and municipal elections vs. 70% of men. Amongst the CEPPS survey respondents who did not vote, far more women than men cited a lack of interest as a main factor in deciding not to vote. 25% of women who did not vote in the 2016 elections reported a lack of interest as the primary reason for not voting vs 5% of men who did not vote. 21% of women who did not vote in the 2017 elections reported lack of interest as the primary reason for not voting vs 6% of men who did not vote. Running for office and participating in political parties were the two least common reported avenues for civic engagement. Given the age of the respondents, the high barriers to entry to running for office an and the general lack of trust of political parties in Jordan these findings are not surprising. However, it is important to note that while very few respondents reported running for office in the past, 47% of survey respondents stated they were likely to run for office in the future. FIGURE 8. HOW LIKELY ARE YOU TO RUN FOR OFFICE IN THE FUTURE? Furthermore 47% of survey respondents stated that they would be likely to join or start a political campaign in the future. 40 36 22 60 64 78 Ana Usharek (n= 204) Usharek+ (n=208) YLA (n=50) Did You Vote in the 2017 Decentralization Elections? (Shown, percent surveyed by program) No Yes 33 34 30 67 66 70 Ana Usharek (n= 204) Usharek+ (n=208) YLA (n=50) Did You Vote in the 2016 Parliamentary Elections? (Shown, percent surveyed by program) No Yes 12% 16% 40% 32% 41% 32% 10% 8% 10% 52% 38% 26% Ana Usharek (n=204) Usharek+ (n=208) YLA (n=50) (Shown, percent surveyed by program) Very Likley Somewhat Likley Somewhat Unlikley Very Unlikey USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 163 Nearly equal percentages of men and women surveyed reported running for office. However only 40% of women surveyed vs. 57% of men surveyed stated that they were likely to run for office in the future. More men than women reported engaging with politics and civic affairs informally, through conversations with their peers and family. 17% of women surveyed reported discussing news and political events with their friends on a daily basis vs 45% of men surveyed. 32% of women surveyed reported never discussing news and political events with their friends vs only 15% of men surveyed. 14% of women surveyed reported discussing news and political events with the peers on a daily basis vs. 34% of men surveyed. 49% of women surveyed reported never discussing news and political events with their peers vs 27% of men surveyed PERCEPTIONS OF KEY CIVIC/POLITICAL INSTITUTIONS FIGURE 9: PERCENT REPORTING LITTLE TO NO CONFIDENCE IN THE ABILITY/RESPONSIVENESS IN THE FOLLOWING INSTITUTIONS CEPPS survey respondents had similar if not worse perceptions of key governmental institutions compared the respondents from the 2017 IRI national poll. The survey respondents showed a strong lack of confidence in parliament, municipal governments and political parties. There were large differences in the perceptions of political parties and the prime minister between the IRI data and the CEPPS survey data. Rather than being truly indicative of a difference between these two populations, these changes are more likely a result of the timing of the fielding of the CEPPS survey. The timing of the Survey may have biased the data around perceptions of certain governmental institutions. The CEPPS phone survey was conducted during the end and directly after the resolution of widespread protests over proposed changes to the income tax law in late June 2018. The protests were resolved after the sacking of Prime Minister xxx and the instatement of Dr xxx as Prime Minister. In an address to the nation shortly after his appointment the Prime Minster stated that he would take the views of the protestors into consideration and hold off on passing the new tax law. This was seen by many as a huge success for the protestors, which may account for the high favorability rating of the 14% 54% 69% 45% 29% 8% 16% 71% 47% 59% Judiciary Prime Minister Parliament Municipal Government Political Parties (Shown percent of populations surveyed) IRI 2017 Poll CEPPS Survey 164 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV Prime Minster in the CEPPS survey. In order to get an accurate reading of how this group feels generally about the prime minister this question should be re-fielded. However, this is an interesting insight into the interplay between what is happing on the ground politically in Jordan and the perceptions of key institutions amongst civic minded youth. While the CEPPS survey respondents reported relative high levels of confidence in the IEC there is still a sizable minatory that believe elections in Jordan are generally unfair. 61% of survey respondents reported believing that the IEC was either somewhat or very effective in carrying out free and fair elections in Jordan. XX% of survey respondents stated that they believed elections were generally unfair in Jordan. Additionally, 35% of survey respondents reported being unlikely to vote in the next elections. FIGURE 10. TO WHAT EXTENT DO YOU BELIEVE THAT ELECTIONS ARE GENERALLY FAIR IN JORDAN? PERCEPTIONS ON GENDER ROLES IN POLITICAL LIFE The CEPPS survey data shows that strong gender norms around the role of women persist even amongst a young, educated and privileged sub-set of the Jordanian population. Survey Respondents reported similar attitudes towards gender equality as the general population in IRI poll: 25% CEPPS respondents reported disagreeing/strongly disagreeing with women's equal representation vs. 28% in the 2017 IRI poll) 9 7 12 50 47 51 37 43 37 4 3 Ana Usharek (n= 204) Usharek+ (n=208) YLA (n=50) (Shown: Percent surveyed by program) Elections in Jordan are generally very fair Elections in Jordan are generally somewhat fair Elections in Jordan are generally not fair Don't Know/Refused USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 165 FIGURE 11. “WOMEN SHOULD BE EQUALLY REPRESENTED IN POLITICAL DECISION MAKING IN JORDAN" AGREE OR DISAGREE ? Amongst the surveyed population, far more men than women did not believe that women should not be equally represented in political decision making. 16% of women surveyed disagreed with the statement “women should be equally represented in political decision making in Jordan” vs. 32% of men surveyed. While 84% of women surveyed agreed with the statement “women should be equally represented in political decision making in Jordan” vs. 67% of men The majority of the survey respondents believed that men and women were equally effective as parliamentarians, however a sizeable minority of respondents favored one gender over the other. Amongst the surveyed population, just over half of men and women believe that male and female members of parliament are equally effective (55% of women surveyed and 58% of men surveyed), which from a gender perspective is the most desired response. However, a large minority of men and women reported feeling that men were more effective parliamentarians than women and a significant minority of women reported feeling that women were more effective parliamentarians than men 21% of women surveyed and 25% of men surveyed felt that men were more effective members of parliament than women. 18% of women surveyed felt that women were more effective parliamentarians than men vs. 13% of men surveyed. SURVEY CONCLUSIONS While very privileged, the CEPPS survey population is not immune to the pressures of the contracting economy, namely unemployment. However, the CEPPS survey population had a different experience of economic challenges of poverty and rising prices as compared to the rest of the Jordan population. The psycho/social impact of unemployment and perceptions economic challenges around is an important contextual indicator to monitor amongst this peer group. A number of recent studies have been published linking the economic disillusionment of the middle to upper middle-class youth as key factors in civil unrest. The studies suggest that while economic pressures of rising prices and poverty are not felt by the middle class, political grievances turn into political apathy. However, when those economic pressures are 44.3% 39.6% 57.1% 31.3% 31.7% 22.4% 17.9% 17.8% 10.2% 6.5% 9.9% 6.1% 1% 4% Ana Usharek( n=201) Usharek+ (n=202) YLA (n=49) (Shown, percent surveyed by program) Strongly Agree Agree Disagree Strongly Disagree Don't Know 166 | CEPPS PROGRAM EVALUATION REPORT USAID.GOV felt but by the middle-class youth and they are more likely to resort alternative means of redress for their political grievances. Avenues for engagement that have easier entry points or easier access have higher levels of engagement. The survey data showed that respondents more frequently reported be actively engaged through voting and community activism. These are two avenues that have the lowest barriers to entry (as opposed to something like joining a political party or running for office). This, in and of itself is not very surprising or groundbreaking, however, tracking to see whether or not those that engage via these types of avenues actually end up participating more in the more complex and harder to access avenues could give great insights into future political participation programming. Increasing feelings of agency, while necessary, is not sufficient to see large scale increases in engagement. Despite reporting extremely high levels of agency to engage in key civic and political issues, few respondents reported engaging in activities outside of voting and community advocacy. This trend is seen even more strongly among women beneficiaries. This suggests that agency alone is not enough to get youth to actively engage in politics and civic life. A big barrier to engagement is the lack of trust in public institutions. The survey data clearly shows that there is a low level of trust and confidence in key public institutions. In order to want to engage in political and civic life, youth must feel like they have competent, willing and at some level cooperative partners in public institutions. Attitudes towards public institutions are shaped in large part by on the ground realities. There are political and structural realities concerning transparency, corruption, and effectiveness of public institutions that shape how the Jordanian public feel about said institutions. The CEPPS survey cohort is not immune to these experiences and it shows in their reported attitudes towards institutions like parliament, the municipal government and the electoral process. The survey also showed how fast changing ground realities can affect the perceptions of civic minded youth (like the CEPPS beneficiaries). Donors and implementing partners must keep ground realities and histories in mind when setting their expectations of how their programs will affect perceptions and participation and must reassess their assumptions after major political changes. As with engagement generally, raising senses of personal agency is not enough increase women’s civic participation and political empowerment. Surveyed women held similar, if not more optimistic perceptions, as men on key thematic issues and their own agency to engage in civic life. However, significantly fewer women reported participating in civic life (including voting, attending public meetings, meeting with officials, etc.) than men (23% gap in attending public meetings; 14% gap in engaging with public officials; 14% gap voting -2016 and 2017). The survey data indicates persistent biases around the role of women in politics, rather than a lack of agency or knowledge, may be key barriers keeping women from participating at the same level as their male peers. The survey data showed that on issues of gender equality and empowerment there were sizable minorities of both men and women that felt that women and men were not equally capable of engaging in politics nor should be equally represented in political decision making. The entrenched sense of gender roles and politics sitting firmly in the male sphere may also explain why far more women than men reported being uninterested in political activities such as voting and why far fewer of them discuss politics with their family and peers. To affect women’s political participation and USAID.GOV CEPPS PROGRAM EVALUATION REPORT | 167 empowerment programs must be also directed at tackling the structural and cultural barriers that are keeping women from full participation.