USAID/KOSOVO MID-TERM PERFORMANCE EVALUATION OF THE STRENGTHENING ELECTIONS ADMINISTRATION ACTIVITY (SEAA) FINAL EVALUATION REPORT JULY 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International (MSI), a Tetra Tech Company. MID-TERM PERFORMANCE EVALUATION OF THE STRENGTHENING ELECTIONS ADMINISTRATION ACTIVITY (SEAA) FINAL EVALUATION REPORT July 30, 2018 IDIQ No. AID-167-I-17-00002 Order No: 72016718F00007 Prepared for United States Agency for International Development/Kosovo Ismail Qemali (Arberia), House 1 Pristina – Kosovo 1000 DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV CONTENTS ACRONYMS II EXECUTIVE SUMMARY 1 ANALYSIS 3 Introduction and Programming Context 3 Evaluation Purpose and Evaluation Questions Error! Bookmark not defined.3 Key Questions 5 Evaluation Approach and Limitations 5 FINDINGS/CONCLUSIONS 7 Evaluation Question 1 9 Evaluation Question 2 14 Evaluation Question 3 19 RECOMMENDATIONS 21 USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | ii ACRONYMS BIRN Balkan Investigative Reporting Network BRIDGE Building Resources in Democracy, Governance, and Elections CDCS Country Development Cooperation Strategy CEC Central Elections Commission CECS CEC Secretariat CEPPS Consortium for Elections and Political Process Strengthening CRC Counting Results Center CSO Civil Society Organization D4D Democracy for Development ECAP Elections Complaints and Appeals Process EQ Evaluation Question ESP Election Support in Kosovo EOM Election Observation Mission ET Evaluation Team EU European Union FVL Final Voters List IFES International Foundation for Electoral Systems IIMVEP Increasing Inclusion and Mitigating Violence in Electoral Processes IT Information Technology KDI Kosovo Democratic Institute KI Key Informant KII Key Informant Interview LGE Law on General Elections LLE Law on Local Elections MEC Municipal Election Commission MEO Municipal Election Officer MSI Management Systems International NDI National Democratic Institute OPPRC Office of Political Party Registration and Certification OSCE Organization for Security and Cooperation in Europe PSC Polling Station Commission PSA Public Service Announcement PwD Persons with Disabilities RRF Results and Reconciliation Form SEAA Strengthening Elections Administration Activity SEAK Strengthening Election Administration in Kosovo TA Technical Assistance 1 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY Free and fair elections are generally viewed as the most prominent feature of democracies. The Organization for Security and Cooperation in Europe (OSCE) administered elections in Kosovo until independence in 2008, including control over the Central Election Commission (CEC) and all operations. USAID/Kosovo has been the primary funder for strengthening elections administration in Kosovo since independence, with a series of five projects that worked with the full panoply of election administration bodies: the CEC, the CEC Secretariat, the Elections Complaints and Appeals Panel, the Counting and Results Center, and the Office of Political Party Registration and Certification. While OSCE was viewed as the only credible actor for elections administration in the Serb-majority northern municipalities for the first elections following the 2013 Brussels Agreement, the CEC administered the elections there for both 2017 national and municipal elections. USAID/Kosovo needs a performance evaluation of its election assistance since 2009 and a midterm performance evaluation of its current Strengthening Elections Administration Activity (SEAA) to gather key findings, conclusions, and recommendations to ensure that USAID resources are utilized for maximum development results. The evaluation questions address a number of specific programming areas, but can be summarized as: 1. To what extent have the efficiency and administration of elections improved since 2009? 2. To what extent are those improvements sustainable? 3. What challenges and opportunities remain for the current International Foundation for Electoral Systems (IFES) activity? The evaluation implemented a mixed-methods design for data collection based on review of a significant volume of documents and more than 50 key informants. The evaluation team used parallel analysis to derive findings for each data source and then drew conclusions across a set of data sources for an evaluation question. As with any evaluation, limitations and biases exist in the forms of faded memories, carefully couched answers, and keenly interested interviewees; the design attempted to mitigate these potential problems through careful use of multiple sources of data and experiential questions. The evaluation team concluded overall that USAID-funded IFES programming has contributed to the efficiency, functioning, transparency, and sustainability of Kosovo elections administration, but challenges outside of IFES programming remain. The evaluation team concluded that IFES had contributed significantly to the strengthening of elections administration in Kosovo through a diverse set of interventions since 2009. The set of electoral institutions have adopted and adapted new practices that improved the efficiency of elections administration. The Counting and Results Center has become notably more efficient, professional, and transparent over the last 10 years. The analysis particularly noted strengths in the use of strategic planning, technology, training, and access for persons with disabilities, but weaknesses in public information campaigns. Other areas of concern that could be only partially addressed by the election administration bodies and IFES include the voters list, gender equality among poll workers, and research. The evaluation team concluded that the CEC, the CEC Secretariat, and the Elections Complaints and Appeals Panel are much more operational and sustainable, although some weaknesses remain, and the Elections Complaints and Appeals Panel faces budget constraints. Most of the technology platforms are fully operational and could be taken over by CEC Secretariat staff. The Counting and Results Center is highly operational but is vulnerable to the potential departure of its sole staff member, the coordinator. In addition, its main server and software are 10 years old and should be updated completely. The Office USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 2 for Political Party Registration and Certification is fully operational but, like the Counting and Results Center, is vulnerable to the imminent departure of its division head. The evaluation team concluded that the key challenge to implementing elections administration assistance has been the interruption of capacity-building activities due to snap elections for the Assembly, which forced IFES to refocus on intensive technical assistance in critical areas to support election administration as needed. Partially as a result of multiple instances of this shifting, the election administration bodies — particularly the CEC Secretariat — have an emotional dependence on IFES and have not grappled on their own with internal process obstacles; the evaluation team characterizes these as missed opportunities to use non-election periods more effectively. The primary opportunity stems from experience gained conducting back-to-back Assembly and municipal elections in 2017, which brought to light additional ways to enhance the roles of municipal election officers (MEOs) and civil society organizations (CSOs) in strengthening the electoral process. The evaluation team asserts that the CEC, the CEC Secretariat, the Counting and Results Center, the Office of Political Party Registration and Certification, and the Elections Complaints and Appeals Panel are capable of administering elections without external technical assistance, despite some specific areas of weakness or vulnerability. Specifically, the evaluation team recommends that IFES continue to work with the main bodies to make technology products more operational, facilitate a second round of strategic planning, overhaul public information functions, emphasize voter list improvements, and continue pushing for poll worker gender equality. The evaluation team also recommends five areas that go beyond the scope of the current activity: advise the Counting and Results Center on new systems; expand the role of municipal election officers; mentor CECS to partner more with CSOs on public information campaigns and voter outreach; coordinate research across all elections administration bodies; and seek professional development opportunities outside the realm of elections administration. 3 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV ANALYSIS INTRODUCTION AND PROGRAMMING CONTEXT Free and fair elections are generally viewed as the most prominent feature of democracies. The Organization for Security and Cooperation in Europe (OSCE) administered elections in Kosovo until independence in 2008, including control over the Central Election Commission (CEC) and all operations. After independence, the nascent Kosovo state took over the CEC and began working with the international community and technical assistance organizations to administer the municipal elections in late 2009, followed in December 2010 by the first national elections after independence; both were viewed as flawed exercises that failed to increase public confidence in the process. The years between elections were taken up by extensive capacity building of the CEC, the CEC Secretariat (CECS), the Elections Complaints and Appeals Panel (ECAP), the Counting and Results Center (CRC), and the Office of Political Party Registration and Certification (OPPRC). The Brussels Agreement in 2013 allowed Kosovo to establish authority over the four Serb-majority northern municipalities in time for the scheduled November 2013 municipal elections. OSCE was viewed as the only credible actor for administering those elections, as they were again for the snap national elections in June 2014. As with the previous period between elections, technical assistance was provided to the CEC, CECS, ECAP, and CRC to build their capacity to administer elections independently. The scheduled municipal elections in October 2017 were preceded by snap national elections in June 2017; for both, the CEC administered elections in the northern municipalities. USAID/Kosovo has focused substantial efforts on strengthening elections administration in Kosovo since before independence. The Strengthening Elections Administration in Kosovo (SEAK) project was the first post-independence effort, concentrated on basic capacity and infrastructure for administering elections. Following this was SEAK II, which was directed at capacity-building and electoral reform, and the Elections Support Program (ESP), which expanded capacity building to other state actors and grappled more fully with the integration of the Kosovo Serb northern municipalities; the Increasing Inclusion and Mitigating Violence in Electoral Processes (IIMVEP) activity was added to ESP. The International Foundation for Electoral Systems (IFES) implemented all of these projects. The Strengthening Elections Administration Activity (SEAA) is ongoing, also implemented by IFES, and is intended to assist with another substantial step for Kosovo toward deep capacity for planning and administering elections. As Figure 1 shows, programming has progressed from fundamental processes and infrastructure to complex and elaborate capacity and public outreach efforts. EVALUATION PURPOSE AND EVALUATION QUESTIONS USAID/Kosovo needs a performance evaluation of its election assistance since 2009 and a midterm performance evaluation of its current SEAA. The purpose of the evaluation is to: 1) Provide key findings and conclusions regarding USAID’s election assistance since 2009; and 2) Provide key findings, conclusions, and recommendations for the current SEAA activity. The U.S. Government will also use the evaluation results to ensure utilization of resources in a manner that maximizes development results. USAID will use the evaluation to make adjustments to the implementation of the current SEAA activity and share lessons learned on elections programming since 2009 with other stakeholders. USAID/Kosovo will be the primary audience of the evaluation report and will use the report to make adjustments to its current implementation strategy. Secondary audiences include IFES, which will use learning about activity strengths and weaknesses to adjust its interventions, and other stakeholders interested in lessons learned. USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 4 FIGURE 1: ELECTIONS ADMINISTRATION PROGRAMMING, 2009–19 Electoral Reform, Election Law Revisions Voter List Accuracy Public Awareness & Outreach ECAC Support SEAK 2009-2010 CRC Support CEC/CECS Capacity Structural Rvw BRIDGE SEAK II 2010-2013 Electoral Reform ECAP Voter Registry Maintenance by CECS Public Awareness and Outreach Public Info Campaign Civic Education via CSOs CEC/CECS Capacity BRIDGE Regional Practices Organization/Personnel Northern Municipalities CRC Capacity/Support ESP 2013-2016 IIMVEP 2014-2016 Assembly Engagement Public Awareness and Outreach Public Info Campaigns PwD/Gender/Youth Campaigns Declaration of Gender Inclusion Political Violence CEC/CECS/CRC/ECAP Capacity Structural Review Strategic Planning Training/Materials Tabulation/Results Processes Transparency Voter List Maintenance Tech/Software/Apps Northern Municipalities SEAA 2016-2019 Underserved Populations PwD/Gender Inclusion Non-Majority Communities CEC/CECS/CRC/ECAP Capacity Strategic Planning Risk Assessment Training/Materials Public Info Campaigns Transparency Voter List Maintenance Tech/Software/Apps Northern Municipalities 5 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV KEY QUESTIONS This work plan addresses the following evaluation questions (EQs) proposed by USAID/Kosovo: EQ 1: To what extent have USAID-funded elections activities since 2009 contributed to strengthening electoral institutions and improving efficiency of processes related to better administration of elections and the appeals and complaints processes?1 ● To what extent have USAID’s elections activities contributed to new practices in the Central Elections Commission (CEC) (including the Secretariat and its Counting Results Center [CRC]) and the Elections Complaints and Appeals Process (ECAP) (for example, but not limited to: use of technology and IT tools; public information and voter education; inclusion of persons with disabilities (PwDs), minorities, women, and youth; development and implementation of strategies and work plans; improvement of voter lists, etc.). ● To what extent have elections activities contributed to the efficiency and functioning of the CRC? EQ 2: To what extent have USAID-funded elections activities since 2009 supported a sustainable and independent elections system? ● To what extent are the practices (discussed above) functional and operational? EQ 3: What are the challenges and opportunities in implementing USAID’s current election activity, SEEA? Based on the findings and conclusions emerging from analysis, the team will provide recommendations for the current SEAA activity. EVALUATION APPROACH AND LIMITATIONS This section outlines the methodology implemented for the midterm performance evaluation of SEAA. The nature of the evaluation questions drove the need for a more deliberate phasing of information-gathering and analysis: EQ 1 and EQ 2 have aspects of not just a standard performance evaluation focus on results, but also a process-tracing focus to identify how results interrelate over time. The full work plan is in Annex 2. EVALUATION APPROACH The evaluation design used a mixed-methods approach for collecting data from multiple types of data sources: a) review of project documents and other secondary materials and b) key informant interviews (KIIs). While this appears to be a standard performance evaluation design for data collection, the team utilized project documents and other secondary materials to create a timeline of progress that was traced and validated to the extent possible through findings based on the KIIs. ● Document and materials review. This data source includes implementer periodic reports, training materials, guidebooks, and the Activity Monitoring and Evaluation Plan, in addition to election observation mission final reports and other secondary reports or analyses that were relevant and available. 1 This evaluation addresses only the administration of elections, not the quality or nature of political competition in Serb-majority municipalities. USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 6 ● Key informant interviews. Types of key informants differ based on “causal distance”: direct participants in election administration strengthening activities, e.g., at the Central Elections Commission (CEC), the CEC Secretariat (CECS), the Elections Complaints and Appeals Panel (ECAP), the Counting Results Center (CRC), and municipal election officers (MEOs); people who are indirectly affected by new skills or processes generated by activities, e.g., political parties, civil society, and persons with disabilities (PwD), non-majority communities; USAID personnel involved in activity design and monitoring; IFES and partner staff involved in implementing the activities; and other donor and external experts. Each semi-structured KII differed slightly depending on the key informant’s role, the extent of the key informant’s involvement in SEAA and/or predecessor activities, and the key informant’s available time to be interviewed. Given the relatively small population set of potential key informants with direct knowledge of most of the interventions under election administration strengthening activities, the team selected key informants purposively instead of randomly; this also allowed the team to attempt to ensure that the small number of MEOs who are women and/or non-majority community members could be interviewed. The evaluation team made field visits to Prizren, Novo Brdo, Gjilan, Zvecan, and North Mitrovice. In total, the team interviewed 50 people over an 11-day period: 32% women, 68% men; 20% outside Prishtina, 80% within. The evaluation team (ET) used parallel analysis to examine the evidence from the materials review and KIIs. For each evaluation question, the team first analyzed relevant implementer materials and secondary sources to develop preliminary findings. Second, the team reviewed data to draw preliminary findings for each type of key informant before synthesizing across all types of key informants. For each of the first two steps, the findings were mapped to the timeline and highlights to develop preliminary findings about USAID-supported election administration strengthening activities. Finally, the team drew conclusions about activities across the span of time beginning in 2009 to answer each of the three EQs and their sub￾questions. Actionable recommendations were derived from the conclusions and a broader understanding of Kosovo’s political context. EVALUATION LIMITATIONS The essence of evaluation is comparison across time, space, and/or treatment. In the context of this midterm performance evaluation of the SEAA activity, however, the evaluation team was aware of barriers to comparison as it endeavored to draw valid conclusions. The ET foresaw several factors that indeed limited the ability of the evaluation team to address all of the evaluation questions. Fundamentally for this evaluation, many election administration strengthening activities would have been completed some years in the past. Due to the short timeframe for this evaluation, it was difficult for the team to determine the level of contribution for each specific area of intervention. Key biases that the team watched for include the following: ● Recall bias: The multiple activities related to election administration mean it is quite possible that key informants may not accurately remember details of particular intervention activities; activity beginning and end dates do not have much meaning outside USAID or the implementing partner. Of course, fading or faulty memories are typically problematic for events even two years in the past and become a much larger problem over longer periods of time. The team anticipated that recall bias would be a significant problem for collecting useful information on activities prior to SEAA, but IFES’s activities were distinct enough that key informants remembered important details even years later. 7 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV ● Response bias: Informants may give the team positive remarks about the activity because they would like to stay involved with the intervention in the future, and because they think that a negative evaluation could mean the end of activity-related opportunities. Given that USAID has been, far and away, the primary funder of technical assistance for elections administration, the team anticipated that response bias would be particularly pronounced among direct beneficiaries. The team had two strategies for mitigating this bias: 1) stressing for each informant that the team would maintain confidentiality and then explaining its independence from both USAID and the activity; and 2) using questions designed to elicit specific examples to help identify response bias, as general questions allow too much leeway for general responses. ● Selection bias: Using contacts provided by the implementers could mean that the team hears only from people who have positive experiences. This is often a problem for activities in which the main contacts typically have a longstanding relationship with the implementer, which may be the case for SEAA and its predecessor activities. The standard evaluation approach to mitigating this bias is to expand beyond implementer-provided contacts, collect multiple points of data from different types of key informants, and ask questions designed to elicit specific examples. Overall, the team attempted to combat all of these biases by clearly discussing the intentions of the evaluation with key informants and by using multiple sources of data for analysis. By combining information found in documents or interviews from multiple sources, any one piece of biased data will not skew the analysis. Finally, rather than asking questions that elicit answers based only on perceptions, the team asked questions about behavioral changes, requesting specific examples of knowledge use. FINDINGS/CONCLUSIONS The ET generated many findings and conclusions at the level of specific intervention areas or outcomes that are specified below. On the whole, the ET concludes that the USAID-funded IFES programming has contributed to the efficiency, functioning, transparency, and sustainability of Kosovo elections administration, but challenges outside of IFES programming remain. Figure 2 presents highlights of legal, administration, and organizational development achievements alongside a synthesis of election observation reports. USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 8 FIGURE 2: SYNTHESIS OF IFES ACTIVITIES, CAPACITY DEVELOPMENT, AND OBSERVATION REPORTS 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 SEAK SEAK II ESP IIMVEP SEAA L: LGE, LLE OD: Tabulation/Results Sys OD: CEC Org Chart A: Public Info Campaing, CRC, Mobile Polling (North Ibar) L: LGE (ECAP Expanded) A: OSCE Only Advisory OD: 5 Permanent Cmtes, Tabulation/ Results Sys Revised OD: CECS Re-Org Proposal, Structural Rvw A: CRC, Polling Stations (North Ibar) L: Law on Civil Svc A: Street Addressing Launched OD: IT Dept Established A: CSO-led Civic Ed Initiative OD: BRIDGE, Electoral Reform WG, CEC Re-Org A: CRC, Revised K-Vote A: CRC, PwD Access Campaign, GOTV OD: IT (CECS, OPPCR, ECAP), Strat Plan & work plans (CECS, ECAP), Trng Plan (CECS, ECAP), Trng Mgmt Grp, Municipal 2017 Public Info Plan, CEC Risk Ass'mt A: Pollworker Manuals A: Cascade Training, Assistance to IT OD: Transition to IT A: CRC, OSCE (North Ibar) A: CRC, Public Info Campaign, PwD Access, OSCE (North Ibar) OD: Election Access WG, Declaration Inclusion Women, Updated Structural Rvw, Trng Needs Ass'mt A: CRC, CEC (North Ibar) OD: Second Trng Plan (CECS) Nov/Dec: Municipal Dec: National Jun: National Nov/Dec: Municipal Jun: National Oct/Nov: Municipal CEC: ! ECAP: ! List: !!! C/OOK: ! Invalid: ! Secret: ! CEC: ! CRC:! ECAP: ! List: !!! C/OOK: ! Invalid: -- Secret: ! CEC:-- CRC:-- ECAP: -- List: !! C/OOK: ! Invalid: ! CEC:  List: ! C/OOK: ! Invalid: -- Secret: ! CEC:  List: ! C/OOK: ! Secret: ! CRC:  ECAP:  CRC:-- ECAP: -- Invalid: ! CEC:  List: ! C/OOK: ! Secret: ! CRC:  ECAP:  Invalid: ! Selected IFES Activities Selected Capacity Synthesis Ratings from Observation Mission Reports L = Legal; A = Administration; OD = Organizational Development CEC = CEC Operational Capacity; CRC = CRC Operational Capacity; ECAP = ECAP Operational Capacity List = Voters List; C/OOK = Conditional/Out-of-Kosovo; Invalid = Invalid Ballots Secret = Family Voting, Assisted Voting, Other 9 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV EVALUATION QUESTION 1 To what extent have USAID-funded elections activities since 2009 contributed to strengthening electoral institutions and improving efficiency of processes related to better administration of elections and the appeals and complaints processes? ● To what extent have USAID’s elections activities contributed to new practices in the Central Elections Commission (CEC) (including the Secretariat and its Counting Results Center [CRC]) and the Elections Complaints and Appeals Process (ECAP) (for example, but not limited to: use of technology and IT tools; public information and voter education; inclusion of persons with disabilities (PwD), minorities, women, and youth; development and implementation of strategies and work plans; improvement of voter lists, etc.)? ● To what extent have elections activities contributed to the efficiency and functioning of the CRC? The ET concluded that IFES has contributed significantly to the strengthening of elections administration in Kosovo through a diverse set of interventions. The set of electoral institutions have adopted and adapted new practices that improve the efficiency of elections administration. The CRC has become notably more efficient, professional, and transparent over the last 10 years. Per the EQ and sub-questions, the analysis of findings and conclusions is structured according to specific areas or outcomes. Strategic Planning. The ET concluded that CEC, CECS, and ECAP valued and benefited from the strategic planning processes facilitated by IFES and are now preparing for the next planning process. We found that all KIs queried about the strategic planning process said that this was another example of how IFES brought ideas to them; KIs from CEC and CECS reported that they were initially skeptical about the value of such a process, but very quickly learned to value the result. IFES facilitated the strategic planning process with a series of expert-led workshops from late 2014 through late 2015, but the CEC did not approve the strategic plan until the end of 2015. The ET found that the plan was subsequently used to develop annual work plans for CECS as a whole, for each department and division, and even for MEOs. ECAP also developed a strategic plan with IFES facilitation. The CECS strategic plan runs through the end of 2018, and ECAP’s through the end of 2019. KIIs from CEC, CECS, and ECAP asserted that they want to develop follow-on strategic plans, but do not feel confident about doing it on their own; they said they would like IFES to participate in at least an advisory role to provide ideas and comparative knowledge. Information Technology (IT). The evaluation team concluded that multiple new IT platforms to enhance and track performance (as enumerated below) would not have occurred without IFES’s comparative expertise to develop and implement these software modules. All key informants (KIs) from election administration bodies valued the innovative ideas and comparative models that IFES brought to them to improve the efficiency, functioning, and transparency of their operations. While the software development funding came from the IFES project, these platforms are now in use and sustainable due to knowledge transfer to the CEC’s IT staff, who are capable of maintaining these platforms. The ET found that CECS IT staff and the IFES IT staff have a close and collaborative working relationship that allows for productive mentoring and hands-on engagement. The CECS IT staff is small and its budget does not allow for the rapid embrace of new ideas, such as the ones IFES brought to CECS based on its comparative expertise. USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 10 The IFES-developed platforms include an online portal for citizens to obtain general election information, change their polling station, or check the disability access status of any polling station. This portal was fully available for the 2017 elections. Another IT product is the Election Staff Management System (ESMS), which is a database of all permanent and temporary workers. While this is fully functional, only the back￾end database side is in use, not the front-end recruiting side.2 The Electoral Violence Monitoring System (EVMS), adapted from the Ushahidi platform for reporting and mapping of electoral violence, was put in use for the first time in the 2017 elections. For ECAP, IFES developed the Complaints and Appeals Monitoring System (CAMS) for implementation in 2016, which tracks the status of each case in real time and on ECAP’s website, from the time it was entered into the system to adjudication, including details on status, party appeals, fines, etc. As detailed elsewhere, IFES worked with the CRC to revise the iVote system developed by a Macedonian vendor, creating the K-Vote application that connects tabulation and results at the polling center level to the central processing system. Training. The ET concluded that IFES helped build training capacity over the years, to the point that CECS is now capable of producing and implementing its own training plan. Over multiple years, IFES has developed training content and provided expert trainers and supported training workshops for many aspects of CECS’ work. The ET found the CEC has now built institutional capacity in training. One IFES staff member said, “We’re sitting beside them but they [CEC and ECAP] are doing the training now; it is a sustainable operation.” The ET learned that in early preparation for the 2017 local elections, IFES collaborated with the CEC to implement the CECS plan to build internal capacity for training by changing the poll worker training approach from centralized training to a train-the-trainer model, with an IFES contract training expert delivering a five-day training program to all 38 MEOs. A second five days of intensive training was delivered the following week to 10 MEOs identified to form a Training Management Group. After receiving training, each MEO subsequently trained their municipality’s trainers, many of whom are experienced trainers from previous elections. The new format involved training sessions with fewer attendees, incorporating an interactive, engaging, hands-on approach with simulation of Election Day activities and requirements, and participants taking a brief test at end of the training session. All KIIs found that the train-the-trainer decentralized approach was assessed positively and was fully supported by the CEC and especially by the MEOs, as they have more direct involvement. Counting/Results. The ET concluded that the CRC has developed capacity to perform the counting, tabulation, and materials storage for elections. IFES’s emphasis on the importance of transparency has now become institutionalized as CEC recognizes its value to their operations and reputation. The ET concluded that IFES helped build CRC capacity, including initially training up to 800 temporary employees, assisting with statistical report generation, analyzing data, and disseminating information to stakeholders. In addition, IFES worked closely with the CRC to develop training manuals for various areas of CRC operations. Over the past eight years, IFES has brought in international experts to assist and advise the CRC, which has benefited operationally from their presence, including as a shield against criticism. All KIs interviewed mentioned the CEC’s commitment to operating the CRC in a transparent manner. The ET found that IFES involvement has diminished over time, as 14 international IFES staff were embedded in the CRC operations for the 2010 election, but assistance dropped to two international experts during the 2017 elections as knowledge transfer occurred. The ET found that the volume of elections over the past few years has reaped benefits in standardizing and institutionalizing many CRC procedures. This almost continuous election cycle has also facilitated 2 The front end could be used for position applications by the general public, as well as reviewing and vetting by the human resources unit. The overwhelming majority of temporary positions for elections are poll workers, who must be nominated by political parties, and the parties have difficulties even submitting lists of names on deadline. 11 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV continuity of the returning 60 to 65 highly trained core team of temporary employees. The core team has now assumed responsibility for training and supervising the work of the approximately 800 CRC temporary employees. However, this constant election cycle has highlighted the critical need to update the CRC’s aging infrastructure of a 2009-era server and software, as well as K-Vote election night reporting software and equipment dating from the same 2009 time period. Public access to preliminary vote results on K-Vote failed for multiple hours during both the national and local 2017 elections, resulting in blackouts on the CEC’s website several times both on election nights and occasionally during the days following the 2017 elections. 3 This downtime negatively impacted the public’s ability to view preliminary results and tarnished the CEC’s reputation. KIs revealed that the CRC software vendor recently informed the CEC that the software is end-of-life and will no longer be supported. Public Information. The ET concluded that IFES assistance had been helpful, particularly in time-critical circumstances, but CECS does not appear to prioritize the management and planning of this crucial aspect of elections administration. IFES has been directly involved in all aspects of public information campaigns and civic education initiatives during the last 10 years. At times, IFES’s engagement has come at critical moments that arose out of CECS procurement or planning problems, but CECS has assumed more operational responsibility gradually since the 2014 national elections. As seen in project implementation reports, IFES’s local and international experts helped design public service announcements (PSAs) in multiple languages through multiple dissemination modes. During SEAA, CECS moved to more fully take over public information campaigns, with IFES providing advice on media strategy and messaging to the multiple internal committees reviewing portions of campaign planning. However, all KIs from non-majority communities complained that the media strategy and messaging in the 2017 elections were inappropriate for their communities. For example, translations from Albanian into Bosnian or Turkish were judged to be stilted or grammatically inaccurate, or the strategy used media outlets that could not reach the intended audiences. The ET found that all KIs outside CECS with direct knowledge of public information campaigns characterized the 2017 strategy as outdated and limited, perhaps even a reduced version of the 2014 strategy: message dissemination focused on traditional print and broadcast media, posters, and brochures, with only limited use of social media in the form of short Facebook posts4 by the media contractor. The media strategy was developed centrally with only minimal input from MEOs and none from CSOs. CECS has not conducted analyses of the 2017 public information campaigns. IFES also supported multiple civic education initiatives carried out by CSOs that have had positive effects on citizens and CECS. Under SEAK II, grants were made to six CSOs for the ‘Participate’ campaign in late 2012, which sought to increase knowledge of and confidence in elections among youth, women, non￾majority communities, and persons with disabilities. IFES’s provision of Building Resources in Democracy, Governance, and Elections (BRIDGE) training to CECS personnel in late 2012 expanded to include CSO staff to familiarize them with the complexity of elections administration; all KIs from participating CSOs praised the training and connected the gained knowledge to their subsequent domestic election observation missions. One activity under IIMVEP worked with the Kosovo Democratic Institute (KDI) to promote inclusion of persons with disabilities through PSAs and newspaper op-ed articles. A more extensive effort under IIMVEP aimed to increase youth engagement through social media, online games, youth-led journalism, and mock elections. 3 In the 2017 national election, a firewall system located within the Ministry of Public Administration blocked public access to the preliminary results website (K-Vote) for around 3 hours. In the 2017 local elections, there was a 2-3 hour delay of publicly available K-Vote generated results after the polls closed because of a software issue. 4 Serbian-speaking KIs asserted that the translations of Facebook posts were very poor. USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 12 All CECS KIs pointed to a mismatch between lengthy procurement processes and short electoral periods; in particular, procurement of public information campaign services for the June 2017 snap elections was complicated by a protest from losing bidders. IFES provided time-critical assistance for the June 2017 elections. Although the broader media and outreach strategies were developed between elections, it was not clear to the ET why a more detailed and operational media strategy and message design could not be accomplished before an election was called, and the team notes that the same procurement problem occurred four months later for the regularly scheduled municipal elections. An additional obstacle to timely and efficient public information campaigns is the cumbersome internal process involving multiple review committees. PwD Access. The ET concluded that IFES succeeded in raising awareness of PwD access issues, and has stimulated ongoing and independent actions to address persistent problems. IFES worked closely with CECS on access issues for persons with disabilities, primarily during IIMEVP and continuing with SEAA. In doing so, CECS has become much more aware of access issues, has moved to address problems, and has formally and informally engaged with stakeholders to improve elections administration. CECS and its MEOs have produced a stable model for improving access. IFES partnered with KDI and HandiKOS to field the Disability Access Monitoring Mission for the 2014 snap national elections, with observers in the field and a call-in center. Access information collected through this monitoring mission was the core of the data eventually used for the disability access software module. The team found that all ten of the ten KIs with knowledge of PWD issues found the module to be useful. CECS itself revised processes and materials on the basis of a post-election report produced by the two CSOs. Subsequent activities included the establishment of the Election Access Working Group, which brought together CSO representatives, assembly deputies, and representatives from CEC, CECS, ECAP, and government ministries. With IFES assistance, this group produced manuals to guide election officials. CECS continued to improve access for persons with disabilities through polling station improvements and braille materials. The ET found that CECS staff and MEOs all valued IFES’s efforts to improve access for persons with disabilities, as seen in the clear understanding of the norms of inclusion, the revision of manuals, the effect on work plans, and the engagement with CSOs like HandiKOS. These Kis all stated that this brought more citizens into the elections process and helped avoid operational problems that occurred in the past. Another finding is that the CSOs appreciated engagement with MEOs on such issues as physical access, lists for mobile team visits, and persons with disabilities as poll workers. The ET learned that the biggest access problems remains the number of polling stations that are not on the ground floor in buildings without elevators; helpers are available to physically bring persons with disabilities up to the polling station, but CSO KIs noted that PwD communities want to be as independent as possible. 5 For the 2017 elections, MEOs worked with local CSOs to obtain lists of people making proper requests for mobile teams.6 The ET found that three of the five MEO KIs had engaged in informal discussions with political parties to place local CSO staff on mobile teams in an effort to enhance efficiency. Gender Equality. The ET concluded that IFES and its partners had successfully raised the issue to the attention of political parties, but action has not matched rhetoric. IFES raised visibility of the gender equality issue, from candidates on parties’ lists to poll workers, to staffing at CECS and MEOs. The Law on General Elections (2008) requires a political party’s list to contain at least 30 percent female candidates. The ET found that the OPPRC rejects party lists that don’t comply, and those parties are required to 5 This demonstrates that PwDs could not physically access the polling location on their own. HandiKOS representatives noted further that being physically carried was not only embarrassing to the disabled, but also exposed them to physical harm from accidents or untrained helpers. 6 The ET had one report of a municipality where mobile teams made no home visits, only trips to the hospital, a prison, and a special school, despite properly submitted requests. 13 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV resubmit lists that are compliant. The Law on Gender Equality (2015) requires equality in all public institutions, including election management bodies, with the goal of 50 percent gender equality; however, this law is not followed and there are no sanctions to ensure compliance. ECAP. The ET concluded that IFES has significantly contributed to building the capacity of ECAP and has been able to withdraw from almost all technical assistance. Given the singular nature of ECAP’s role, compared with multi-faceted CEC functions, and the institutional memory and professionalism of key members and staff, fully weaning ECAP from IFES entails less risk regarding ongoing functionality. The ET found that IFES’s development of the Complaints & Appeals Management System (CAMS) is rated as the most valuable achievement by ECAP. Mule Desku, ECAP’s Head of Secretariat, said he recognized a need for case management when he assumed his position in 2009. IFES, working over a two-year period encompassing 30 meetings, designed and developed the system.7 ECAP KIs all emphasized how CAMS has greatly enhanced their operations and transparency to the public, but expressed concern about budget constraints that limit their ability to train legal associates hired for elections. ECAP judges are not available to train the legal associates and ECAP staff are busy preparing for the elections, so only on-the-job training is conducted. ECAP KIs also pointed out that they needed further training on the query and reporting functions of CAMS, which would allow them to conduct more detailed analyses of their work. Voters List. The ET concluded that the voters list remains problematic because of both political and complex technical issues, although IFES has worked with CECS to improve the list’s accuracy. The voters list has always been a problem for Kosovo elections, and IFES programming has addressed it in various ways throughout the 10 years of Kosovo independence. IFES’s SEAK programming in 2009 addressed the quality of the voters list, and the subsequent SEAK II and ESP activities also addressed CECS’s capacity to maintain a clean voters list. SEAA includes supporting the CECS IT Department’s coordination with government ministries that are improving the cadaster, as that information would make polling station assignment more accurate. The voters list is inaccurate for three technical reasons. First, some Albanians avoided interacting with the oppressive Serbian state during the late 1980s and 1990s, so vital records for those people do not exist. Second, many Serbs in the territory of Kosovo used (and still use) municipalities in Serbia for a wide variety of administrative obligations, so vital records for those people do not exist in the Kosovo state. Third, Kosovo law does not mandate reporting of deaths, so deceased persons remain on the list. The voters list is inaccurate for two political reasons: 1) deciding who is and is not a citizen of Kosovo would be an existential threat to efforts at regularizing the relationship between Serbia and Kosovo; and 2) inter￾institutional cooperation and coordination in Kosovo is very poor, so CECS and the small set of relevant state bodies are not able to collaborate well to solve this problem. The most critical technical complication has been the lack of street addresses in most of Kosovo, a problem that has been the target of a large European Union-funded project on the cadaster for several years. CECS is working closely with the project and relevant Kosovo institutions to incorporate more accurate information as it becomes available. This information is used to assign voters to the nearest polling station, and better information could reduce the number of conditional votes generated when voters appear at an unassigned polling station. General Capacity-Building. The ET concluded that IFES has helped the entire set of election administration bodies build capacity through professional development, introduction of best practices, and use of a collaborative approach. As the foregoing discussion indicates, IFES has had substantial influence over the development of Kosovo elections administration. It was clear to the ET that personnel, practices, 7 Recently appointed ECAP judge Adnan Konushevci, who was previously ECAP’s legal director from 2010-2015, attended those meetings, solidifying valuable institutional memory. USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 14 standards, and processes in all of the elections administration bodies (CEC, CECS, CRC, ECAP, OPPRC) benefited from IFES expertise. Several other aspects of general capacity were not addressed in the above sections, however, based on points the ET heard from KIs with direct experience in specific election administration bodies. IFES worked with CEC to develop meaningful permanent committees, consider reorganization for more rational reporting and workflow, and facilitated discussions about electoral and elections reform. Two rounds of structural reviews have been the basis for positive organizational changes. For CECS, IFES provided BRIDGE training and facilitator training, brought comparative models and new ideas, and helped rationalize personnel positions. For CRC, IFES helped revamp tabulation and results processes, due as much to embedded personnel as to technical expertise at detailed levels. IFES has been able to step back from technical assistance to ECAP and OPPRC. The ET also concluded that CECS missed opportunities to use non-election time more productively to improve elections administration. As noted, public information remains a weak function for CECS, and non-election periods can be valuable for preparing more sophisticated and impactful strategies and message content. Research and analysis, particularly across the bodies, is another weak area that should be more prominent during non-election periods. For example, ECAP’s CAMS can provide data on the nature of complaints and appeals that would provide important information about Election Day performance of poll workers, but no coordination or collaboration exists across the bodies for learning purposes. EVALUATION QUESTION 2 To what extent have USAID-funded elections activities since 2009 supported a sustainable and independent elections system? ● To what extent are the practices (discussed above) functional and operational? The ET concluded that the CEC, CECS, and ECAP are much more operational and sustainable, although some weaknesses remain and ECAP faces budget constraints. Most of the IT platforms are fully operational and could be taken over by CECS IT staff. The CRC is highly operational but is vulnerable to the departure of its sole staff member, the CRC coordinator. In addition, its main server and software are 10 years old and should be updated completely. OPPRC is fully operational but, like the CRC, vulnerable to the imminent departure of its Coordinator. As with EQ1, the analysis of findings and conclusions for EQ2 and its sub-question is structured according to specific areas and outcomes. IT. The ET concluded that most, but not all, platforms have been fully implemented or taken over by CECS. The IFES-developed Election Operations Modernization Platform (EOMP) has four modules. The current status is: ● Voters can check online “where to go vote”; more than 500,000 voters used that feature in the 2017 Assembly election. The ET found that this module is in the hands of CECS IT staff. ● Voters can change their polling place online to one closer to their home address. This function was pilot-tested in the 2016 election in Gllogovc. This functionality became available for use by any voter beginning with the June 2017 Assembly election, when 3,350 successfully used the online system, compared with 1,919 who changed their polling place by visiting their MEO office. For the October 2017 local elections, the ET found that 3,477 voters updated their polling place using the online system, compared with 4,741 who did so at an MEO office. The ET concluded that voters 15 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV are not utilizing this module as much as they could. The online system sends voters a confirmation email regarding their polling place change. ● The Electoral Violence Monitoring System (EVMS) allows reporting online in real time on Election Day incidences of related violence or voting procedural problems. CSOs and election observers used this feature for the 2017 elections. The ET found that this module is in the hands of CECS IT staff. ● The candidates’ registration and certification module is currently used only in-house by OPPRC to identify candidates’ citizenship and other qualifications. The OPPRC Executive Director expressed reluctance to use the additional capability for political parties to input information online without significant training for political parties followed by pilot testing, which has not yet been done. IFES developed the Digitized Mapping Platform (DMP), which has been in use for several election cycles. The first module was for disability access, followed by general election information and election results. The ET found that this platform has been handed over to CECS IT staff for future use. IFES developed an Election Staff Management System (ESMS) to compile a database of permanent and temporary staff, which is partially in use. Additional functionality on the front end for recruiting and vetting of temporary election workers is available but not in use. Training. The ET concluded that the new decentralized train-the-trainer approach is successful and the number of high-quality, experienced local trainers is adequate. Prior to the 2017 elections, training of MEOs and their regional trainers occurred centrally outside of Pristina at a training academy; the large class size and lecture format was determined to be less effective than hands-on intensive training of the MEOs who subsequently deliver the poll worker training regionally to their trainers. The ET found that a second-iteration training plan was recently completed with little IFES involvement. In collaboration with the CECS, IFES developed new training manuals, which include numerous diagrams and pictures to assist in visualizing key aspects of the procedures, from how to set up the polling place to filling out the results and reconciliation forms (RRFs) after voting and counting are concluded in each polling station. The RRF is complex, resulting in a high percentage of errors by the polling station chairs who fill it out.8 One MEO the ET met pointed out a problem with the word on the RRF to describe “unused” ballots, saying that the word is confusing in Albanian; he said this results in additional errors when completing the RRF on election night. The training division head noted to the ET that she highlighted the 2017 EU EOM’s recommendation to simplify the RRF, which she expressed a desire to work on; the RRF was simplified in time for the 2017 local elections, but this did not affect error rates substantially. Among all of the instructions, the training manual tells poll workers to place a poster that displays a picture of the ballot with instructions in each voting booth; this emphasizes the importance of marking the ballot correctly, so the vote will count. However, the ET found that all CSOs’ KIs said they did not observe posters in voting booths, and that MEOs interviewed were unaware of this instruction to place a poster in voting booths. The ET notes that this instruction was inadequately highlighted in the poll worker training manual. For the October 2017 municipal elections, IFES also responded to the CEC’s procurement problem by urgently designing and printing 800,000 generic single-page leaflets to visually highlight how to mark the ballot in an attempt to lower the percentage of invalid ballots. This quantity was enough to hand one 8 The high volume of errors on both the RRFs and Candidate Reconciliation Forms are repeatedly cited in EOM reports. The RRFs are particularly problematic for polling stations. USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 16 leaflet to each voter upon entering the polling place, in an attempt to augment the CEC’s public information/voter education campaign materials. However, the printing of the leaflets occurred late in the electoral process and the ET found that the MEOs interviewed were unclear regarding how to distribute these leaflets. By law, the CEC forms municipal election commissions (MECs) 15 days after the announcement of elections with political party appointees in proportion to the Assembly representation and with the MEO serving as chair. Before the 2017 elections, the ET found that the CEC’s training division head developed and organized, without IFES assistance, a new certification process for members of MECs that resulted in training approximately 2,000 MECs to introduce the fundamentals of election administration to MEC members. With six new MEOs expected in the near future, this comprehensive training will be necessary for maintaining readiness and work plan implementation. With the institution of the popular and unanimously supported, decentralized train-the-trainer approach, coupled with new training manuals, the ET concluded that the CEC is prepared to assume ongoing management of poll worker training if the CEC allocates funding to hire training experts in advance of elections. Currently, the training division consists of only the division head, who has raised the issue with the Secretariat, requesting the need for additional staff. There is no training unit. The ET found that the training division head has 10 years of experience, giving her deep institutional memory; she speaks Serbian as well as Albanian and is open to new ideas. She convened the Training Management Group of MEOs to produce a second-iteration draft training manual in April 2018 without IFES assistance. She mentioned that she is awaiting the appointment of new CEC members to submit the new training manual for approval. Multiple instances of family voting and abuse of electoral assistance are consistently noted in more than 30 percent of polling stations in multiple organizations’ election observation reports. The ET concurred with EOM recommendations that training needs to emphasize poll workers’ responsibility to make greater efforts to curtail these unlawful activities. CRC. The ET concluded that the CRC has evolved into a highly functional operation with a wealth of institutional knowledge, especially with regard to the very experienced CRC coordinator. However, the coordinator operates as a full-time staff of one, making the CRC’s ongoing operations vulnerable to a single point of failure, should he no longer hold that position. Additionally, the CRC is heavily reliant upon a returning, trained team of core temporary workers whose availability for future elections is not guaranteed. IFES advised regarding development and assisted with implementation of the election results reporting system, K-Vote, which began in 2013. This assistance included the hiring and training of data entry clerks at all polling centers. The clerks are equipped with tablets to enter key statistics from RRFs on Election Day, which are then uploaded to the CEC’s website as preliminary results that night using the K-Vote system. The ET found that IFES greatly assisted in developing a detailed, step-by-step training manual to document the complex, multiple operations of the CRC. This manual guides the work of CRC temporary staff and is also used as reference material by election observers. The ET found that the CRC is not utilizing barcoding for ballot boxes/supplies, although it could greatly enhance the accuracy and speed of processing on election night. The CRC coordinator explained that they conducted a successful pilot project using barcode technology in 2017 in one municipality and he is seeking approval to purchase equipment to implement barcoding for all ballot boxes for future elections. ECAP. The ET concluded that ECAP operations are functional and sustainable. 17 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV IFES was minimally involved with ECAP training prior to 2013 as OSCE previously took on that role. Since 2013, IFES has provided training, placing an emphasis on communication skills. The ET found that, beginning in June 2016, ECAP launched its first strategic plan, developed with IFES support and modeled after CEC’s three-year strategic plan that IFES developed. The ET found that IFES developed a manual and videos for public information, including highlighting the legal deadlines. ECAP KIs rated the manual developed with IFES assistance as very valuable. ECAP KIIs said that IFES has been instrumental in educating the public regarding how to fill out a form to file a complaint/appeal, stating that this awareness-raising has helped to distinguish ECAP’S role as separate from the CEC, i.e., differentiating CEC members as appointed by political parties, while ECAP members are judges, not political appointees. In 2017, IFES focused on training ECAP legal staff regarding drafting legal decisions, developing a template for a consistent approach. ECAP staffs legal associates, many of them with previous ECAP experience, during snap and regularly scheduled elections. ECAP Kls described a similar trend over the past 10 years regarding the number of complaints/appeals filed in each election cycle. For regularly scheduled elections, which have a 30-day window for electoral campaigning, typically 1,000 to 1,300 complaints are filed. For snap elections, with only 10 days of campaigning, approximately 300 complaints are filed, which equates to similar numbers statistically. Most complaints come from political parties and CSOs, with others from individual candidates; most involve claims regarding pre-election campaign violations (about a third involve Election Day complaints). ECAP KIs said that in the 2017 national elections, the largest number of complaints were by CSOs, especially regarding code-of-conduct violations. ECAP KIs expressed a desire for CAMS to offer more customized system reports and queries. IFES told the ET that robust reporting features exist, mentioning that IFES’s IT staff person appears daily at either CEC or ECAP. The ET found that more training is needed from CECS IT staff to solidify knowledge of the CAMS system’s full functionality, for example with custom queries to further their analysis of complaints. IFES recently provided data entry of complaints/appeals dating back to 2009, so data is now publicly available on ECAP’s website from three election cycles, and ECAP is capable of adding data on an ongoing basis. The ET concluded that coordination between CEC and ECAP could be enhanced, based on ECAP KIs mentioning that their staff requested involvement regarding training poll workers on what they need to know about the complaints and appeals process. However, the ET found the poll worker training manual does include a page on ECAP and instructs polling station commissions (PSCs) to inform voters/organizations of their right to file a complaint or appeal with ECAP. Additionally, ECAP KIs told the ET that ECAP recently conducted roundtables with CSOs, MEOs, and political party representatives in seven regions of Kosovo to discuss the appeals process. The roundtables highlighted the need to publicize awareness on how to file a complaint, including using Facebook (a function that was introduced for the 2017 elections). ECAP KIs all highlighted the 72-hour deadline for adjudicating cases as problematic. However, any extension would require a legal change as the deadlines are encased in law, and ECAP KIs admitted that they were able to resolve all complaints/appeals within the allocated time frame. In 2017, ECAP adopted its first annual work plan aligned with strategic planning goals. ECAP KIs stated that IFES organized and developed their training curriculum, but that ECAP delivers the training for their judges on legal reasoning and on investigation. Mule Desku, ECAP Secretariat, said, “We have only seen positive things from IFES as an institution; there is no political influence whatsoever.” USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 18 ECAP KIs said they don’t have the budget to organize trainings, conference space, etc. The ECAP annual budget is €200,000; however, electoral fines that ECAP collected in conjunction with the 2017 elections amounted to €350,000. OPPRC. The ET concluded that the office is operational, but sustainability is at risk. OPPRC currently has only two staff, and the executive director is retiring in January 2019. This is even riskier because the office still processes paperwork manually instead of digitally, as the executive director did not feel comfortable with the online system until further testing occurs. PwDs. The ET concluded that CECS is capable of continuing to ensure access. The ET found that the MEO work plans, based on the CEC strategic plan, included access considerations in identifying new polling stations. Moreover, the MEC Coordinator and MEOs were working to adjust polling station set-up to enhance access where possible. The CECS IT staff is capable of revising information for the module showing access quality. One point of concern for the ET is that CECS does not engage much with CSOs, so the consistency and quality of access information may be in question in the future. Gender Equality. The ET concluded that gender equality goals will not be met without increased attention from political parties when compiling their poll worker lists. IFES has been working with the CEC to take the lead role in promoting this goal. Political parties gave a “lip service” commitment via signing the Declaration for Inclusion of Women in Electoral Management in September 2015, pledging to ensure at least 40 percent representation. However, follow-up action is lacking. Consequently, CSOs’ initial euphoria regarding the declaration has turned to disillusionment. The CEC, with IFES assistance, produced a gender action plan focusing on staffing and outreach, but CECS staffing, including at the MEO level — where only two MEOs are women — remains very male dominant, reflecting cultural attitudes. Current statistics reveal the following election staffing by gender: MECs, 10.5 percent women in 2017; and women poll workers, 17 percent in 2013, 20 percent in 2014, 24 percent in June 2017 (of whom 18 percent were chairs), and 25.7 percent in October 2017. The reserve poll workers pool did include 31 percent women in 2017, and 51 percent of trainers under the MEOs are women. IFES distributed this statistical information to political parties prior to the 2017 elections to remind them that they committed to 40 percent female participation in election staffing when signing the 2015 declaration. The ET found that IFES collaborates with CSOs to promote gender equality. One example is Democracy for Development’s (D4D’s) 2016 Manual for Gender Equality in the Electoral Process, which attributes editing to IFES’s inclusion staff expert. This manual was a coordinated project with CSOs, political parties, and the CEC. Women’s voter turnout has been declining for both the parliamentary and municipal elections. 9 The ET found that the National Democratic Institute (NDI) is emphasizing this statistic, which they said is getting political parties’ attention, noting the general belief that more women election staff at polling stations would likely enhance women’s comfort level regarding voting. Public Information. The ET concluded that public information functions are not adequately operational, due largely to management and planning issues under CECS control. IFES has worked with CECS to build public information capacity for almost 10 years, including civic education initiatives. The troubled public information campaigns for the 2017 elections, especially the June 9 Data shows a decline from 57 percent in the 2013 local elections to 38.7 percent in the 2017 local elections and from 36 percent in the 2014 parliamentary election to 34 percent in the 2017 parliamentary election. 19 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV snap national elections, were caused most immediately by procurement obstacles; much of the campaign planning with respect to developing a media strategy and designing PSAs could have taken place during the non-election period, however, which indicates in part that the real problem lies in poor planning. The dissemination modes of the 2017 public information campaigns show a retrenchment in more traditional media and materials, which is at odds with current best practices around the world. The ET found that CSOs were effective in civic education campaigns and, in the case of persons with disabilities access, collaborative partners, and yet CSOs were not engaged for public information or outreach for the 2017 elections. Similarly, MEOs know their municipalities well, but their public information campaign input is limited to providing quantities requests and they engage with young voters once a year at school presentations. CSOs and MEOs represent another missed opportunity by CECS. General capacity. The ET concluded that CEC, CECS, and ECAP are fully operational and can sustainably administer elections, while CRC and OPPRC are vulnerable to personnel turnover, but are otherwise fully operational. CEC, CECS, CRC, and ECAP are all capable of medium-term and long-term strategic planning, including translating strategic plans into annual work plans. CECS and ECAP are capable of conducting training, and the ET found that CECS has already produced its own follow-on training plan without direct IFES involvement. The CECS IT Department is capable of maintaining or updating various systems developed by IFES and can find a ready supply of programming contractors to hire if procurement processes were more adaptable. Two weak areas are public information and research. The ET concluded that the Voter Information and Education Division is not capable of mounting a modern, sophisticated public information campaign without external technical assistance. The reliance on outdated dissemination modes, the use of poorly￾translated materials, and the lack of forward planning all indicate serious weakness. Research and analysis are the lifeblood of learning organizations, but CECS misses opportunities to learn: no complex analysis of the 2017 media strategy and messaging, no collaboration with other election administration bodies to share information, and no direct partnering with CSOs to generate needed insights. EVALUATION QUESTION 3 What are the challenges and opportunities in implementing USAID’s current election activity, SEEA? The ET concluded that the key challenge to implementing SEAA has been the interruption of capacity-building activities due to snap elections for the Assembly, which forced IFES to refocus on intensive technical assistance in critical areas to support election administration as needed. The primary opportunity stems from experience gained conducting back-to-back Assembly and municipal elections in 2017, which identified additional ways to enhance MEOs and CSOs roles in strengthening the electoral process. Challenges include: ● Snap elections have impeded or postponed organizational development, and refocused IFES’s role from capacity building to intensive technical support and skills enhancement. Electoral management bodies generally use the non-election periods for organizational development, but the rapid pace and frequency of elections in Kosovo has been problematic for this. ● Budget constraints within each of the election administration bodies, especially during non￾election periods, prevented the addition of staff in key areas identified as critical, including in the CRC, CECS IT Department, and the CECS training divisions. The CRC and the CECS training USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 20 division were identified as especially vulnerable due to the lack of any permanent staff to assist these seasoned-veteran division heads. Work plans evolving from strategic plans and updated training manuals are valuable management tools, but cannot reproduce institutional memory, so hiring and training a deputy for these key positions is essential for continuous, effective operations. ● Replacing the decade-old CRC server and software is vital to ensure continuous operation of this critical function. The CRC softward is similarly antiquated and is no longer supported by the vendor. To enhance operations and to avoid repeating an election night failure of CEC’s website, IT equipment modernization needs to be a top CEC budget priority. Additionally, the CRC needs to adopt technology that will streamline operations, including barcoding ballot boxes and digitizing manual processes. ● Faulty, cumbersome procurement procedures are hampering operations — as seen, for example, in the 2017 public information campaign problems. This issue was one of many problem areas addressed in the February 2018 “Lessons Learned” workshop held with CECS and MEOs to review the 2017 elections administration. Creative thinking is needed to overcome obstacles, such as breaking procurements into smaller components or executing year-round procurement agreements. ● CEC and CECS exhibit emotional dependence on IFES, despite less-technical dependence than in the past. The ET found that all CEC and CECS KIs were confident in their skills and experience, but reluctant to go into the next election without IFES presence. There is a need to foster CECS ownership of internal professional development efforts. ● CECS has missed opportunities for efficient use of non-election preparation time for expanding research/analysis and applying lessons learned; for example, more in-depth analysis of invalid ballots, gauging voter knowledge gains through surveys or focus groups in close temporal proximity to the elections or learning across ECAP and CRC. The current excuse is that CEC has no members, but most CECS activities can occur without CEC intervention. ● Six new municipalities will soon require appointment of new MEOs, who will need intensive training on their roles and responsibilities. In 2017, the CEC Training Division head designed and delivered basic election training to 2,000 MEC members and now has the capacity to deliver similar training to new MEOs. Numerous opportunities exist to strengthen election administration capacity, including: ● The growing acceptance and use of technology by citizens provide the CEC with multiple avenues for a more modern approach to electoral information dissemination, including expanded use of social media. The ET found that CECS IT staff were capable of handling software modules received from IFES, but also concluded that the limited use of some modules was likely due to low levels of public awareness. ● The ET found that MEOs have capacity and desire to be more involved and could offer valuable assistance in many activities, such as implementing a year-round process for removing deceased voters after an appropriate protocol has been established to insure a consistent approach. MEOs also have more immediate connections to non-majority community leaders and persons with disabilities CSOs. ● The ET found that CSOs have capacity and reach to assist with public information campaigns, but are currently underutilized. Enhancing collaboration with CSOs is listed in CEC’s work plan for this year. 21 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV ● Gender equality among poll workers could be promoted by the CEC through reliance on the constitution and the Law on Gender Equality to open up positions for nomination by CSOs. ● The ET concluded that improvement in the accuracy of the voters list is possible, especially due to CECS’s renewed emphasis in 2018 on capturing addressing data in conjunction with the cadaster registry. CECS IT Department and Voter Services Unit recently completed addressing updates in 14 municipalities and expects to complete the addressing in all small municipalities by September 2018. However, the IT Department cautioned that this process will take several more years in Pristina and Prizren, for example, as those large cities have more problems with street names and incomplete addressing systems. ● Six new municipalities will provide an opportunity for greater gender balance in selecting new MEOs. The ET found that only four MEOs are women. ● Conducting research and analysis and development of a coordinated strategy to reduce the unacceptably high percentage of invalid ballots (see Annex 1 for detailed recommendation). RECOMMENDATIONS The ET presents below a set of recommendations based on conclusions for each of the EQs that relate to ongoing SEAA programming and additional emphases for however much longer an elections administration technical assistance activity will exist. The ET asserts that the CEC, CECS, CRC, OPPRC, and ECAP are capable of administering elections without external technical assistance, despite some specific areas of weakness or vulnerability. For SEAA’s current programming, the ET recommends: ● Continue push to make IT products more operational. Most of the software and applications are in use, but not all are despite being fully operational. The CECS IT Department has technical capacity and procurement processes to maintain and update software, however the lack of uptake on some IT products is due to management and confidence problems that could be overcome with IFES involvement. ● Facilitate strategic planning driven by CEC, CECS, and ECAP. KIs from all three bodies placed a high value on the first strategic planning process but were less confident about their ability to run that process on their own. IFES involvement as co-facilitator and advisor would be valuable. ● Overhaul public information strategies and planning processes. This is perhaps the biggest weakness in CECS, and one of the more critical influences on the efficient and credible administration of elections. It was clear to the ET that CEC and CECS recognized that the 2017 public information campaigns were problematic, but also that blame was being assigned solely to the problematic procurement process and future reliance on IFES was being assumed. ● Emphasize voter list improvements. While this is being addressed as a special initiative within the CECS IT Department in cooperation with government ministries, it would still be important to have an external expert organization on CECS’s side to help ensure proper addressing of election administration needs. ● Continue pushing for poll worker gender equality. IFES could work with CEC and CECS to develop a gender-focused recruiting strategy that is not dependent on political parties acting under intense time pressure. For example, CECS could include a statement in job advertisements to encourage women to apply for permanent and temporary staff positions. The CEC could also USAID.GOV USAID/KOSOVO SEAA EVALUATION REPORT | 22 invoke the constitution and Law on Gender Equality to justify reaching out via MEOs to CSOs for female poll workers. The ET also presents recommendations that go beyond the budgeted work plan for SEAA’s final year: ● Work closely with the CRC and the CECS IT Department to the CRC’s aging software and servers. The 10-year-old IT system that is the backbone of the CRC’s work has been revised and patched multiple times but is no longer supported by the original vendor. This is a critical element in the CEC’s capacity to administer elections. ● Expand/regularize MEO input and engagement. CECS views MEOs as field staff to be directed, and only solicits their input on limited details or for a one-day post-election workshop. The MEOs are generally veteran elections professionals who implement complex projects and have direct experience with the reality of elections administration. The MEOs as a group represent a missed opportunity for learning by CECS on election administration procedures, public information campaigns, and voter outreach. ● Mentor CECS to partner with CSOs for public information campaigns and voter outreach. CSOs played key positive roles in earlier civic education initiatives, public information campaigns, and voter outreach programs under successive IFES projects, but CEC and CECS have not followed these models in their planning as IFES’s direct role shrank. CEC and CECS view CSOs as critics and competitors, not as potential partners for improving elections administration. ● Coordinate research, analysis, and lessons learned across elections administration bodies. CECS, CRC, ECAP, and OPPRC collect basic statistics and CRC conducts a cursory invalid ballot analysis, but much more could and should be done: post-election surveys and focus groups of voters to gauge the efficacy of the public information media strategy and messaging; deeper analysis of invalid ballots that could inform messaging via political parties; and/or analysis of complaints and appeals that would yield insights into poll worker performance and training. ● Provide more CECS staff with professional development opportunities, particularly those with functions that exist outside the realm of elections administration, such as training and public information campaigns. Annex 1 contains additional recommendations on legal or regulatory issues and on comparative practices that are beyond the scope of technical assistance programs. These include: enhancing the role of MEOs in selecting poll workers; strengthening verification protocols for Out-of-Kosovo Voting; ensuring consistent procedures are followed regarding counting of conditional ballots; adopting a process to recount a small, random sample of ballot boxes to enhance voter confidence in election results; and conducting an in￾depth qualitative analysis of invalid ballots to identify approaches to reduce the high percentage of invalidated ballots. 23 | USAID/KOSOVO SEAA EVALUATION REPORT USAID.GOV U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523